Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251027 21:07:43.059351 9273 test_util.cc:276] Using random seed: 1246735721
2025-10-27T21:07:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T21:07:43Z Disabled control of system clock
I20251027 21:07:43.090767 9273 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu
/tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.14.126:34057
--webserver_interface=127.9.14.126
--webserver_port=0
--builtin_ntp_servers=127.9.14.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.14.126:34057,127.9.14.125:46313,127.9.14.124:45071
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251027 21:07:43.396771 9281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 21:07:43.397069 9281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 21:07:43.397184 9281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 21:07:43.404717 9281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 21:07:43.404804 9281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 21:07:43.404889 9281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 21:07:43.404955 9281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 21:07:43.413727 9281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.14.84:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.14.126:34057,127.9.14.125:46313,127.9.14.124:45071
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.14.126:34057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.14.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 27 Oct 2025 20:43:25 UTC on 24a791456cd2
build id 8748
ASAN enabled
I20251027 21:07:43.415202 9281 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 21:07:43.416842 9281 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 21:07:43.424582 9287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 21:07:43.424614 9286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 21:07:43.424851 9289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 21:07:43.425256 9281 server_base.cc:1047] running on GCE node
I20251027 21:07:43.426040 9281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 21:07:43.427299 9281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 21:07:43.428506 9281 hybrid_clock.cc:648] HybridClock initialized: now 1761599263428458 us; error 43 us; skew 500 ppm
I20251027 21:07:43.428817 9281 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 21:07:43.430754 9281 webserver.cc:492] Webserver started at http://127.9.14.126:38503/ using document root <none> and password file <none>
I20251027 21:07:43.431391 9281 fs_manager.cc:362] Metadata directory not provided
I20251027 21:07:43.431495 9281 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 21:07:43.431814 9281 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 21:07:43.433822 9281 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-0/data/instance:
uuid: "a4b8b31d47da48aaa386a8e64a5749c2"
format_stamp: "Formatted at 2025-10-27 21:07:43 on dist-test-slave-r942"
I20251027 21:07:43.434567 9281 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-0/wal/instance:
uuid: "a4b8b31d47da48aaa386a8e64a5749c2"
format_stamp: "Formatted at 2025-10-27 21:07:43 on dist-test-slave-r942"
I20251027 21:07:43.439203 9281 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20251027 21:07:43.442206 9295 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 21:07:43.443310 9281 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251027 21:07:43.443462 9281 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-0/data,/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-0/wal
uuid: "a4b8b31d47da48aaa386a8e64a5749c2"
format_stamp: "Formatted at 2025-10-27 21:07:43 on dist-test-slave-r942"
I20251027 21:07:43.443636 9281 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 21:07:43.479701 9281 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 21:07:43.480342 9281 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 21:07:43.480597 9281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 21:07:43.499055 9281 rpc_server.cc:307] RPC server started. Bound to: 127.9.14.126:34057
I20251027 21:07:43.499083 9346 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.14.126:34057 every 8 connection(s)
I20251027 21:07:43.500312 9281 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-0/data/info.pb
I20251027 21:07:43.503957 9347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:43.505719 9273 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu as pid 9281
I20251027 21:07:43.506043 9273 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-0/wal/instance
I20251027 21:07:43.509152 9273 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu
/tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.14.125:46313
--webserver_interface=127.9.14.125
--webserver_port=0
--builtin_ntp_servers=127.9.14.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.14.126:34057,127.9.14.125:46313,127.9.14.124:45071
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251027 21:07:43.511914 9347 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } has no permanent_uuid. Determining permanent_uuid...
I20251027 21:07:43.530567 9347 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } has no permanent_uuid. Determining permanent_uuid...
W20251027 21:07:43.531461 9298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.14.125:46313: connect: Connection refused (error 111)
W20251027 21:07:43.532697 9347 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.14.125:46313: Network error: Client connection negotiation failed: client connection to 127.9.14.125:46313: connect: Connection refused (error 111)
I20251027 21:07:43.582034 9347 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } attempt: 1
W20251027 21:07:43.583631 9347 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.14.125:46313: Network error: Client connection negotiation failed: client connection to 127.9.14.125:46313: connect: Connection refused (error 111)
I20251027 21:07:43.651858 9347 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } attempt: 2
W20251027 21:07:43.653482 9347 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.14.125:46313: Network error: Client connection negotiation failed: client connection to 127.9.14.125:46313: connect: Connection refused (error 111)
I20251027 21:07:43.744722 9347 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } attempt: 3
W20251027 21:07:43.746483 9347 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.14.125:46313: Network error: Client connection negotiation failed: client connection to 127.9.14.125:46313: connect: Connection refused (error 111)
W20251027 21:07:43.813143 9349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 21:07:43.813400 9349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 21:07:43.813506 9349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 21:07:43.821029 9349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 21:07:43.821102 9349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 21:07:43.821174 9349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 21:07:43.821211 9349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 21:07:43.830174 9349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.14.84:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.14.126:34057,127.9.14.125:46313,127.9.14.124:45071
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.14.125:46313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.14.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 27 Oct 2025 20:43:25 UTC on 24a791456cd2
build id 8748
ASAN enabled
I20251027 21:07:43.831717 9349 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 21:07:43.833328 9349 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 21:07:43.840708 9358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 21:07:43.840794 9357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 21:07:43.841759 9360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 21:07:43.842327 9349 server_base.cc:1047] running on GCE node
I20251027 21:07:43.843005 9349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 21:07:43.844214 9349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 21:07:43.845412 9349 hybrid_clock.cc:648] HybridClock initialized: now 1761599263845365 us; error 39 us; skew 500 ppm
I20251027 21:07:43.845773 9349 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 21:07:43.847741 9349 webserver.cc:492] Webserver started at http://127.9.14.125:46529/ using document root <none> and password file <none>
I20251027 21:07:43.848402 9349 fs_manager.cc:362] Metadata directory not provided
I20251027 21:07:43.848500 9349 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 21:07:43.848871 9349 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 21:07:43.851054 9349 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/data/instance:
uuid: "18b15927ce5744ad8ba81dda52725c11"
format_stamp: "Formatted at 2025-10-27 21:07:43 on dist-test-slave-r942"
I20251027 21:07:43.851805 9349 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/wal/instance:
uuid: "18b15927ce5744ad8ba81dda52725c11"
format_stamp: "Formatted at 2025-10-27 21:07:43 on dist-test-slave-r942"
I20251027 21:07:43.856752 9349 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 21:07:43.859763 9366 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 21:07:43.860908 9349 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251027 21:07:43.861056 9349 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/data,/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/wal
uuid: "18b15927ce5744ad8ba81dda52725c11"
format_stamp: "Formatted at 2025-10-27 21:07:43 on dist-test-slave-r942"
I20251027 21:07:43.861239 9349 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 21:07:43.877609 9349 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 21:07:43.878415 9349 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 21:07:43.878660 9349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 21:07:43.889722 9347 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } attempt: 4
W20251027 21:07:43.891824 9347 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.14.125:46313: Network error: Client connection negotiation failed: client connection to 127.9.14.125:46313: connect: Connection refused (error 111)
I20251027 21:07:43.898577 9349 rpc_server.cc:307] RPC server started. Bound to: 127.9.14.125:46313
I20251027 21:07:43.898597 9417 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.14.125:46313 every 8 connection(s)
I20251027 21:07:43.900014 9349 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/data/info.pb
I20251027 21:07:43.903802 9418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:43.905740 9273 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu as pid 9349
I20251027 21:07:43.905905 9273 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/wal/instance
I20251027 21:07:43.906924 9273 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu
/tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.14.124:45071
--webserver_interface=127.9.14.124
--webserver_port=0
--builtin_ntp_servers=127.9.14.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.14.126:34057,127.9.14.125:46313,127.9.14.124:45071
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251027 21:07:43.909461 9418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } has no permanent_uuid. Determining permanent_uuid...
I20251027 21:07:43.921754 9418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } has no permanent_uuid. Determining permanent_uuid...
I20251027 21:07:43.930642 9418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } has no permanent_uuid. Determining permanent_uuid...
W20251027 21:07:43.931510 9367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: connect: Connection refused (error 111)
W20251027 21:07:43.932814 9418 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.14.124:45071: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: connect: Connection refused (error 111)
I20251027 21:07:43.982096 9418 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } attempt: 1
W20251027 21:07:43.983937 9418 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.14.124:45071: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: connect: Connection refused (error 111)
I20251027 21:07:44.052246 9418 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } attempt: 2
W20251027 21:07:44.054080 9418 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.14.124:45071: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: connect: Connection refused (error 111)
I20251027 21:07:44.145339 9418 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } attempt: 3
W20251027 21:07:44.147315 9418 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.14.124:45071: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: connect: Connection refused (error 111)
I20251027 21:07:44.191077 9347 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } attempt: 5
I20251027 21:07:44.198264 9347 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } has no permanent_uuid. Determining permanent_uuid...
W20251027 21:07:44.199894 9347 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.14.124:45071: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: connect: Connection refused (error 111)
W20251027 21:07:44.230240 9420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 21:07:44.230522 9420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 21:07:44.230641 9420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 21:07:44.238572 9420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 21:07:44.238646 9420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 21:07:44.238720 9420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 21:07:44.238775 9420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 21:07:44.248692 9420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.14.84:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.14.126:34057,127.9.14.125:46313,127.9.14.124:45071
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.14.124:45071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.14.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 27 Oct 2025 20:43:25 UTC on 24a791456cd2
build id 8748
ASAN enabled
I20251027 21:07:44.250180 9420 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 21:07:44.251081 9347 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } attempt: 1
I20251027 21:07:44.252106 9420 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 21:07:44.252806 9347 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.14.124:45071: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: connect: Connection refused (error 111)
W20251027 21:07:44.260195 9429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 21:07:44.260308 9428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 21:07:44.261214 9431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 21:07:44.261456 9420 server_base.cc:1047] running on GCE node
I20251027 21:07:44.262171 9420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 21:07:44.263451 9420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 21:07:44.264642 9420 hybrid_clock.cc:648] HybridClock initialized: now 1761599264264600 us; error 34 us; skew 500 ppm
I20251027 21:07:44.264963 9420 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 21:07:44.266985 9420 webserver.cc:492] Webserver started at http://127.9.14.124:41907/ using document root <none> and password file <none>
I20251027 21:07:44.267699 9420 fs_manager.cc:362] Metadata directory not provided
I20251027 21:07:44.267798 9420 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 21:07:44.268167 9420 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 21:07:44.270354 9420 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data/instance:
uuid: "6e86ca5e83dd4df5b72f045e7b787e81"
format_stamp: "Formatted at 2025-10-27 21:07:44 on dist-test-slave-r942"
I20251027 21:07:44.271062 9420 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/wal/instance:
uuid: "6e86ca5e83dd4df5b72f045e7b787e81"
format_stamp: "Formatted at 2025-10-27 21:07:44 on dist-test-slave-r942"
I20251027 21:07:44.276237 9420 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 21:07:44.279230 9437 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 21:07:44.280498 9420 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20251027 21:07:44.280666 9420 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data,/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/wal
uuid: "6e86ca5e83dd4df5b72f045e7b787e81"
format_stamp: "Formatted at 2025-10-27 21:07:44 on dist-test-slave-r942"
I20251027 21:07:44.280856 9420 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 21:07:44.290473 9418 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } attempt: 4
W20251027 21:07:44.292032 9418 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.14.124:45071: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: connect: Connection refused (error 111)
I20251027 21:07:44.306854 9420 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 21:07:44.307574 9420 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 21:07:44.307790 9420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 21:07:44.321022 9347 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } attempt: 2
W20251027 21:07:44.322988 9347 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.14.124:45071: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: connect: Connection refused (error 111)
I20251027 21:07:44.328471 9420 rpc_server.cc:307] RPC server started. Bound to: 127.9.14.124:45071
I20251027 21:07:44.328488 9488 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.14.124:45071 every 8 connection(s)
I20251027 21:07:44.329835 9420 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data/info.pb
I20251027 21:07:44.332775 9273 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu as pid 9420
I20251027 21:07:44.332983 9273 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/wal/instance
I20251027 21:07:44.334177 9489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:44.341291 9489 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } has no permanent_uuid. Determining permanent_uuid...
I20251027 21:07:44.354388 9489 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } has no permanent_uuid. Determining permanent_uuid...
I20251027 21:07:44.361680 9489 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } has no permanent_uuid. Determining permanent_uuid...
I20251027 21:07:44.372004 9489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: Bootstrap starting.
I20251027 21:07:44.374878 9489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:44.376549 9489 log.cc:826] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: Log is configured to *not* fsync() on all Append() calls
I20251027 21:07:44.378717 9489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: No bootstrap required, opened a new log
I20251027 21:07:44.383507 9489 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:44.383747 9489 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:44.383832 9489 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e86ca5e83dd4df5b72f045e7b787e81, State: Initialized, Role: FOLLOWER
I20251027 21:07:44.384300 9489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:44.384999 9498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:44.385232 9498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 21:07:44.385923 9489 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: configured and running, proceeding with master startup.
W20251027 21:07:44.394953 9509 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251027 21:07:44.395062 9509 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251027 21:07:44.392493 9420 master_runner.cc:186] Error getting master registration for 127.9.14.126:34057: OK, instance_id { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" instance_seqno: 1761599263494028 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251027 21:07:44.398264 9420 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251027 21:07:44.429302 9347 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } attempt: 3
I20251027 21:07:44.439323 9347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Bootstrap starting.
I20251027 21:07:44.442245 9347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:44.443763 9347 log.cc:826] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Log is configured to *not* fsync() on all Append() calls
I20251027 21:07:44.446684 9347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: No bootstrap required, opened a new log
I20251027 21:07:44.451385 9347 raft_consensus.cc:359] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:44.451608 9347 raft_consensus.cc:385] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:44.451695 9347 raft_consensus.cc:740] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b8b31d47da48aaa386a8e64a5749c2, State: Initialized, Role: FOLLOWER
I20251027 21:07:44.452245 9347 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:44.453073 9512 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:44.453351 9512 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 21:07:44.454083 9347 sys_catalog.cc:565] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: configured and running, proceeding with master startup.
W20251027 21:07:44.464290 9523 catalog_manager.cc:1568] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251027 21:07:44.464473 9523 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251027 21:07:44.463181 9281 master_runner.cc:186] Error getting master registration for 127.9.14.125:46313: OK, instance_id { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" instance_seqno: 1761599263893304 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251027 21:07:44.469955 9281 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251027 21:07:44.482988 9301 tablet.cc:2378] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 21:07:44.559563 9443 tablet.cc:2378] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 21:07:44.591267 9418 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } attempt: 5
I20251027 21:07:44.604202 9418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11: Bootstrap starting.
I20251027 21:07:44.608701 9418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:44.609938 9418 log.cc:826] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11: Log is configured to *not* fsync() on all Append() calls
I20251027 21:07:44.612573 9418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11: No bootstrap required, opened a new log
I20251027 21:07:44.620064 9418 raft_consensus.cc:359] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:44.620347 9418 raft_consensus.cc:385] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:44.620464 9418 raft_consensus.cc:740] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b15927ce5744ad8ba81dda52725c11, State: Initialized, Role: FOLLOWER
I20251027 21:07:44.621088 9418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:44.622014 9526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:44.622390 9526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 21:07:44.623245 9418 sys_catalog.cc:565] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [sys.catalog]: configured and running, proceeding with master startup.
W20251027 21:07:44.631783 9372 tablet.cc:2378] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 21:07:44.632532 9512 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:44.632802 9512 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:44.634951 9512 leader_election.cc:290] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18b15927ce5744ad8ba81dda52725c11 (127.9.14.125:46313), 6e86ca5e83dd4df5b72f045e7b787e81 (127.9.14.124:45071)
I20251027 21:07:44.635562 9393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b15927ce5744ad8ba81dda52725c11" is_pre_election: true
I20251027 21:07:44.635991 9393 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4b8b31d47da48aaa386a8e64a5749c2 in term 0.
I20251027 21:07:44.635852 9464 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" is_pre_election: true
I20251027 21:07:44.636317 9464 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4b8b31d47da48aaa386a8e64a5749c2 in term 0.
I20251027 21:07:44.636744 9298 leader_election.cc:304] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b15927ce5744ad8ba81dda52725c11, a4b8b31d47da48aaa386a8e64a5749c2; no voters:
I20251027 21:07:44.637430 9512 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:44.637563 9512 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:44.637660 9512 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:44.641427 9512 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:44.642501 9393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b15927ce5744ad8ba81dda52725c11"
I20251027 21:07:44.642692 9393 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:44.645866 9393 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b8b31d47da48aaa386a8e64a5749c2 in term 1.
I20251027 21:07:44.646379 9298 leader_election.cc:304] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b15927ce5744ad8ba81dda52725c11, a4b8b31d47da48aaa386a8e64a5749c2; no voters:
I20251027 21:07:44.649690 9464 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e86ca5e83dd4df5b72f045e7b787e81"
I20251027 21:07:44.649879 9464 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:44.653215 9464 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b8b31d47da48aaa386a8e64a5749c2 in term 1.
I20251027 21:07:44.653975 9512 leader_election.cc:290] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [CANDIDATE]: Term 1 election: Requested vote from peers 18b15927ce5744ad8ba81dda52725c11 (127.9.14.125:46313), 6e86ca5e83dd4df5b72f045e7b787e81 (127.9.14.124:45071)
I20251027 21:07:44.656443 9512 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:44.657073 9512 raft_consensus.cc:697] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 1 LEADER]: Becoming Leader. State: Replica: a4b8b31d47da48aaa386a8e64a5749c2, State: Running, Role: LEADER
I20251027 21:07:44.657968 9512 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:44.666846 9512 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4b8b31d47da48aaa386a8e64a5749c2. Latest consensus state: current_term: 1 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:44.671257 9512 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: This master's current role is: LEADER
I20251027 21:07:44.682737 9540 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 21:07:44.685505 9540 catalog_manager.cc:1494] Initializing Kudu cluster ID...
W20251027 21:07:44.696661 9539 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251027 21:07:44.696784 9539 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251027 21:07:44.700093 9464 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 1 FOLLOWER]: Refusing update from remote peer a4b8b31d47da48aaa386a8e64a5749c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 21:07:44.700968 9512 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:07:44.711478 9393 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 1 FOLLOWER]: Refusing update from remote peer a4b8b31d47da48aaa386a8e64a5749c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 21:07:44.712426 9512 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:07:44.736979 9498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4b8b31d47da48aaa386a8e64a5749c2. Latest consensus state: current_term: 1 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:44.737231 9498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 21:07:44.742049 9542 mvcc.cc:204] Tried to move back new op lower bound from 7215510588201361408 to 7215510588049330176. Current Snapshot: MvccSnapshot[applied={T|T < 7215510588201361408}]
I20251027 21:07:44.746894 9526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4b8b31d47da48aaa386a8e64a5749c2. Latest consensus state: current_term: 1 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:44.747153 9526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 21:07:44.753225 9531 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:44.753551 9531 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: This master's current role is: LEADER
I20251027 21:07:44.754249 9531 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:44.754482 9531 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: This master's current role is: LEADER
I20251027 21:07:44.756443 9540 catalog_manager.cc:1357] Generated new cluster ID: 074cb2d5dc574bb39dc828c9d64a191a
I20251027 21:07:44.756553 9540 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 21:07:44.763290 9526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:44.763484 9526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 21:07:44.772061 9498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:44.772351 9498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 21:07:44.792639 9540 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 21:07:44.794590 9540 catalog_manager.cc:1514] Loading token signing keys...
I20251027 21:07:44.831995 9540 catalog_manager.cc:6022] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Generated new TSK 0
I20251027 21:07:44.833127 9540 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 21:07:44.865939 9273 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu
/tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.14.65:0
--local_ip_for_outbound_sockets=127.9.14.65
--webserver_interface=127.9.14.65
--webserver_port=0
--tserver_master_addrs=127.9.14.126:34057,127.9.14.125:46313,127.9.14.124:45071
--builtin_ntp_servers=127.9.14.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20251027 21:07:45.226899 9550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 21:07:45.227145 9550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 21:07:45.227254 9550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 21:07:45.234474 9550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 21:07:45.234640 9550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.14.65
I20251027 21:07:45.243837 9550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.14.84:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.14.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.14.65
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.9.14.126:34057,127.9.14.125:46313,127.9.14.124:45071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.14.65
--log_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 27 Oct 2025 20:43:25 UTC on 24a791456cd2
build id 8748
ASAN enabled
I20251027 21:07:45.245252 9550 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 21:07:45.246927 9550 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 21:07:45.255857 9556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 21:07:45.255865 9555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 21:07:45.256541 9558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 21:07:45.257272 9550 server_base.cc:1047] running on GCE node
I20251027 21:07:45.257822 9550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 21:07:45.258663 9550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 21:07:45.259850 9550 hybrid_clock.cc:648] HybridClock initialized: now 1761599265259805 us; error 35 us; skew 500 ppm
I20251027 21:07:45.260103 9550 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 21:07:45.261910 9550 webserver.cc:492] Webserver started at http://127.9.14.65:43971/ using document root <none> and password file <none>
I20251027 21:07:45.262580 9550 fs_manager.cc:362] Metadata directory not provided
I20251027 21:07:45.262686 9550 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 21:07:45.263011 9550 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 21:07:45.265197 9550 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/data/instance:
uuid: "51902b6b61ff4d5584ff21896e847e14"
format_stamp: "Formatted at 2025-10-27 21:07:45 on dist-test-slave-r942"
I20251027 21:07:45.265887 9550 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/instance:
uuid: "51902b6b61ff4d5584ff21896e847e14"
format_stamp: "Formatted at 2025-10-27 21:07:45 on dist-test-slave-r942"
I20251027 21:07:45.270730 9550 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20251027 21:07:45.273659 9564 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 21:07:45.274801 9550 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251027 21:07:45.274945 9550 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal
uuid: "51902b6b61ff4d5584ff21896e847e14"
format_stamp: "Formatted at 2025-10-27 21:07:45 on dist-test-slave-r942"
I20251027 21:07:45.275115 9550 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 21:07:45.314663 9550 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 21:07:45.315421 9550 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 21:07:45.315665 9550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 21:07:45.316501 9550 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 21:07:45.317953 9550 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 21:07:45.318050 9550 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 21:07:45.318174 9550 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 21:07:45.318248 9550 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20251027 21:07:45.328228 9592 raft_consensus.cc:493] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4b8b31d47da48aaa386a8e64a5749c2)
I20251027 21:07:45.328928 9596 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4b8b31d47da48aaa386a8e64a5749c2)
I20251027 21:07:45.328956 9592 raft_consensus.cc:515] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:45.329174 9596 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:45.331018 9322 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" is_pre_election: true
I20251027 21:07:45.331171 9321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18b15927ce5744ad8ba81dda52725c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" is_pre_election: true
I20251027 21:07:45.332682 9463 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18b15927ce5744ad8ba81dda52725c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" is_pre_election: true
I20251027 21:07:45.332829 9596 leader_election.cc:290] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4b8b31d47da48aaa386a8e64a5749c2 (127.9.14.126:34057), 18b15927ce5744ad8ba81dda52725c11 (127.9.14.125:46313)
I20251027 21:07:45.332940 9463 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18b15927ce5744ad8ba81dda52725c11 in term 1.
I20251027 21:07:45.332984 9392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "18b15927ce5744ad8ba81dda52725c11" is_pre_election: true
I20251027 21:07:45.333155 9392 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e86ca5e83dd4df5b72f045e7b787e81 in term 1.
I20251027 21:07:45.333501 9367 leader_election.cc:304] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18b15927ce5744ad8ba81dda52725c11, 6e86ca5e83dd4df5b72f045e7b787e81; no voters: a4b8b31d47da48aaa386a8e64a5749c2
I20251027 21:07:45.333654 9440 leader_election.cc:304] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18b15927ce5744ad8ba81dda52725c11, 6e86ca5e83dd4df5b72f045e7b787e81; no voters: a4b8b31d47da48aaa386a8e64a5749c2
I20251027 21:07:45.334429 9596 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251027 21:07:45.334637 9596 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4b8b31d47da48aaa386a8e64a5749c2)
I20251027 21:07:45.334815 9596 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 1 FOLLOWER]: Advancing to term 2
I20251027 21:07:45.338739 9596 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:45.339398 9592 leader_election.cc:290] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4b8b31d47da48aaa386a8e64a5749c2 (127.9.14.126:34057), 6e86ca5e83dd4df5b72f045e7b787e81 (127.9.14.124:45071)
I20251027 21:07:45.339700 9592 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251027 21:07:45.339926 9321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4b8b31d47da48aaa386a8e64a5749c2"
I20251027 21:07:45.339985 9592 raft_consensus.cc:493] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4b8b31d47da48aaa386a8e64a5749c2)
I20251027 21:07:45.340077 9592 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 1 FOLLOWER]: Advancing to term 2
I20251027 21:07:45.341120 9392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "18b15927ce5744ad8ba81dda52725c11"
I20251027 21:07:45.341431 9596 leader_election.cc:290] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [CANDIDATE]: Term 2 election: Requested vote from peers a4b8b31d47da48aaa386a8e64a5749c2 (127.9.14.126:34057), 18b15927ce5744ad8ba81dda52725c11 (127.9.14.125:46313)
I20251027 21:07:45.342973 9592 raft_consensus.cc:515] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:45.343734 9392 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e86ca5e83dd4df5b72f045e7b787e81 in current term 2: Already voted for candidate 18b15927ce5744ad8ba81dda52725c11 in this term.
I20251027 21:07:45.344688 9440 leader_election.cc:304] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e86ca5e83dd4df5b72f045e7b787e81; no voters: 18b15927ce5744ad8ba81dda52725c11, a4b8b31d47da48aaa386a8e64a5749c2
I20251027 21:07:45.344954 9596 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251027 21:07:45.345856 9321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18b15927ce5744ad8ba81dda52725c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4b8b31d47da48aaa386a8e64a5749c2"
I20251027 21:07:45.347025 9592 leader_election.cc:290] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [CANDIDATE]: Term 2 election: Requested vote from peers a4b8b31d47da48aaa386a8e64a5749c2 (127.9.14.126:34057), 6e86ca5e83dd4df5b72f045e7b787e81 (127.9.14.124:45071)
I20251027 21:07:45.347110 9463 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18b15927ce5744ad8ba81dda52725c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e86ca5e83dd4df5b72f045e7b787e81"
I20251027 21:07:45.347354 9463 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 18b15927ce5744ad8ba81dda52725c11 in current term 2: Already voted for candidate 6e86ca5e83dd4df5b72f045e7b787e81 in this term.
I20251027 21:07:45.347842 9367 leader_election.cc:304] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18b15927ce5744ad8ba81dda52725c11; no voters: 6e86ca5e83dd4df5b72f045e7b787e81, a4b8b31d47da48aaa386a8e64a5749c2
I20251027 21:07:45.348278 9592 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251027 21:07:45.359568 9392 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 2 FOLLOWER]: Rejecting Update request from peer a4b8b31d47da48aaa386a8e64a5749c2 for earlier term 1. Current term is 2. Ops: []
I20251027 21:07:45.360173 9654 consensus_queue.cc:1059] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251027 21:07:45.360822 9654 raft_consensus.cc:3055] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 1 LEADER]: Stepping down as leader of term 1
I20251027 21:07:45.360958 9654 raft_consensus.cc:740] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4b8b31d47da48aaa386a8e64a5749c2, State: Running, Role: LEADER
I20251027 21:07:45.361366 9654 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:45.362042 9654 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 1 FOLLOWER]: Advancing to term 2
I20251027 21:07:45.370414 9550 rpc_server.cc:307] RPC server started. Bound to: 127.9.14.65:44531
I20251027 21:07:45.370479 9681 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.14.65:44531 every 8 connection(s)
I20251027 21:07:45.371606 9550 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/data/info.pb
I20251027 21:07:45.379609 9273 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu as pid 9550
I20251027 21:07:45.379871 9273 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/instance
I20251027 21:07:45.398277 9509 catalog_manager.cc:1269] Loaded cluster ID: 074cb2d5dc574bb39dc828c9d64a191a
I20251027 21:07:45.398488 9509 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: loading cluster ID for follower catalog manager: success
I20251027 21:07:45.403182 9509 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: acquiring CA information for follower catalog manager: success
I20251027 21:07:45.406251 9509 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251027 21:07:45.424901 9684 heartbeater.cc:344] Connected to a master server at 127.9.14.125:46313
I20251027 21:07:45.425027 9683 heartbeater.cc:344] Connected to a master server at 127.9.14.126:34057
I20251027 21:07:45.425052 9682 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:07:45.425364 9682 heartbeater.cc:461] Registering TS with master...
I20251027 21:07:45.425366 9683 heartbeater.cc:461] Registering TS with master...
I20251027 21:07:45.425365 9684 heartbeater.cc:461] Registering TS with master...
I20251027 21:07:45.427037 9682 heartbeater.cc:507] Master 127.9.14.124:45071 requested a full tablet report, sending...
I20251027 21:07:45.427066 9684 heartbeater.cc:507] Master 127.9.14.125:46313 requested a full tablet report, sending...
I20251027 21:07:45.427101 9683 heartbeater.cc:507] Master 127.9.14.126:34057 requested a full tablet report, sending...
I20251027 21:07:45.429018 9383 ts_manager.cc:194] Registered new tserver with Master: 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:45.429176 9312 ts_manager.cc:194] Registered new tserver with Master: 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:45.429176 9454 ts_manager.cc:194] Registered new tserver with Master: 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:45.466629 9523 catalog_manager.cc:1612] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251027 21:07:45.698629 9539 catalog_manager.cc:1269] Loaded cluster ID: 074cb2d5dc574bb39dc828c9d64a191a
I20251027 21:07:45.698731 9539 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11: loading cluster ID for follower catalog manager: success
I20251027 21:07:45.701309 9539 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11: acquiring CA information for follower catalog manager: success
I20251027 21:07:45.703058 9539 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251027 21:07:45.811764 9654 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:45.811976 9654 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:45.812530 9654 leader_election.cc:290] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 18b15927ce5744ad8ba81dda52725c11 (127.9.14.125:46313), 6e86ca5e83dd4df5b72f045e7b787e81 (127.9.14.124:45071)
I20251027 21:07:45.812902 9392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "18b15927ce5744ad8ba81dda52725c11" is_pre_election: true
I20251027 21:07:45.812969 9463 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" is_pre_election: true
I20251027 21:07:45.813100 9392 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4b8b31d47da48aaa386a8e64a5749c2 in term 2.
I20251027 21:07:45.813190 9463 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4b8b31d47da48aaa386a8e64a5749c2 in term 2.
I20251027 21:07:45.813550 9298 leader_election.cc:304] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b15927ce5744ad8ba81dda52725c11, a4b8b31d47da48aaa386a8e64a5749c2; no voters:
I20251027 21:07:45.813817 9654 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251027 21:07:45.813892 9654 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:45.813992 9654 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 2 FOLLOWER]: Advancing to term 3
I20251027 21:07:45.815919 9654 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:45.816421 9654 leader_election.cc:290] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [CANDIDATE]: Term 3 election: Requested vote from peers 18b15927ce5744ad8ba81dda52725c11 (127.9.14.125:46313), 6e86ca5e83dd4df5b72f045e7b787e81 (127.9.14.124:45071)
I20251027 21:07:45.816787 9392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "18b15927ce5744ad8ba81dda52725c11"
I20251027 21:07:45.816803 9463 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e86ca5e83dd4df5b72f045e7b787e81"
I20251027 21:07:45.816959 9463 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 2 FOLLOWER]: Advancing to term 3
I20251027 21:07:45.816978 9392 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 2 FOLLOWER]: Advancing to term 3
I20251027 21:07:45.819965 9463 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b8b31d47da48aaa386a8e64a5749c2 in term 3.
I20251027 21:07:45.819953 9392 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b8b31d47da48aaa386a8e64a5749c2 in term 3.
I20251027 21:07:45.820396 9298 leader_election.cc:304] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b15927ce5744ad8ba81dda52725c11, a4b8b31d47da48aaa386a8e64a5749c2; no voters:
I20251027 21:07:45.820639 9654 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 3 FOLLOWER]: Leader election won for term 3
I20251027 21:07:45.820777 9654 raft_consensus.cc:697] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [term 3 LEADER]: Becoming Leader. State: Replica: a4b8b31d47da48aaa386a8e64a5749c2, State: Running, Role: LEADER
I20251027 21:07:45.821004 9654 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:45.821749 9656 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4b8b31d47da48aaa386a8e64a5749c2. Latest consensus state: current_term: 3 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:45.821970 9656 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: This master's current role is: LEADER
I20251027 21:07:45.822523 9692 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 21:07:45.824590 9692 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 21:07:45.825414 9692 catalog_manager.cc:1269] Loaded cluster ID: 074cb2d5dc574bb39dc828c9d64a191a
I20251027 21:07:45.825493 9692 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 21:07:45.826678 9692 catalog_manager.cc:1514] Loading token signing keys...
I20251027 21:07:45.827371 9692 catalog_manager.cc:6033] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Loaded TSK: 0
I20251027 21:07:45.828109 9692 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 21:07:45.838862 9273 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu
/tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.14.66:0
--local_ip_for_outbound_sockets=127.9.14.66
--webserver_interface=127.9.14.66
--webserver_port=0
--tserver_master_addrs=127.9.14.126:34057,127.9.14.125:46313,127.9.14.124:45071
--builtin_ntp_servers=127.9.14.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251027 21:07:45.933736 9312 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.14.65:32849
W20251027 21:07:46.157819 9693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 21:07:46.158067 9693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 21:07:46.158192 9693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 21:07:46.166173 9693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 21:07:46.166342 9693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.14.66
I20251027 21:07:46.175448 9693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.14.84:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.14.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.14.66
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.9.14.126:34057,127.9.14.125:46313,127.9.14.124:45071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.14.66
--log_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 27 Oct 2025 20:43:25 UTC on 24a791456cd2
build id 8748
ASAN enabled
I20251027 21:07:46.176927 9693 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 21:07:46.178676 9693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 21:07:46.188091 9698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 21:07:46.188128 9699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 21:07:46.188829 9701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 21:07:46.189275 9693 server_base.cc:1047] running on GCE node
I20251027 21:07:46.189757 9693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 21:07:46.190610 9693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 21:07:46.191802 9693 hybrid_clock.cc:648] HybridClock initialized: now 1761599266191751 us; error 44 us; skew 500 ppm
I20251027 21:07:46.192057 9693 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 21:07:46.194051 9693 webserver.cc:492] Webserver started at http://127.9.14.66:42823/ using document root <none> and password file <none>
I20251027 21:07:46.194751 9693 fs_manager.cc:362] Metadata directory not provided
I20251027 21:07:46.194851 9693 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 21:07:46.195186 9693 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 21:07:46.197269 9693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/data/instance:
uuid: "77b0513f17c0472088901a1ce8b08218"
format_stamp: "Formatted at 2025-10-27 21:07:46 on dist-test-slave-r942"
I20251027 21:07:46.197932 9693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/instance:
uuid: "77b0513f17c0472088901a1ce8b08218"
format_stamp: "Formatted at 2025-10-27 21:07:46 on dist-test-slave-r942"
I20251027 21:07:46.203248 9693 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20251027 21:07:46.207446 9707 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 21:07:46.208770 9693 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 21:07:46.208935 9693 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal
uuid: "77b0513f17c0472088901a1ce8b08218"
format_stamp: "Formatted at 2025-10-27 21:07:46 on dist-test-slave-r942"
I20251027 21:07:46.209129 9693 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 21:07:46.223188 9693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 21:07:46.223975 9693 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 21:07:46.224221 9693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 21:07:46.225088 9693 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 21:07:46.226688 9693 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 21:07:46.227244 9693 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 21:07:46.227352 9693 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 21:07:46.227422 9693 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 21:07:46.261518 9693 rpc_server.cc:307] RPC server started. Bound to: 127.9.14.66:38579
I20251027 21:07:46.261579 9819 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.14.66:38579 every 8 connection(s)
I20251027 21:07:46.262940 9693 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/data/info.pb
I20251027 21:07:46.264022 9273 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu as pid 9693
I20251027 21:07:46.264225 9273 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/instance
I20251027 21:07:46.270958 9273 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu
/tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.14.67:0
--local_ip_for_outbound_sockets=127.9.14.67
--webserver_interface=127.9.14.67
--webserver_port=0
--tserver_master_addrs=127.9.14.126:34057,127.9.14.125:46313,127.9.14.124:45071
--builtin_ntp_servers=127.9.14.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251027 21:07:46.291963 9820 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:07:46.292341 9820 heartbeater.cc:461] Registering TS with master...
I20251027 21:07:46.293803 9820 heartbeater.cc:507] Master 127.9.14.124:45071 requested a full tablet report, sending...
I20251027 21:07:46.295629 9454 ts_manager.cc:194] Registered new tserver with Master: 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:46.296630 9822 heartbeater.cc:344] Connected to a master server at 127.9.14.125:46313
I20251027 21:07:46.296756 9822 heartbeater.cc:461] Registering TS with master...
I20251027 21:07:46.297181 9822 heartbeater.cc:507] Master 127.9.14.125:46313 requested a full tablet report, sending...
I20251027 21:07:46.297382 9821 heartbeater.cc:344] Connected to a master server at 127.9.14.126:34057
I20251027 21:07:46.297519 9821 heartbeater.cc:461] Registering TS with master...
I20251027 21:07:46.297991 9821 heartbeater.cc:507] Master 127.9.14.126:34057 requested a full tablet report, sending...
I20251027 21:07:46.298000 9383 ts_manager.cc:194] Registered new tserver with Master: 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:46.298833 9312 ts_manager.cc:194] Registered new tserver with Master: 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:46.299566 9312 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.14.66:44229
I20251027 21:07:46.375283 9392 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 3 FOLLOWER]: Refusing update from remote peer a4b8b31d47da48aaa386a8e64a5749c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251027 21:07:46.375984 9831 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251027 21:07:46.377882 9832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4b8b31d47da48aaa386a8e64a5749c2. Latest consensus state: current_term: 3 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:46.378126 9832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 21:07:46.380745 9463 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 3 FOLLOWER]: Refusing update from remote peer a4b8b31d47da48aaa386a8e64a5749c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251027 21:07:46.387279 9834 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:46.387822 9834 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: This master's current role is: LEADER
I20251027 21:07:46.392164 9837 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251027 21:07:46.395219 9832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:46.395632 9832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 21:07:46.399487 9843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:46.399600 9840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4b8b31d47da48aaa386a8e64a5749c2. Latest consensus state: current_term: 3 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:46.399912 9843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 21:07:46.399960 9840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 21:07:46.435981 9683 heartbeater.cc:499] Master 127.9.14.126:34057 was elected leader, sending a full tablet report...
W20251027 21:07:46.631146 9830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 21:07:46.631415 9830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 21:07:46.631515 9830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 21:07:46.639475 9830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 21:07:46.639664 9830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.14.67
I20251027 21:07:46.648557 9830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.14.84:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.14.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.14.67
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.9.14.126:34057,127.9.14.125:46313,127.9.14.124:45071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.14.67
--log_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 27 Oct 2025 20:43:25 UTC on 24a791456cd2
build id 8748
ASAN enabled
I20251027 21:07:46.650095 9830 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 21:07:46.652383 9830 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 21:07:46.661357 9848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 21:07:46.661327 9849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 21:07:46.662339 9851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 21:07:46.662709 9830 server_base.cc:1047] running on GCE node
I20251027 21:07:46.663198 9830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 21:07:46.664072 9830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 21:07:46.665305 9830 hybrid_clock.cc:648] HybridClock initialized: now 1761599266665226 us; error 41 us; skew 500 ppm
I20251027 21:07:46.665576 9830 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 21:07:46.667582 9830 webserver.cc:492] Webserver started at http://127.9.14.67:43391/ using document root <none> and password file <none>
I20251027 21:07:46.668272 9830 fs_manager.cc:362] Metadata directory not provided
I20251027 21:07:46.668382 9830 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 21:07:46.668735 9830 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 21:07:46.671356 9830 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/data/instance:
uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
format_stamp: "Formatted at 2025-10-27 21:07:46 on dist-test-slave-r942"
I20251027 21:07:46.672055 9830 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/instance:
uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
format_stamp: "Formatted at 2025-10-27 21:07:46 on dist-test-slave-r942"
I20251027 21:07:46.677094 9830 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20251027 21:07:46.680207 9857 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 21:07:46.681350 9830 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:46.681491 9830 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal
uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
format_stamp: "Formatted at 2025-10-27 21:07:46 on dist-test-slave-r942"
I20251027 21:07:46.681689 9830 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 21:07:46.704474 9830 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 21:07:46.705201 9830 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 21:07:46.705442 9830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 21:07:46.706327 9830 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 21:07:46.707917 9830 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 21:07:46.708016 9830 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 21:07:46.708127 9830 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 21:07:46.708204 9830 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 21:07:46.742637 9830 rpc_server.cc:307] RPC server started. Bound to: 127.9.14.67:36705
I20251027 21:07:46.742702 9969 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.14.67:36705 every 8 connection(s)
I20251027 21:07:46.743947 9830 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/data/info.pb
I20251027 21:07:46.746529 9273 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu as pid 9830
I20251027 21:07:46.746722 9273 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/instance
I20251027 21:07:46.764015 9971 heartbeater.cc:344] Connected to a master server at 127.9.14.126:34057
I20251027 21:07:46.764052 9972 heartbeater.cc:344] Connected to a master server at 127.9.14.125:46313
I20251027 21:07:46.764088 9970 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:07:46.764429 9972 heartbeater.cc:461] Registering TS with master...
I20251027 21:07:46.764427 9970 heartbeater.cc:461] Registering TS with master...
I20251027 21:07:46.764425 9971 heartbeater.cc:461] Registering TS with master...
I20251027 21:07:46.765419 9970 heartbeater.cc:507] Master 127.9.14.124:45071 requested a full tablet report, sending...
I20251027 21:07:46.765446 9972 heartbeater.cc:507] Master 127.9.14.125:46313 requested a full tablet report, sending...
I20251027 21:07:46.765506 9971 heartbeater.cc:507] Master 127.9.14.126:34057 requested a full tablet report, sending...
I20251027 21:07:46.767079 9383 ts_manager.cc:194] Registered new tserver with Master: 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:46.767241 9312 ts_manager.cc:194] Registered new tserver with Master: 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:46.767248 9454 ts_manager.cc:194] Registered new tserver with Master: 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:46.768129 9312 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.14.67:53829
I20251027 21:07:46.772272 9273 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251027 21:07:46.801680 9838 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:46.801968 9838 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: This master's current role is: LEADER
I20251027 21:07:46.802030 9821 heartbeater.cc:499] Master 127.9.14.126:34057 was elected leader, sending a full tablet report...
I20251027 21:07:46.807750 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7e1b33b945ca44d48b797020a2167d39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:46.808463 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9a3efebd4fad4f4481cb57884444980b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:46.809401 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e1b33b945ca44d48b797020a2167d39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:46.807770 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_90ab0139d6ba451180b656d8c2375772"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:46.809624 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_8bb3dcc0f76241e2bb66fa54dc5d2efd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:46.810053 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90ab0139d6ba451180b656d8c2375772 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:46.810235 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8bb3dcc0f76241e2bb66fa54dc5d2efd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:46.810940 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4338d9c5400c4fda8382a52f5751bb1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:46.811223 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_43ee9fe687314dd3881426382e00d764"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:46.811717 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4338d9c5400c4fda8382a52f5751bb1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:46.811862 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43ee9fe687314dd3881426382e00d764 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:46.812623 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_dc75925f762d4a23848912fc7dfe433b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:46.813742 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc75925f762d4a23848912fc7dfe433b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:46.814901 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a3efebd4fad4f4481cb57884444980b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:46.815531 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6b10a0dc13ed42069afa442d910159c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:46.816232 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b10a0dc13ed42069afa442d910159c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:46.818379 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_64b119e025bc400f84830380dad02c82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:46.819500 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5db78f4dac4340909b3b715b71937e65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:46.820144 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5db78f4dac4340909b3b715b71937e65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:46.819181 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64b119e025bc400f84830380dad02c82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:46.836608 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_90ab0139d6ba451180b656d8c2375772" } new_table_name: "default.Table_d233023476bd4092bc445f39c7dd0ec6" modify_external_catalogs: true
I20251027 21:07:46.869597 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6b10a0dc13ed42069afa442d910159c2" } new_table_name: "default.Table_6dd476b02add42ac91e9c5943bb3978b" modify_external_catalogs: true
I20251027 21:07:46.870985 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5db78f4dac4340909b3b715b71937e65" } new_table_name: "default.Table_40644b4bc09f4693866503d3b5fb7df7" modify_external_catalogs: true
I20251027 21:07:46.871228 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_dc75925f762d4a23848912fc7dfe433b" } new_table_name: "default.Table_e6bc4379784d49cbae1c20888dc66289" modify_external_catalogs: true
I20251027 21:07:46.872282 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9a3efebd4fad4f4481cb57884444980b" } modify_external_catalogs: true
I20251027 21:07:46.872524 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9a3efebd4fad4f4481cb57884444980b" } modify_external_catalogs: true
I20251027 21:07:46.871269 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_43ee9fe687314dd3881426382e00d764" } new_table_name: "default.Table_324426b580004200b83e6449cea91ce8" modify_external_catalogs: true
I20251027 21:07:46.872304 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4338d9c5400c4fda8382a52f5751bb1c" } modify_external_catalogs: true
I20251027 21:07:46.875692 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4338d9c5400c4fda8382a52f5751bb1c" } modify_external_catalogs: true
I20251027 21:07:46.928300 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7e1b33b945ca44d48b797020a2167d39" } modify_external_catalogs: true
I20251027 21:07:46.928568 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7e1b33b945ca44d48b797020a2167d39" } modify_external_catalogs: true
I20251027 21:07:46.930325 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_64b119e025bc400f84830380dad02c82" } modify_external_catalogs: true
I20251027 21:07:46.930564 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_64b119e025bc400f84830380dad02c82" } modify_external_catalogs: true
I20251027 21:07:46.932674 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8bb3dcc0f76241e2bb66fa54dc5d2efd" } modify_external_catalogs: true
I20251027 21:07:46.932907 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8bb3dcc0f76241e2bb66fa54dc5d2efd" } modify_external_catalogs: true
W20251027 21:07:46.940351 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f5b6f80625f54d818d251a8150578491 has no consensus state
W20251027 21:07:46.941219 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 347ec88f0a314ea9af6a7dfbd42fe305 consensus state has no leader
W20251027 21:07:46.943849 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3adbe5db3442462a8ae64a4b08129d69 has no consensus state
W20251027 21:07:46.945932 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4bcd083a30a44ba898211eb769bc4252 has no consensus state
I20251027 21:07:46.948189 9308 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 52fe153bb91942099609748aef2a3fcc
W20251027 21:07:46.950640 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 91d0f6a5b96e406dac97ba2a77aaf597 has no consensus state
I20251027 21:07:46.952864 9306 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 6ea0334b8a46415b95b5933c0421dbe8
I20251027 21:07:46.955114 9312 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 000565c7b9924874881e43c302812b80
I20251027 21:07:46.958487 9309 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 7bcd374026c94191a3a2a183ea363477
I20251027 21:07:46.964725 9303 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet dbefb124163243978cf730005d370552
W20251027 21:07:46.976158 9296 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 4bcd083a30a44ba898211eb769bc4252 (table default.Table_e6bc4379784d49cbae1c20888dc66289 [id=0b6aecd6b64c4f2e837bf7fe73166ab2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4bcd083a30a44ba898211eb769bc4252 has no consensus state
I20251027 21:07:46.993439 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 347ec88f0a314ea9af6a7dfbd42fe305 (DEFAULT_TABLE table=default.Table_90ab0139d6ba451180b656d8c2375772 [id=7c1a1f82393b4e658ddb7b05bee043e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:46.994990 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 347ec88f0a314ea9af6a7dfbd42fe305. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:46.999294 9751 tablet_service.cc:1505] Processing CreateTablet for tablet 91d0f6a5b96e406dac97ba2a77aaf597 (DEFAULT_TABLE table=default.Table_324426b580004200b83e6449cea91ce8 [id=f28dfce7eed645eaa5f7465a9fc66668]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:46.998955 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 4bcd083a30a44ba898211eb769bc4252 (DEFAULT_TABLE table=default.Table_e6bc4379784d49cbae1c20888dc66289 [id=0b6aecd6b64c4f2e837bf7fe73166ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.000696 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bcd083a30a44ba898211eb769bc4252. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.001879 9613 tablet_service.cc:1505] Processing CreateTablet for tablet 3adbe5db3442462a8ae64a4b08129d69 (DEFAULT_TABLE table=default.Table_40644b4bc09f4693866503d3b5fb7df7 [id=09c88f2f898844b88b70f5ed9e837841]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.002413 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3adbe5db3442462a8ae64a4b08129d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.002688 9752 tablet_service.cc:1505] Processing CreateTablet for tablet f5b6f80625f54d818d251a8150578491 (DEFAULT_TABLE table=default.Table_6dd476b02add42ac91e9c5943bb3978b [id=117c8f5066e149db8482c2601b040e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.003115 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5b6f80625f54d818d251a8150578491. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.003358 9612 tablet_service.cc:1505] Processing CreateTablet for tablet f5b6f80625f54d818d251a8150578491 (DEFAULT_TABLE table=default.Table_6dd476b02add42ac91e9c5943bb3978b [id=117c8f5066e149db8482c2601b040e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.003788 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5b6f80625f54d818d251a8150578491. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.006484 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 91d0f6a5b96e406dac97ba2a77aaf597 (DEFAULT_TABLE table=default.Table_324426b580004200b83e6449cea91ce8 [id=f28dfce7eed645eaa5f7465a9fc66668]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.006772 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 347ec88f0a314ea9af6a7dfbd42fe305 (DEFAULT_TABLE table=default.Table_90ab0139d6ba451180b656d8c2375772 [id=7c1a1f82393b4e658ddb7b05bee043e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.006985 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91d0f6a5b96e406dac97ba2a77aaf597. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.007159 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 347ec88f0a314ea9af6a7dfbd42fe305. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.010396 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 4bcd083a30a44ba898211eb769bc4252 (DEFAULT_TABLE table=default.Table_e6bc4379784d49cbae1c20888dc66289 [id=0b6aecd6b64c4f2e837bf7fe73166ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.011541 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bcd083a30a44ba898211eb769bc4252. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.013557 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 3adbe5db3442462a8ae64a4b08129d69 (DEFAULT_TABLE table=default.Table_40644b4bc09f4693866503d3b5fb7df7 [id=09c88f2f898844b88b70f5ed9e837841]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.002050 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 4bcd083a30a44ba898211eb769bc4252 (DEFAULT_TABLE table=default.Table_e6bc4379784d49cbae1c20888dc66289 [id=0b6aecd6b64c4f2e837bf7fe73166ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.002522 9902 tablet_service.cc:1505] Processing CreateTablet for tablet f5b6f80625f54d818d251a8150578491 (DEFAULT_TABLE table=default.Table_6dd476b02add42ac91e9c5943bb3978b [id=117c8f5066e149db8482c2601b040e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.014168 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3adbe5db3442462a8ae64a4b08129d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.015120 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5b6f80625f54d818d251a8150578491. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.006489 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 347ec88f0a314ea9af6a7dfbd42fe305 (DEFAULT_TABLE table=default.Table_90ab0139d6ba451180b656d8c2375772 [id=7c1a1f82393b4e658ddb7b05bee043e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.018922 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 347ec88f0a314ea9af6a7dfbd42fe305. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.019418 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91d0f6a5b96e406dac97ba2a77aaf597. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.006352 9903 tablet_service.cc:1505] Processing CreateTablet for tablet 3adbe5db3442462a8ae64a4b08129d69 (DEFAULT_TABLE table=default.Table_40644b4bc09f4693866503d3b5fb7df7 [id=09c88f2f898844b88b70f5ed9e837841]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.020092 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3adbe5db3442462a8ae64a4b08129d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.020985 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bcd083a30a44ba898211eb769bc4252. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.021126 10029 tablet_bootstrap.cc:492] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:47.006693 9901 tablet_service.cc:1505] Processing CreateTablet for tablet 91d0f6a5b96e406dac97ba2a77aaf597 (DEFAULT_TABLE table=default.Table_324426b580004200b83e6449cea91ce8 [id=f28dfce7eed645eaa5f7465a9fc66668]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.021916 9901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91d0f6a5b96e406dac97ba2a77aaf597. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.025619 10029 tablet_bootstrap.cc:654] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.027002 10029 log.cc:826] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14: Log is configured to *not* fsync() on all Append() calls
I20251027 21:07:47.032845 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_80070541c25d4fcf9f6a06e4e2d67e4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.033514 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80070541c25d4fcf9f6a06e4e2d67e4f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.044952 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_ecfc7ac169bd4561b05a7f24315b4ea5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.045545 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecfc7ac169bd4561b05a7f24315b4ea5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.046967 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3fa43de68faa4fdb8ff9761c02c8a7b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.047461 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fa43de68faa4fdb8ff9761c02c8a7b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.055915 10029 tablet_bootstrap.cc:492] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:47.058964 9750 tablet_service.cc:1505] Processing CreateTablet for tablet bc2b7a771a1647c290ed9c3e394c60fe (DEFAULT_TABLE table=default.Table_80070541c25d4fcf9f6a06e4e2d67e4f [id=5280a5a662394d82b01929699833a1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.059212 9900 tablet_service.cc:1505] Processing CreateTablet for tablet bc2b7a771a1647c290ed9c3e394c60fe (DEFAULT_TABLE table=default.Table_80070541c25d4fcf9f6a06e4e2d67e4f [id=5280a5a662394d82b01929699833a1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.059669 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc2b7a771a1647c290ed9c3e394c60fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.059779 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc2b7a771a1647c290ed9c3e394c60fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.069013 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d3ceec0bb5c34e09804764c3f8d72cc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.069583 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3ceec0bb5c34e09804764c3f8d72cc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.070773 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_580fa0e190ef44df94c80af9a4303990"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.071272 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_580fa0e190ef44df94c80af9a4303990 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.072803 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_41a04598f12247fbac5f4103d2cbf22d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.073264 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41a04598f12247fbac5f4103d2cbf22d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.074334 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b0ffce0039984ea1a0fcf74346d3cc76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.074785 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0ffce0039984ea1a0fcf74346d3cc76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.074762 9611 tablet_service.cc:1505] Processing CreateTablet for tablet bc2b7a771a1647c290ed9c3e394c60fe (DEFAULT_TABLE table=default.Table_80070541c25d4fcf9f6a06e4e2d67e4f [id=5280a5a662394d82b01929699833a1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.075419 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc2b7a771a1647c290ed9c3e394c60fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.075829 9308 master_service.cc:954] ReplaceTablet: received request to replace tablet 3adbe5db3442462a8ae64a4b08129d69 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:47.080240 9310 master_service.cc:954] ReplaceTablet: received request to replace tablet f5b6f80625f54d818d251a8150578491 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:47.080503 9305 master_service.cc:954] ReplaceTablet: received request to replace tablet 3adbe5db3442462a8ae64a4b08129d69 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:47.082361 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet 3adbe5db3442462a8ae64a4b08129d69 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:47.082569 9304 master_service.cc:954] ReplaceTablet: received request to replace tablet 347ec88f0a314ea9af6a7dfbd42fe305 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:47.084256 10032 tablet_bootstrap.cc:492] T 91d0f6a5b96e406dac97ba2a77aaf597 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:47.086211 10031 tablet_bootstrap.cc:492] T f5b6f80625f54d818d251a8150578491 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:47.088085 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0c05919c1a104864a4e1cd182ae59e88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.088644 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c05919c1a104864a4e1cd182ae59e88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.092219 10029 ts_tablet_manager.cc:1403] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.071s user 0.011s sys 0.004s
I20251027 21:07:47.092285 10032 tablet_bootstrap.cc:654] T 91d0f6a5b96e406dac97ba2a77aaf597 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.093577 10032 log.cc:826] T 91d0f6a5b96e406dac97ba2a77aaf597 P 0fbd567f929a4f6d8e4d78feef7ed678: Log is configured to *not* fsync() on all Append() calls
I20251027 21:07:47.098227 9752 tablet_service.cc:1505] Processing CreateTablet for tablet a4c022969ea448cf933d5220f7ae8f66 (DEFAULT_TABLE table=default.Table_3fa43de68faa4fdb8ff9761c02c8a7b5 [id=98143d42942742568f2c3a1d0fbcc100]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.098686 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c022969ea448cf933d5220f7ae8f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.103552 9900 tablet_service.cc:1505] Processing CreateTablet for tablet a4c022969ea448cf933d5220f7ae8f66 (DEFAULT_TABLE table=default.Table_3fa43de68faa4fdb8ff9761c02c8a7b5 [id=98143d42942742568f2c3a1d0fbcc100]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.103924 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c022969ea448cf933d5220f7ae8f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.107297 10031 tablet_bootstrap.cc:654] T f5b6f80625f54d818d251a8150578491 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.108547 10031 log.cc:826] T f5b6f80625f54d818d251a8150578491 P 77b0513f17c0472088901a1ce8b08218: Log is configured to *not* fsync() on all Append() calls
I20251027 21:07:47.114183 9612 tablet_service.cc:1505] Processing CreateTablet for tablet a4c022969ea448cf933d5220f7ae8f66 (DEFAULT_TABLE table=default.Table_3fa43de68faa4fdb8ff9761c02c8a7b5 [id=98143d42942742568f2c3a1d0fbcc100]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.114566 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c022969ea448cf933d5220f7ae8f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.114820 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_40426ce1c83642a7bb5432c7590d54f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.115463 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40426ce1c83642a7bb5432c7590d54f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.116357 10031 tablet_bootstrap.cc:492] T f5b6f80625f54d818d251a8150578491 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:47.116618 10031 ts_tablet_manager.cc:1403] T f5b6f80625f54d818d251a8150578491 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.007s
I20251027 21:07:47.119509 9273 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu pid 9420
I20251027 21:07:47.120703 9484 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 21:07:47.120777 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_39e8cb18c67c485a80272c4ed5011362"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.121232 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39e8cb18c67c485a80272c4ed5011362 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.128046 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 347ec88f0a314ea9af6a7dfbd42fe305
I20251027 21:07:47.128770 9304 catalog_manager.cc:6719] ReplaceTablet: tablet 347ec88f0a314ea9af6a7dfbd42fe305 deleted and replaced by tablet a52f41629a2744ef8db0d57e06f8e0a4
I20251027 21:07:47.128749 10029 raft_consensus.cc:359] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.129279 10029 raft_consensus.cc:385] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.129434 10029 raft_consensus.cc:740] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.130090 10029 consensus_queue.cc:260] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.130404 9754 tablet_service.cc:1552] Processing DeleteTablet for tablet 347ec88f0a314ea9af6a7dfbd42fe305 with delete_type TABLET_DATA_DELETED (replaced by tablet a52f41629a2744ef8db0d57e06f8e0a4) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.133919 9900 tablet_service.cc:1552] Processing DeleteTablet for tablet 347ec88f0a314ea9af6a7dfbd42fe305 with delete_type TABLET_DATA_DELETED (replaced by tablet a52f41629a2744ef8db0d57e06f8e0a4) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.137835 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet f5b6f80625f54d818d251a8150578491
I20251027 21:07:47.138859 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 3adbe5db3442462a8ae64a4b08129d69
I20251027 21:07:47.139319 9308 catalog_manager.cc:6719] ReplaceTablet: tablet 3adbe5db3442462a8ae64a4b08129d69 deleted and replaced by tablet 3218f18a797c40c9ad6ac64b2670cd19
I20251027 21:07:47.140204 9901 tablet_service.cc:1552] Processing DeleteTablet for tablet 3adbe5db3442462a8ae64a4b08129d69 with delete_type TABLET_DATA_DELETED (replaced by tablet 3218f18a797c40c9ad6ac64b2670cd19) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.140522 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 3adbe5db3442462a8ae64a4b08129d69 with delete_type TABLET_DATA_DELETED (replaced by tablet 3218f18a797c40c9ad6ac64b2670cd19) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.140915 9310 catalog_manager.cc:6719] ReplaceTablet: tablet f5b6f80625f54d818d251a8150578491 deleted and replaced by tablet 783d845009d54d20b8e6626c3e73d2a7
I20251027 21:07:47.141748 9901 tablet_service.cc:1552] Processing DeleteTablet for tablet f5b6f80625f54d818d251a8150578491 with delete_type TABLET_DATA_DELETED (replaced by tablet 783d845009d54d20b8e6626c3e73d2a7) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.141592 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet f5b6f80625f54d818d251a8150578491 with delete_type TABLET_DATA_DELETED (replaced by tablet 783d845009d54d20b8e6626c3e73d2a7) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.142343 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 3adbe5db3442462a8ae64a4b08129d69 with delete_type TABLET_DATA_DELETED (replaced by tablet 3218f18a797c40c9ad6ac64b2670cd19) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.142185 9611 tablet_service.cc:1552] Processing DeleteTablet for tablet f5b6f80625f54d818d251a8150578491 with delete_type TABLET_DATA_DELETED (replaced by tablet 783d845009d54d20b8e6626c3e73d2a7) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.145988 10032 tablet_bootstrap.cc:492] T 91d0f6a5b96e406dac97ba2a77aaf597 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:47.146298 10032 ts_tablet_manager.cc:1403] T 91d0f6a5b96e406dac97ba2a77aaf597 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.062s user 0.002s sys 0.007s
I20251027 21:07:47.155766 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d3ceec0bb5c34e09804764c3f8d72cc6" } new_table_name: "default.Table_4c5433970bd64291a2b1289a84f35590" modify_external_catalogs: true
I20251027 21:07:47.156415 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e6bc4379784d49cbae1c20888dc66289" } modify_external_catalogs: true
I20251027 21:07:47.156633 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e6bc4379784d49cbae1c20888dc66289" } modify_external_catalogs: true
I20251027 21:07:47.156679 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_580fa0e190ef44df94c80af9a4303990" } new_table_name: "default.Table_1c94591b5a734f7a8ce72ed245351b55" modify_external_catalogs: true
I20251027 21:07:47.157783 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3fa43de68faa4fdb8ff9761c02c8a7b5" } new_table_name: "default.Table_dfe6a2fa4dd545dd95c2f1d8bc0e5d80" modify_external_catalogs: true
I20251027 21:07:47.160244 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6dd476b02add42ac91e9c5943bb3978b" } new_table_name: "default.Table_53596f51be6a408a8987da9e37e4d779" modify_external_catalogs: true
I20251027 21:07:47.161006 10044 raft_consensus.cc:493] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:47.161094 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_39e8cb18c67c485a80272c4ed5011362" } modify_external_catalogs: true
I20251027 21:07:47.161262 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_39e8cb18c67c485a80272c4ed5011362" } modify_external_catalogs: true
I20251027 21:07:47.161211 10044 raft_consensus.cc:515] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.161338 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_80070541c25d4fcf9f6a06e4e2d67e4f" } new_table_name: "default.Table_e46f2bc398e145209af5db2546ad90f7" modify_external_catalogs: true
I20251027 21:07:47.162559 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_40426ce1c83642a7bb5432c7590d54f0" } modify_external_catalogs: true
I20251027 21:07:47.162715 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_40426ce1c83642a7bb5432c7590d54f0" } modify_external_catalogs: true
I20251027 21:07:47.163239 10044 leader_election.cc:290] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:47.133999 9612 tablet_service.cc:1552] Processing DeleteTablet for tablet 347ec88f0a314ea9af6a7dfbd42fe305 with delete_type TABLET_DATA_DELETED (replaced by tablet a52f41629a2744ef8db0d57e06f8e0a4) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:47.169334 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 347ec88f0a314ea9af6a7dfbd42fe305 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 347ec88f0a314ea9af6a7dfbd42fe305 already in progress: creating tablet
W20251027 21:07:47.169411 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 988052533a1042d9bba5ed5711d09d9c has no consensus state
W20251027 21:07:47.170946 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a878184a3a444761aff8730bdf43ccaf has no consensus state
W20251027 21:07:47.175693 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 3adbe5db3442462a8ae64a4b08129d69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3adbe5db3442462a8ae64a4b08129d69 already in progress: creating tablet
W20251027 21:07:47.176440 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet f5b6f80625f54d818d251a8150578491 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5b6f80625f54d818d251a8150578491 already in progress: creating tablet
W20251027 21:07:47.181669 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 347ec88f0a314ea9af6a7dfbd42fe305 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 347ec88f0a314ea9af6a7dfbd42fe305 already in progress: creating tablet
I20251027 21:07:47.184029 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_324426b580004200b83e6449cea91ce8" } modify_external_catalogs: true
I20251027 21:07:47.184257 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_324426b580004200b83e6449cea91ce8" } modify_external_catalogs: true
I20251027 21:07:47.185039 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d233023476bd4092bc445f39c7dd0ec6" } modify_external_catalogs: true
I20251027 21:07:47.185241 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d233023476bd4092bc445f39c7dd0ec6" } modify_external_catalogs: true
W20251027 21:07:47.186765 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 3adbe5db3442462a8ae64a4b08129d69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3adbe5db3442462a8ae64a4b08129d69 already in progress: creating tablet
W20251027 21:07:47.187387 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet f5b6f80625f54d818d251a8150578491 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5b6f80625f54d818d251a8150578491 already in progress: creating tablet
W20251027 21:07:47.188738 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 3adbe5db3442462a8ae64a4b08129d69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3adbe5db3442462a8ae64a4b08129d69 already in progress: creating tablet
W20251027 21:07:47.189091 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet f5b6f80625f54d818d251a8150578491 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5b6f80625f54d818d251a8150578491 already in progress: creating tablet
W20251027 21:07:47.189431 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 347ec88f0a314ea9af6a7dfbd42fe305 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 347ec88f0a314ea9af6a7dfbd42fe305 already in progress: creating tablet
I20251027 21:07:47.196529 10029 ts_tablet_manager.cc:1434] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.104s user 0.011s sys 0.026s
I20251027 21:07:47.197026 10029 tablet_bootstrap.cc:492] T 347ec88f0a314ea9af6a7dfbd42fe305 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:47.198848 10029 tablet_bootstrap.cc:654] T 347ec88f0a314ea9af6a7dfbd42fe305 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:47.199985 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4c022969ea448cf933d5220f7ae8f66 consensus state has no leader
I20251027 21:07:47.200234 9307 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet cc84692bb6c14e61a483f97cb8c54999
W20251027 21:07:47.200446 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 783d845009d54d20b8e6626c3e73d2a7 has no consensus state
W20251027 21:07:47.200639 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bc2b7a771a1647c290ed9c3e394c60fe consensus state has no leader
I20251027 21:07:47.196686 10031 raft_consensus.cc:359] T f5b6f80625f54d818d251a8150578491 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.201882 10031 raft_consensus.cc:385] T f5b6f80625f54d818d251a8150578491 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.201984 10031 raft_consensus.cc:740] T f5b6f80625f54d818d251a8150578491 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.202544 10032 raft_consensus.cc:359] T 91d0f6a5b96e406dac97ba2a77aaf597 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.202936 10032 raft_consensus.cc:385] T 91d0f6a5b96e406dac97ba2a77aaf597 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.203040 10032 raft_consensus.cc:740] T 91d0f6a5b96e406dac97ba2a77aaf597 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.207867 10031 consensus_queue.cc:260] T f5b6f80625f54d818d251a8150578491 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.208137 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3adbe5db3442462a8ae64a4b08129d69" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:47.208695 10032 consensus_queue.cc:260] T 91d0f6a5b96e406dac97ba2a77aaf597 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:47.210604 9568 leader_election.cc:343] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:47.210870 9304 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 8f9d9712432e44bfb3df0e6a1243f7d7
I20251027 21:07:47.219430 10029 tablet_bootstrap.cc:492] T 347ec88f0a314ea9af6a7dfbd42fe305 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:47.219591 10029 ts_tablet_manager.cc:1403] T 347ec88f0a314ea9af6a7dfbd42fe305 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.018s
I20251027 21:07:47.220669 10029 raft_consensus.cc:359] T 347ec88f0a314ea9af6a7dfbd42fe305 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:47.220897 10029 raft_consensus.cc:385] T 347ec88f0a314ea9af6a7dfbd42fe305 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.220995 10029 raft_consensus.cc:740] T 347ec88f0a314ea9af6a7dfbd42fe305 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.221249 10029 consensus_queue.cc:260] T 347ec88f0a314ea9af6a7dfbd42fe305 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:47.221771 10029 ts_tablet_manager.cc:1434] T 347ec88f0a314ea9af6a7dfbd42fe305 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:07:47.222057 10029 tablet_bootstrap.cc:492] T 4bcd083a30a44ba898211eb769bc4252 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:47.221944 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3adbe5db3442462a8ae64a4b08129d69" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:47.223744 10029 tablet_bootstrap.cc:654] T 4bcd083a30a44ba898211eb769bc4252 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.225266 9312 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet a52f41629a2744ef8db0d57e06f8e0a4
I20251027 21:07:47.227262 9750 tablet_service.cc:1505] Processing CreateTablet for tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b (DEFAULT_TABLE table=default.Table_ecfc7ac169bd4561b05a7f24315b4ea5 [id=7a31bdff8eab4483b0d88c59e7a2e914]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.227547 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 0cd4f5fa03d4417383df279da475345f (DEFAULT_TABLE table=default.Table_b0ffce0039984ea1a0fcf74346d3cc76 [id=05edeb649c544c2f899ed42a87cef2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.227731 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.227849 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 0cd4f5fa03d4417383df279da475345f (DEFAULT_TABLE table=default.Table_b0ffce0039984ea1a0fcf74346d3cc76 [id=05edeb649c544c2f899ed42a87cef2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.228009 9752 tablet_service.cc:1505] Processing CreateTablet for tablet bd6230765d6448498132e24482100b39 (DEFAULT_TABLE table=default.Table_41a04598f12247fbac5f4103d2cbf22d [id=2e28a4d0f9204351b93335736a9abe18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.228185 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cd4f5fa03d4417383df279da475345f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.230427 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 988052533a1042d9bba5ed5711d09d9c (DEFAULT_TABLE table=default.Table_4c5433970bd64291a2b1289a84f35590 [id=a1b2d46db3764c7f97712a0c5f891881]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.230417 9612 tablet_service.cc:1505] Processing CreateTablet for tablet bd6230765d6448498132e24482100b39 (DEFAULT_TABLE table=default.Table_41a04598f12247fbac5f4103d2cbf22d [id=2e28a4d0f9204351b93335736a9abe18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.230775 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6230765d6448498132e24482100b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.230839 9751 tablet_service.cc:1505] Processing CreateTablet for tablet a878184a3a444761aff8730bdf43ccaf (DEFAULT_TABLE table=default.Table_1c94591b5a734f7a8ce72ed245351b55 [id=77687d6093f741d3905c660d0650ae13]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:47.230930 9566 leader_election.cc:343] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:47.231058 9566 leader_election.cc:304] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:47.231786 10039 raft_consensus.cc:2749] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:47.232640 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cd4f5fa03d4417383df279da475345f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.233093 10032 ts_tablet_manager.cc:1434] T 91d0f6a5b96e406dac97ba2a77aaf597 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.087s user 0.009s sys 0.006s
I20251027 21:07:47.233481 10032 tablet_bootstrap.cc:492] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:47.234382 9611 tablet_service.cc:1505] Processing CreateTablet for tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b (DEFAULT_TABLE table=default.Table_ecfc7ac169bd4561b05a7f24315b4ea5 [id=7a31bdff8eab4483b0d88c59e7a2e914]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.234743 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.235183 10031 ts_tablet_manager.cc:1434] T f5b6f80625f54d818d251a8150578491 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.118s user 0.014s sys 0.004s
I20251027 21:07:47.235555 10031 tablet_bootstrap.cc:492] T 3adbe5db3442462a8ae64a4b08129d69 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:47.235666 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6230765d6448498132e24482100b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.236616 10032 tablet_bootstrap.cc:654] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.238343 9971 heartbeater.cc:499] Master 127.9.14.126:34057 was elected leader, sending a full tablet report...
I20251027 21:07:47.239394 9900 tablet_service.cc:1505] Processing CreateTablet for tablet 0cd4f5fa03d4417383df279da475345f (DEFAULT_TABLE table=default.Table_b0ffce0039984ea1a0fcf74346d3cc76 [id=05edeb649c544c2f899ed42a87cef2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.239756 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cd4f5fa03d4417383df279da475345f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.240037 10031 tablet_bootstrap.cc:654] T 3adbe5db3442462a8ae64a4b08129d69 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.240875 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 988052533a1042d9bba5ed5711d09d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.242434 9901 tablet_service.cc:1505] Processing CreateTablet for tablet bd6230765d6448498132e24482100b39 (DEFAULT_TABLE table=default.Table_41a04598f12247fbac5f4103d2cbf22d [id=2e28a4d0f9204351b93335736a9abe18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.242758 9901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6230765d6448498132e24482100b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.246346 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a878184a3a444761aff8730bdf43ccaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.246626 9905 tablet_service.cc:1505] Processing CreateTablet for tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b (DEFAULT_TABLE table=default.Table_ecfc7ac169bd4561b05a7f24315b4ea5 [id=7a31bdff8eab4483b0d88c59e7a2e914]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.246930 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.250425 9903 tablet_service.cc:1505] Processing CreateTablet for tablet 988052533a1042d9bba5ed5711d09d9c (DEFAULT_TABLE table=default.Table_4c5433970bd64291a2b1289a84f35590 [id=a1b2d46db3764c7f97712a0c5f891881]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.250743 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 988052533a1042d9bba5ed5711d09d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.250751 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9cac798aeef2436f9a044ec54e72daef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.251410 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cac798aeef2436f9a044ec54e72daef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:47.254242 9973 tablet.cc:2378] T 91d0f6a5b96e406dac97ba2a77aaf597 P 0fbd567f929a4f6d8e4d78feef7ed678: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 21:07:47.254809 9902 tablet_service.cc:1505] Processing CreateTablet for tablet a878184a3a444761aff8730bdf43ccaf (DEFAULT_TABLE table=default.Table_1c94591b5a734f7a8ce72ed245351b55 [id=77687d6093f741d3905c660d0650ae13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.255136 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a878184a3a444761aff8730bdf43ccaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.257929 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet f5b6f80625f54d818d251a8150578491 with delete_type TABLET_DATA_DELETED (replaced by tablet 783d845009d54d20b8e6626c3e73d2a7) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.258200 10040 tablet_replica.cc:333] T f5b6f80625f54d818d251a8150578491 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:47.258643 10040 raft_consensus.cc:2243] T f5b6f80625f54d818d251a8150578491 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.259020 10040 raft_consensus.cc:2272] T f5b6f80625f54d818d251a8150578491 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:47.260560 10040 ts_tablet_manager.cc:1916] T f5b6f80625f54d818d251a8150578491 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:47.269204 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_bd8cb2021a524172b1db94022b230275"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.269697 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd8cb2021a524172b1db94022b230275 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.270941 10040 ts_tablet_manager.cc:1929] T f5b6f80625f54d818d251a8150578491 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:47.271085 10040 log.cc:1199] T f5b6f80625f54d818d251a8150578491 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/f5b6f80625f54d818d251a8150578491
I20251027 21:07:47.271483 10040 ts_tablet_manager.cc:1950] T f5b6f80625f54d818d251a8150578491 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:47.272993 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet f5b6f80625f54d818d251a8150578491 (table default.Table_53596f51be6a408a8987da9e37e4d779 [id=117c8f5066e149db8482c2601b040e17]) successfully deleted
I20251027 21:07:47.275254 9613 tablet_service.cc:1505] Processing CreateTablet for tablet 988052533a1042d9bba5ed5711d09d9c (DEFAULT_TABLE table=default.Table_4c5433970bd64291a2b1289a84f35590 [id=a1b2d46db3764c7f97712a0c5f891881]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.275681 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 988052533a1042d9bba5ed5711d09d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.280247 9615 tablet_service.cc:1505] Processing CreateTablet for tablet a878184a3a444761aff8730bdf43ccaf (DEFAULT_TABLE table=default.Table_1c94591b5a734f7a8ce72ed245351b55 [id=77687d6093f741d3905c660d0650ae13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.280652 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a878184a3a444761aff8730bdf43ccaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.287092 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 347ec88f0a314ea9af6a7dfbd42fe305 with delete_type TABLET_DATA_DELETED (replaced by tablet a52f41629a2744ef8db0d57e06f8e0a4) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.287814 10043 tablet_replica.cc:333] T 347ec88f0a314ea9af6a7dfbd42fe305 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:47.287829 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d7a6b578109241ccb05e5ffc8df713b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.288429 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7a6b578109241ccb05e5ffc8df713b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.288503 10043 raft_consensus.cc:2243] T 347ec88f0a314ea9af6a7dfbd42fe305 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.287503 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet 3adbe5db3442462a8ae64a4b08129d69 with delete_type TABLET_DATA_DELETED (replaced by tablet 3218f18a797c40c9ad6ac64b2670cd19) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.289203 10043 raft_consensus.cc:2272] T 347ec88f0a314ea9af6a7dfbd42fe305 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:47.294184 10031 tablet_bootstrap.cc:492] T 3adbe5db3442462a8ae64a4b08129d69 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:47.294327 10031 ts_tablet_manager.cc:1403] T 3adbe5db3442462a8ae64a4b08129d69 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.021s
I20251027 21:07:47.295050 10031 raft_consensus.cc:359] T 3adbe5db3442462a8ae64a4b08129d69 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.295233 10031 raft_consensus.cc:385] T 3adbe5db3442462a8ae64a4b08129d69 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.295332 10031 raft_consensus.cc:740] T 3adbe5db3442462a8ae64a4b08129d69 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.295593 10031 consensus_queue.cc:260] T 3adbe5db3442462a8ae64a4b08129d69 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.296092 10031 ts_tablet_manager.cc:1434] T 3adbe5db3442462a8ae64a4b08129d69 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 21:07:47.296342 10031 tablet_bootstrap.cc:492] T 91d0f6a5b96e406dac97ba2a77aaf597 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:47.297962 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_25db75d097904faa87a9dfd372d1c7eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.298553 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25db75d097904faa87a9dfd372d1c7eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.299185 10032 tablet_bootstrap.cc:492] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:47.299345 10032 ts_tablet_manager.cc:1403] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.066s user 0.013s sys 0.008s
I20251027 21:07:47.300045 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_450e3d88ee8f4a328418b279a354b9e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:47.300284 10032 raft_consensus.cc:359] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:47.300467 10032 raft_consensus.cc:385] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251027 21:07:47.300550 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_450e3d88ee8f4a328418b279a354b9e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.301196 10032 raft_consensus.cc:740] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.301545 10032 consensus_queue.cc:260] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:47.301926 10043 ts_tablet_manager.cc:1916] T 347ec88f0a314ea9af6a7dfbd42fe305 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:47.302064 10032 ts_tablet_manager.cc:1434] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251027 21:07:47.304910 10032 tablet_bootstrap.cc:492] T 4bcd083a30a44ba898211eb769bc4252 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:47.306420 10032 tablet_bootstrap.cc:654] T 4bcd083a30a44ba898211eb769bc4252 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.309315 10031 tablet_bootstrap.cc:654] T 91d0f6a5b96e406dac97ba2a77aaf597 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.315176 9306 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet a4c022969ea448cf933d5220f7ae8f66 (table default.Table_dfe6a2fa4dd545dd95c2f1d8bc0e5d80 [id=98143d42942742568f2c3a1d0fbcc100]). Expected version 1 got 0
I20251027 21:07:47.315977 9306 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet bc2b7a771a1647c290ed9c3e394c60fe (table default.Table_e46f2bc398e145209af5db2546ad90f7 [id=5280a5a662394d82b01929699833a1dc]). Expected version 1 got 0
I20251027 21:07:47.316545 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6045ed3a70d245f09dfb531db1119c9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.317029 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6045ed3a70d245f09dfb531db1119c9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.318027 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 4bcd083a30a44ba898211eb769bc4252
I20251027 21:07:47.319242 10029 tablet_bootstrap.cc:492] T 4bcd083a30a44ba898211eb769bc4252 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:47.319458 10029 ts_tablet_manager.cc:1403] T 4bcd083a30a44ba898211eb769bc4252 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.097s user 0.005s sys 0.027s
I20251027 21:07:47.320290 10029 raft_consensus.cc:359] T 4bcd083a30a44ba898211eb769bc4252 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.320477 10029 raft_consensus.cc:385] T 4bcd083a30a44ba898211eb769bc4252 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.320593 10029 raft_consensus.cc:740] T 4bcd083a30a44ba898211eb769bc4252 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.320830 10029 consensus_queue.cc:260] T 4bcd083a30a44ba898211eb769bc4252 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.321043 10032 tablet_bootstrap.cc:492] T 4bcd083a30a44ba898211eb769bc4252 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:47.321096 10043 ts_tablet_manager.cc:1929] T 347ec88f0a314ea9af6a7dfbd42fe305 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:47.321198 10032 ts_tablet_manager.cc:1403] T 4bcd083a30a44ba898211eb769bc4252 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20251027 21:07:47.321246 10043 log.cc:1199] T 347ec88f0a314ea9af6a7dfbd42fe305 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/347ec88f0a314ea9af6a7dfbd42fe305
I20251027 21:07:47.321341 10029 ts_tablet_manager.cc:1434] T 4bcd083a30a44ba898211eb769bc4252 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20251027 21:07:47.321694 10043 ts_tablet_manager.cc:1950] T 347ec88f0a314ea9af6a7dfbd42fe305 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:47.321916 10032 raft_consensus.cc:359] T 4bcd083a30a44ba898211eb769bc4252 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.322104 10032 raft_consensus.cc:385] T 4bcd083a30a44ba898211eb769bc4252 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.322219 10032 raft_consensus.cc:740] T 4bcd083a30a44ba898211eb769bc4252 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.322484 10032 consensus_queue.cc:260] T 4bcd083a30a44ba898211eb769bc4252 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.322980 10032 ts_tablet_manager.cc:1434] T 4bcd083a30a44ba898211eb769bc4252 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:47.323295 10032 tablet_bootstrap.cc:492] T f5b6f80625f54d818d251a8150578491 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:47.325063 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 91d0f6a5b96e406dac97ba2a77aaf597
I20251027 21:07:47.326195 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_50cddb609f3043f29b42899ad521fbee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:47.326478 10029 tablet_bootstrap.cc:492] T 91d0f6a5b96e406dac97ba2a77aaf597 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
W20251027 21:07:47.326691 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50cddb609f3043f29b42899ad521fbee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.326884 10032 tablet_bootstrap.cc:654] T f5b6f80625f54d818d251a8150578491 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.328154 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_948cff022cff49c9b8148e67df44ca00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:47.328265 10029 tablet_bootstrap.cc:654] T 91d0f6a5b96e406dac97ba2a77aaf597 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:47.328600 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_948cff022cff49c9b8148e67df44ca00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.333607 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 347ec88f0a314ea9af6a7dfbd42fe305 (table default.Table_d233023476bd4092bc445f39c7dd0ec6 [id=7c1a1f82393b4e658ddb7b05bee043e9]) successfully deleted
I20251027 21:07:47.333916 10043 tablet_replica.cc:333] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:47.334175 10043 raft_consensus.cc:2243] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.334554 10043 raft_consensus.cc:2272] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:47.336172 10050 raft_consensus.cc:493] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:47.336424 10050 raft_consensus.cc:515] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:47.338524 10050 leader_election.cc:290] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:47.341245 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d1fb7f4ef09e4bfdac00a75f752f0873"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.341718 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1fb7f4ef09e4bfdac00a75f752f0873 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.341771 9903 tablet_service.cc:1552] Processing DeleteTablet for tablet 3adbe5db3442462a8ae64a4b08129d69 with delete_type TABLET_DATA_DELETED (replaced by tablet 3218f18a797c40c9ad6ac64b2670cd19) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.342170 9901 tablet_service.cc:1552] Processing DeleteTablet for tablet f5b6f80625f54d818d251a8150578491 with delete_type TABLET_DATA_DELETED (replaced by tablet 783d845009d54d20b8e6626c3e73d2a7) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.341214 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d7caf208b9864991a0e50ced803eb60d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:47.342483 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 91d0f6a5b96e406dac97ba2a77aaf597 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.342764 10041 tablet_replica.cc:333] T 91d0f6a5b96e406dac97ba2a77aaf597 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
W20251027 21:07:47.342842 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7caf208b9864991a0e50ced803eb60d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.343291 10041 raft_consensus.cc:2243] T 91d0f6a5b96e406dac97ba2a77aaf597 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.343667 10041 raft_consensus.cc:2272] T 91d0f6a5b96e406dac97ba2a77aaf597 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:47.343909 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bcd083a30a44ba898211eb769bc4252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:47.343971 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 3adbe5db3442462a8ae64a4b08129d69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3adbe5db3442462a8ae64a4b08129d69 already in progress: creating tablet
I20251027 21:07:47.344218 9900 tablet_service.cc:1552] Processing DeleteTablet for tablet 347ec88f0a314ea9af6a7dfbd42fe305 with delete_type TABLET_DATA_DELETED (replaced by tablet a52f41629a2744ef8db0d57e06f8e0a4) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:47.344237 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet f5b6f80625f54d818d251a8150578491 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5b6f80625f54d818d251a8150578491 already in progress: creating tablet
I20251027 21:07:47.344528 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bcd083a30a44ba898211eb769bc4252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.345577 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet 91d0f6a5b96e406dac97ba2a77aaf597 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.346073 10041 ts_tablet_manager.cc:1916] T 91d0f6a5b96e406dac97ba2a77aaf597 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:47.346526 9754 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bcd083a30a44ba898211eb769bc4252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.346786 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 3adbe5db3442462a8ae64a4b08129d69 with delete_type TABLET_DATA_DELETED (replaced by tablet 3218f18a797c40c9ad6ac64b2670cd19) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.347009 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 91d0f6a5b96e406dac97ba2a77aaf597 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.348244 10040 tablet_replica.cc:333] T 3adbe5db3442462a8ae64a4b08129d69 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:47.348461 10040 raft_consensus.cc:2243] T 3adbe5db3442462a8ae64a4b08129d69 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.348631 10040 raft_consensus.cc:2272] T 3adbe5db3442462a8ae64a4b08129d69 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:07:47.349439 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 4bcd083a30a44ba898211eb769bc4252 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4bcd083a30a44ba898211eb769bc4252 already in progress: creating tablet
I20251027 21:07:47.351343 10043 ts_tablet_manager.cc:1916] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:47.354130 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet bd6230765d6448498132e24482100b39 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:47.354943 10040 ts_tablet_manager.cc:1916] T 3adbe5db3442462a8ae64a4b08129d69 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:47.356215 10029 tablet_bootstrap.cc:492] T 91d0f6a5b96e406dac97ba2a77aaf597 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:47.356215 10041 ts_tablet_manager.cc:1929] T 91d0f6a5b96e406dac97ba2a77aaf597 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:47.356359 10029 ts_tablet_manager.cc:1403] T 91d0f6a5b96e406dac97ba2a77aaf597 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.003s
I20251027 21:07:47.357052 10041 log.cc:1199] T 91d0f6a5b96e406dac97ba2a77aaf597 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/91d0f6a5b96e406dac97ba2a77aaf597
I20251027 21:07:47.357194 10029 raft_consensus.cc:359] T 91d0f6a5b96e406dac97ba2a77aaf597 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.357370 10029 raft_consensus.cc:385] T 91d0f6a5b96e406dac97ba2a77aaf597 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.357468 10029 raft_consensus.cc:740] T 91d0f6a5b96e406dac97ba2a77aaf597 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.357743 10041 ts_tablet_manager.cc:1950] T 91d0f6a5b96e406dac97ba2a77aaf597 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:47.357892 10029 consensus_queue.cc:260] T 91d0f6a5b96e406dac97ba2a77aaf597 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.359321 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 91d0f6a5b96e406dac97ba2a77aaf597 (table default.Table_324426b580004200b83e6449cea91ce8 [id=f28dfce7eed645eaa5f7465a9fc66668]) successfully deleted
I20251027 21:07:47.360270 10029 ts_tablet_manager.cc:1434] T 91d0f6a5b96e406dac97ba2a77aaf597 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20251027 21:07:47.362013 10041 tablet_replica.cc:333] T 4bcd083a30a44ba898211eb769bc4252 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:47.362210 10041 raft_consensus.cc:2243] T 4bcd083a30a44ba898211eb769bc4252 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.362356 10029 tablet_bootstrap.cc:492] T f5b6f80625f54d818d251a8150578491 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:47.362371 10041 raft_consensus.cc:2272] T 4bcd083a30a44ba898211eb769bc4252 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:47.363101 9308 master_service.cc:954] ReplaceTablet: received request to replace tablet 91d0f6a5b96e406dac97ba2a77aaf597 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:47.363230 9309 master_service.cc:954] ReplaceTablet: received request to replace tablet 0cd4f5fa03d4417383df279da475345f from {username='slave'} at 127.0.0.1:41710
W20251027 21:07:47.363539 9860 leader_election.cc:343] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 347ec88f0a314ea9af6a7dfbd42fe305
I20251027 21:07:47.364192 10029 tablet_bootstrap.cc:654] T f5b6f80625f54d818d251a8150578491 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.364307 10041 ts_tablet_manager.cc:1916] T 4bcd083a30a44ba898211eb769bc4252 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:47.365967 9900 tablet_service.cc:1505] Processing CreateTablet for tablet 87ec5a6b9b3c425da4e96cc6736d6310 (DEFAULT_TABLE table=default.Table_0c05919c1a104864a4e1cd182ae59e88 [id=16df16d9dc6144cabc7f453c319255d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.366392 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87ec5a6b9b3c425da4e96cc6736d6310. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.366645 10032 tablet_bootstrap.cc:492] T f5b6f80625f54d818d251a8150578491 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:47.366760 10032 ts_tablet_manager.cc:1403] T f5b6f80625f54d818d251a8150578491 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.001s
I20251027 21:07:47.367385 10043 ts_tablet_manager.cc:1929] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:47.367532 10043 log.cc:1199] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/3adbe5db3442462a8ae64a4b08129d69
I20251027 21:07:47.367563 10032 raft_consensus.cc:359] T f5b6f80625f54d818d251a8150578491 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.367743 10032 raft_consensus.cc:385] T f5b6f80625f54d818d251a8150578491 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.367870 10043 ts_tablet_manager.cc:1950] T 3adbe5db3442462a8ae64a4b08129d69 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:47.367846 10032 raft_consensus.cc:740] T f5b6f80625f54d818d251a8150578491 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.368995 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 3adbe5db3442462a8ae64a4b08129d69 (table default.Table_40644b4bc09f4693866503d3b5fb7df7 [id=09c88f2f898844b88b70f5ed9e837841]) successfully deleted
I20251027 21:07:47.370666 9312 master_service.cc:954] ReplaceTablet: received request to replace tablet bc2b7a771a1647c290ed9c3e394c60fe from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:47.371567 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 87ec5a6b9b3c425da4e96cc6736d6310 (DEFAULT_TABLE table=default.Table_0c05919c1a104864a4e1cd182ae59e88 [id=16df16d9dc6144cabc7f453c319255d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.371944 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87ec5a6b9b3c425da4e96cc6736d6310. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.374955 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 3218f18a797c40c9ad6ac64b2670cd19 (DEFAULT_TABLE table=default.Table_40644b4bc09f4693866503d3b5fb7df7 [id=09c88f2f898844b88b70f5ed9e837841]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.375067 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 3218f18a797c40c9ad6ac64b2670cd19 (DEFAULT_TABLE table=default.Table_40644b4bc09f4693866503d3b5fb7df7 [id=09c88f2f898844b88b70f5ed9e837841]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.375119 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b0ffce0039984ea1a0fcf74346d3cc76" } new_table_name: "default.Table_178e5d554faa4e119ac8f52da2d284d7" modify_external_catalogs: true
I20251027 21:07:47.375319 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3218f18a797c40c9ad6ac64b2670cd19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.375427 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3218f18a797c40c9ad6ac64b2670cd19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.377210 10043 tablet_replica.cc:333] T 4bcd083a30a44ba898211eb769bc4252 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:47.377362 10043 raft_consensus.cc:2243] T 4bcd083a30a44ba898211eb769bc4252 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.377578 10043 raft_consensus.cc:2272] T 4bcd083a30a44ba898211eb769bc4252 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:47.378641 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_bd8cb2021a524172b1db94022b230275" } new_table_name: "default.Table_fce1fc7d60384a6b86622ff2afae27b6" modify_external_catalogs: true
I20251027 21:07:47.378720 10029 tablet_bootstrap.cc:492] T f5b6f80625f54d818d251a8150578491 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:47.378649 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 783d845009d54d20b8e6626c3e73d2a7 (DEFAULT_TABLE table=default.Table_53596f51be6a408a8987da9e37e4d779 [id=117c8f5066e149db8482c2601b040e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.378855 10029 ts_tablet_manager.cc:1403] T f5b6f80625f54d818d251a8150578491 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.003s
I20251027 21:07:47.379006 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 783d845009d54d20b8e6626c3e73d2a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.379572 10029 raft_consensus.cc:359] T f5b6f80625f54d818d251a8150578491 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.379806 10029 raft_consensus.cc:385] T f5b6f80625f54d818d251a8150578491 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.379884 10029 raft_consensus.cc:740] T f5b6f80625f54d818d251a8150578491 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.380133 10029 consensus_queue.cc:260] T f5b6f80625f54d818d251a8150578491 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.380193 9612 tablet_service.cc:1505] Processing CreateTablet for tablet 783d845009d54d20b8e6626c3e73d2a7 (DEFAULT_TABLE table=default.Table_53596f51be6a408a8987da9e37e4d779 [id=117c8f5066e149db8482c2601b040e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.380285 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 87ec5a6b9b3c425da4e96cc6736d6310 (DEFAULT_TABLE table=default.Table_0c05919c1a104864a4e1cd182ae59e88 [id=16df16d9dc6144cabc7f453c319255d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.380790 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 3218f18a797c40c9ad6ac64b2670cd19 (DEFAULT_TABLE table=default.Table_40644b4bc09f4693866503d3b5fb7df7 [id=09c88f2f898844b88b70f5ed9e837841]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.381042 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 783d845009d54d20b8e6626c3e73d2a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.381299 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87ec5a6b9b3c425da4e96cc6736d6310. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.381837 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3218f18a797c40c9ad6ac64b2670cd19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.384683 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet f5b6f80625f54d818d251a8150578491 with delete_type TABLET_DATA_DELETED (replaced by tablet 783d845009d54d20b8e6626c3e73d2a7) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.384938 10029 ts_tablet_manager.cc:1434] T f5b6f80625f54d818d251a8150578491 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251027 21:07:47.388953 10029 tablet_bootstrap.cc:492] T bc2b7a771a1647c290ed9c3e394c60fe P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:47.391470 9305 master_service.cc:954] ReplaceTablet: received request to replace tablet bc2b7a771a1647c290ed9c3e394c60fe from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:47.393389 10032 consensus_queue.cc:260] T f5b6f80625f54d818d251a8150578491 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.393893 10032 ts_tablet_manager.cc:1434] T f5b6f80625f54d818d251a8150578491 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20251027 21:07:47.394069 10040 ts_tablet_manager.cc:1929] T 3adbe5db3442462a8ae64a4b08129d69 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:47.394219 10040 log.cc:1199] T 3adbe5db3442462a8ae64a4b08129d69 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/3adbe5db3442462a8ae64a4b08129d69
I20251027 21:07:47.394567 10040 ts_tablet_manager.cc:1950] T 3adbe5db3442462a8ae64a4b08129d69 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:47.395581 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 3adbe5db3442462a8ae64a4b08129d69 (table default.Table_40644b4bc09f4693866503d3b5fb7df7 [id=09c88f2f898844b88b70f5ed9e837841]) successfully deleted
I20251027 21:07:47.395818 10032 tablet_bootstrap.cc:492] T 3adbe5db3442462a8ae64a4b08129d69 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:07:47.396087 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 91d0f6a5b96e406dac97ba2a77aaf597 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91d0f6a5b96e406dac97ba2a77aaf597 already in progress: creating tablet
I20251027 21:07:47.397393 10032 tablet_bootstrap.cc:654] T 3adbe5db3442462a8ae64a4b08129d69 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.397454 10029 tablet_bootstrap.cc:654] T bc2b7a771a1647c290ed9c3e394c60fe P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.399004 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 783d845009d54d20b8e6626c3e73d2a7 (DEFAULT_TABLE table=default.Table_53596f51be6a408a8987da9e37e4d779 [id=117c8f5066e149db8482c2601b040e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.399344 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 783d845009d54d20b8e6626c3e73d2a7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:47.400946 9685 tablet.cc:2378] T 91d0f6a5b96e406dac97ba2a77aaf597 P 51902b6b61ff4d5584ff21896e847e14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 21:07:47.402999 10031 tablet_bootstrap.cc:492] T 91d0f6a5b96e406dac97ba2a77aaf597 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:47.403152 10031 ts_tablet_manager.cc:1403] T 91d0f6a5b96e406dac97ba2a77aaf597 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.107s user 0.010s sys 0.010s
I20251027 21:07:47.403824 10031 raft_consensus.cc:359] T 91d0f6a5b96e406dac97ba2a77aaf597 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.404003 10031 raft_consensus.cc:385] T 91d0f6a5b96e406dac97ba2a77aaf597 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.404067 10031 raft_consensus.cc:740] T 91d0f6a5b96e406dac97ba2a77aaf597 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.404270 10031 consensus_queue.cc:260] T 91d0f6a5b96e406dac97ba2a77aaf597 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.405333 10031 ts_tablet_manager.cc:1434] T 91d0f6a5b96e406dac97ba2a77aaf597 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:47.405616 10031 tablet_bootstrap.cc:492] T 347ec88f0a314ea9af6a7dfbd42fe305 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:47.407045 10043 ts_tablet_manager.cc:1916] T 4bcd083a30a44ba898211eb769bc4252 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:47.407490 10031 tablet_bootstrap.cc:654] T 347ec88f0a314ea9af6a7dfbd42fe305 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.408324 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet f5b6f80625f54d818d251a8150578491 with delete_type TABLET_DATA_DELETED (replaced by tablet 783d845009d54d20b8e6626c3e73d2a7) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.410199 10041 ts_tablet_manager.cc:1929] T 4bcd083a30a44ba898211eb769bc4252 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:47.410344 10041 log.cc:1199] T 4bcd083a30a44ba898211eb769bc4252 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/4bcd083a30a44ba898211eb769bc4252
I20251027 21:07:47.410677 10041 ts_tablet_manager.cc:1950] T 4bcd083a30a44ba898211eb769bc4252 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:47.411770 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 4bcd083a30a44ba898211eb769bc4252 (table default.Table_e6bc4379784d49cbae1c20888dc66289 [id=0b6aecd6b64c4f2e837bf7fe73166ab2]) successfully deleted
I20251027 21:07:47.412231 10041 tablet_replica.cc:333] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:47.412189 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ecfc7ac169bd4561b05a7f24315b4ea5" } new_table_name: "default.Table_0377267e434442dfb2e1cce27b28dd35" modify_external_catalogs: true
I20251027 21:07:47.412398 10041 raft_consensus.cc:2243] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.412549 10041 raft_consensus.cc:2272] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:47.414181 10041 ts_tablet_manager.cc:1916] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:47.422952 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet bd6230765d6448498132e24482100b39
I20251027 21:07:47.423542 9303 catalog_manager.cc:6719] ReplaceTablet: tablet bd6230765d6448498132e24482100b39 deleted and replaced by tablet 84305bcf21b64d2c8c1df015a2e9615c
I20251027 21:07:47.425251 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_40644b4bc09f4693866503d3b5fb7df7" } modify_external_catalogs: true
I20251027 21:07:47.425316 9900 tablet_service.cc:1552] Processing DeleteTablet for tablet bd6230765d6448498132e24482100b39 with delete_type TABLET_DATA_DELETED (replaced by tablet 84305bcf21b64d2c8c1df015a2e9615c) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.425451 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_40644b4bc09f4693866503d3b5fb7df7" } modify_external_catalogs: true
I20251027 21:07:47.427337 10041 ts_tablet_manager.cc:1929] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:47.427459 10041 log.cc:1199] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/347ec88f0a314ea9af6a7dfbd42fe305
I20251027 21:07:47.427827 10041 ts_tablet_manager.cc:1950] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:47.428411 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 347ec88f0a314ea9af6a7dfbd42fe305 (table default.Table_d233023476bd4092bc445f39c7dd0ec6 [id=7c1a1f82393b4e658ddb7b05bee043e9]) successfully deleted
I20251027 21:07:47.428608 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_dfe6a2fa4dd545dd95c2f1d8bc0e5d80" } new_table_name: "default.Table_96482998755b4c28840abc6ccea5d311" modify_external_catalogs: true
I20251027 21:07:47.430284 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e46f2bc398e145209af5db2546ad90f7" } modify_external_catalogs: true
I20251027 21:07:47.430693 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e46f2bc398e145209af5db2546ad90f7" } modify_external_catalogs: true
I20251027 21:07:47.431205 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 0cd4f5fa03d4417383df279da475345f
I20251027 21:07:47.432070 9900 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cd4f5fa03d4417383df279da475345f with delete_type TABLET_DATA_DELETED (replaced by tablet d642da56a67449aa95932180bc5b2c5f) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.432623 10041 tablet_replica.cc:333] T f5b6f80625f54d818d251a8150578491 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:47.432777 10043 ts_tablet_manager.cc:1929] T 4bcd083a30a44ba898211eb769bc4252 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:47.432821 10041 raft_consensus.cc:2243] T f5b6f80625f54d818d251a8150578491 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.432919 10043 log.cc:1199] T 4bcd083a30a44ba898211eb769bc4252 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/4bcd083a30a44ba898211eb769bc4252
I20251027 21:07:47.432976 10041 raft_consensus.cc:2272] T f5b6f80625f54d818d251a8150578491 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:47.433362 10043 ts_tablet_manager.cc:1950] T 4bcd083a30a44ba898211eb769bc4252 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:47.433938 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 91d0f6a5b96e406dac97ba2a77aaf597 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.434212 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cd4f5fa03d4417383df279da475345f with delete_type TABLET_DATA_DELETED (replaced by tablet d642da56a67449aa95932180bc5b2c5f) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.434213 10041 ts_tablet_manager.cc:1916] T f5b6f80625f54d818d251a8150578491 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:47.434463 10040 tablet_replica.cc:333] T 91d0f6a5b96e406dac97ba2a77aaf597 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:47.434617 10040 raft_consensus.cc:2243] T 91d0f6a5b96e406dac97ba2a77aaf597 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.434787 10040 raft_consensus.cc:2272] T 91d0f6a5b96e406dac97ba2a77aaf597 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:47.435518 9612 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cd4f5fa03d4417383df279da475345f with delete_type TABLET_DATA_DELETED (replaced by tablet d642da56a67449aa95932180bc5b2c5f) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.435994 10040 ts_tablet_manager.cc:1916] T 91d0f6a5b96e406dac97ba2a77aaf597 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:47.438026 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet bd6230765d6448498132e24482100b39 with delete_type TABLET_DATA_DELETED (replaced by tablet 84305bcf21b64d2c8c1df015a2e9615c) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.438817 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_25db75d097904faa87a9dfd372d1c7eb" } modify_external_catalogs: true
I20251027 21:07:47.439707 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6045ed3a70d245f09dfb531db1119c9e" } new_table_name: "default.Table_36bfef90af1d48f78b2cce6a78527a44" modify_external_catalogs: true
I20251027 21:07:47.442595 9754 tablet_service.cc:1552] Processing DeleteTablet for tablet bd6230765d6448498132e24482100b39 with delete_type TABLET_DATA_DELETED (replaced by tablet 84305bcf21b64d2c8c1df015a2e9615c) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.443282 10040 ts_tablet_manager.cc:1929] T 91d0f6a5b96e406dac97ba2a77aaf597 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:47.443399 10040 log.cc:1199] T 91d0f6a5b96e406dac97ba2a77aaf597 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/91d0f6a5b96e406dac97ba2a77aaf597
I20251027 21:07:47.443747 10040 ts_tablet_manager.cc:1950] T 91d0f6a5b96e406dac97ba2a77aaf597 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:47.444712 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_25db75d097904faa87a9dfd372d1c7eb" } modify_external_catalogs: true
I20251027 21:07:47.446233 9309 catalog_manager.cc:6719] ReplaceTablet: tablet 0cd4f5fa03d4417383df279da475345f deleted and replaced by tablet d642da56a67449aa95932180bc5b2c5f
I20251027 21:07:47.446408 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet bc2b7a771a1647c290ed9c3e394c60fe
I20251027 21:07:47.446471 10043 tablet_replica.cc:333] T 91d0f6a5b96e406dac97ba2a77aaf597 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:47.446664 10043 raft_consensus.cc:2243] T 91d0f6a5b96e406dac97ba2a77aaf597 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.446807 10043 raft_consensus.cc:2272] T 91d0f6a5b96e406dac97ba2a77aaf597 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:47.446960 9312 catalog_manager.cc:6719] ReplaceTablet: tablet bc2b7a771a1647c290ed9c3e394c60fe deleted and replaced by tablet 59184dbaa912461aa2114b9e19a8d9e4
I20251027 21:07:47.447360 9900 tablet_service.cc:1552] Processing DeleteTablet for tablet bc2b7a771a1647c290ed9c3e394c60fe with delete_type TABLET_DATA_DELETED (replaced by tablet 59184dbaa912461aa2114b9e19a8d9e4) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.448540 10043 ts_tablet_manager.cc:1916] T 91d0f6a5b96e406dac97ba2a77aaf597 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:47.449361 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d642da56a67449aa95932180bc5b2c5f has no consensus state
I20251027 21:07:47.449798 10041 ts_tablet_manager.cc:1929] T f5b6f80625f54d818d251a8150578491 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:47.449927 10041 log.cc:1199] T f5b6f80625f54d818d251a8150578491 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/f5b6f80625f54d818d251a8150578491
I20251027 21:07:47.450241 10041 ts_tablet_manager.cc:1950] T f5b6f80625f54d818d251a8150578491 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:47.450650 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 91d0f6a5b96e406dac97ba2a77aaf597 (table default.Table_324426b580004200b83e6449cea91ce8 [id=f28dfce7eed645eaa5f7465a9fc66668]) successfully deleted
I20251027 21:07:47.451212 9754 tablet_service.cc:1552] Processing DeleteTablet for tablet bc2b7a771a1647c290ed9c3e394c60fe with delete_type TABLET_DATA_DELETED (replaced by tablet 59184dbaa912461aa2114b9e19a8d9e4) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.451568 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 4bcd083a30a44ba898211eb769bc4252 (table default.Table_e6bc4379784d49cbae1c20888dc66289 [id=0b6aecd6b64c4f2e837bf7fe73166ab2]) successfully deleted
W20251027 21:07:47.452611 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9cec6294051b41868c9c7c7fadafa138 has no consensus state
W20251027 21:07:47.452908 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b consensus state has no leader
W20251027 21:07:47.453365 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4c022969ea448cf933d5220f7ae8f66 consensus state has no leader
I20251027 21:07:47.453699 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet f5b6f80625f54d818d251a8150578491 (table default.Table_53596f51be6a408a8987da9e37e4d779 [id=117c8f5066e149db8482c2601b040e17]) successfully deleted
W20251027 21:07:47.454028 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 0cd4f5fa03d4417383df279da475345f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cd4f5fa03d4417383df279da475345f already in progress: creating tablet
W20251027 21:07:47.454337 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet bd6230765d6448498132e24482100b39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd6230765d6448498132e24482100b39 already in progress: creating tablet
W20251027 21:07:47.454998 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet bc2b7a771a1647c290ed9c3e394c60fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc2b7a771a1647c290ed9c3e394c60fe already in progress: creating tablet
W20251027 21:07:47.455251 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet bd6230765d6448498132e24482100b39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd6230765d6448498132e24482100b39 already in progress: creating tablet
W20251027 21:07:47.455652 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 0cd4f5fa03d4417383df279da475345f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cd4f5fa03d4417383df279da475345f already in progress: creating tablet
W20251027 21:07:47.455933 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet bc2b7a771a1647c290ed9c3e394c60fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc2b7a771a1647c290ed9c3e394c60fe already in progress: creating tablet
I20251027 21:07:47.456298 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet bc2b7a771a1647c290ed9c3e394c60fe with delete_type TABLET_DATA_DELETED (replaced by tablet 59184dbaa912461aa2114b9e19a8d9e4) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.457610 9754 tablet_service.cc:1552] Processing DeleteTablet for tablet 91d0f6a5b96e406dac97ba2a77aaf597 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:47.458179 9297 catalog_manager.cc:4952] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 91d0f6a5b96e406dac97ba2a77aaf597 because the tablet was not found. No further retry: Not found: Tablet not found: 91d0f6a5b96e406dac97ba2a77aaf597
I20251027 21:07:47.470760 10043 ts_tablet_manager.cc:1929] T 91d0f6a5b96e406dac97ba2a77aaf597 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:47.470942 10043 log.cc:1199] T 91d0f6a5b96e406dac97ba2a77aaf597 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/91d0f6a5b96e406dac97ba2a77aaf597
I20251027 21:07:47.471338 10043 ts_tablet_manager.cc:1950] T 91d0f6a5b96e406dac97ba2a77aaf597 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:47.473546 9305 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 4a5c8abc9ac64822bd731bd191947f60
I20251027 21:07:47.473930 9310 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 59184dbaa912461aa2114b9e19a8d9e4
I20251027 21:07:47.474818 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 91d0f6a5b96e406dac97ba2a77aaf597 (table default.Table_324426b580004200b83e6449cea91ce8 [id=f28dfce7eed645eaa5f7465a9fc66668]) successfully deleted
I20251027 21:07:47.481403 10029 tablet_bootstrap.cc:492] T bc2b7a771a1647c290ed9c3e394c60fe P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:47.481624 10029 ts_tablet_manager.cc:1403] T bc2b7a771a1647c290ed9c3e394c60fe P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.093s user 0.011s sys 0.011s
I20251027 21:07:47.482213 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 3218f18a797c40c9ad6ac64b2670cd19
I20251027 21:07:47.482583 10029 raft_consensus.cc:359] T bc2b7a771a1647c290ed9c3e394c60fe P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.482796 9900 tablet_service.cc:1552] Processing DeleteTablet for tablet 3218f18a797c40c9ad6ac64b2670cd19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.483068 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 3218f18a797c40c9ad6ac64b2670cd19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.483224 10029 raft_consensus.cc:385] T bc2b7a771a1647c290ed9c3e394c60fe P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.483258 9754 tablet_service.cc:1552] Processing DeleteTablet for tablet 3218f18a797c40c9ad6ac64b2670cd19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.483317 10029 raft_consensus.cc:740] T bc2b7a771a1647c290ed9c3e394c60fe P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
W20251027 21:07:47.483529 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 3218f18a797c40c9ad6ac64b2670cd19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3218f18a797c40c9ad6ac64b2670cd19 already in progress: creating tablet
I20251027 21:07:47.483561 10029 consensus_queue.cc:260] T bc2b7a771a1647c290ed9c3e394c60fe P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:47.483983 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 3218f18a797c40c9ad6ac64b2670cd19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3218f18a797c40c9ad6ac64b2670cd19 already in progress: creating tablet
I20251027 21:07:47.484800 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "347ec88f0a314ea9af6a7dfbd42fe305" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:47.485430 9859 leader_election.cc:343] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:47.485607 9859 leader_election.cc:304] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:07:47.486045 9859 raft_consensus.cc:2686] T 347ec88f0a314ea9af6a7dfbd42fe305 P 0fbd567f929a4f6d8e4d78feef7ed678: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251027 21:07:47.486124 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 12aa6e3af0694429b45bef5af3592377 has no consensus state
I20251027 21:07:47.486424 10029 ts_tablet_manager.cc:1434] T bc2b7a771a1647c290ed9c3e394c60fe P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251027 21:07:47.486788 10029 tablet_bootstrap.cc:492] T a4c022969ea448cf933d5220f7ae8f66 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:47.488103 10043 tablet_replica.cc:333] T f5b6f80625f54d818d251a8150578491 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:47.488467 10031 tablet_bootstrap.cc:492] T 347ec88f0a314ea9af6a7dfbd42fe305 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:47.488597 10031 ts_tablet_manager.cc:1403] T 347ec88f0a314ea9af6a7dfbd42fe305 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.083s user 0.019s sys 0.026s
I20251027 21:07:47.488756 10043 raft_consensus.cc:2243] T f5b6f80625f54d818d251a8150578491 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.489009 10043 raft_consensus.cc:2272] T f5b6f80625f54d818d251a8150578491 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:47.489380 10031 raft_consensus.cc:359] T 347ec88f0a314ea9af6a7dfbd42fe305 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:47.488760 10029 tablet_bootstrap.cc:654] T a4c022969ea448cf933d5220f7ae8f66 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.489552 10031 raft_consensus.cc:385] T 347ec88f0a314ea9af6a7dfbd42fe305 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.489575 10032 tablet_bootstrap.cc:492] T 3adbe5db3442462a8ae64a4b08129d69 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:47.489643 10031 raft_consensus.cc:740] T 347ec88f0a314ea9af6a7dfbd42fe305 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.489769 10032 ts_tablet_manager.cc:1403] T 3adbe5db3442462a8ae64a4b08129d69 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.094s user 0.015s sys 0.032s
I20251027 21:07:47.490410 10043 ts_tablet_manager.cc:1916] T f5b6f80625f54d818d251a8150578491 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:47.490546 10032 raft_consensus.cc:359] T 3adbe5db3442462a8ae64a4b08129d69 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.490744 10032 raft_consensus.cc:385] T 3adbe5db3442462a8ae64a4b08129d69 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.490823 10032 raft_consensus.cc:740] T 3adbe5db3442462a8ae64a4b08129d69 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.491077 10032 consensus_queue.cc:260] T 3adbe5db3442462a8ae64a4b08129d69 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.491539 10032 ts_tablet_manager.cc:1434] T 3adbe5db3442462a8ae64a4b08129d69 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:47.491861 10032 tablet_bootstrap.cc:492] T bc2b7a771a1647c290ed9c3e394c60fe P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:47.493216 9754 tablet_service.cc:1505] Processing CreateTablet for tablet f5da8d7ecccc4c3492f9d0d30ffea423 (DEFAULT_TABLE table=default.Table_50cddb609f3043f29b42899ad521fbee [id=63988803b0384c0b9766dccecf60804a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.493371 10032 tablet_bootstrap.cc:654] T bc2b7a771a1647c290ed9c3e394c60fe P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.493593 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5da8d7ecccc4c3492f9d0d30ffea423. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.493952 9900 tablet_service.cc:1505] Processing CreateTablet for tablet f5da8d7ecccc4c3492f9d0d30ffea423 (DEFAULT_TABLE table=default.Table_50cddb609f3043f29b42899ad521fbee [id=63988803b0384c0b9766dccecf60804a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.494333 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5da8d7ecccc4c3492f9d0d30ffea423. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.502553 9612 tablet_service.cc:1505] Processing CreateTablet for tablet 12aa6e3af0694429b45bef5af3592377 (DEFAULT_TABLE table=default.Table_36bfef90af1d48f78b2cce6a78527a44 [id=0065037e95714bb6ad48cbe647173541]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.503057 9751 tablet_service.cc:1505] Processing CreateTablet for tablet 0029b70ed3ff4db9a816a6190d36bf88 (DEFAULT_TABLE table=default.Table_450e3d88ee8f4a328418b279a354b9e2 [id=0f311c7d59a4471fbad8c2a0c59430f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.503173 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 0029b70ed3ff4db9a816a6190d36bf88 (DEFAULT_TABLE table=default.Table_450e3d88ee8f4a328418b279a354b9e2 [id=0f311c7d59a4471fbad8c2a0c59430f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.503377 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 40bd202c1b834c949f9a5cee23b3486d (DEFAULT_TABLE table=default.Table_948cff022cff49c9b8148e67df44ca00 [id=fd2beefa0629466daea4da402b033347]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.503621 9613 tablet_service.cc:1505] Processing CreateTablet for tablet 21b785277d604543bfa7816ed96d6181 (DEFAULT_TABLE table=default.Table_d7a6b578109241ccb05e5ffc8df713b5 [id=e3de2e5ea6484be7af68eb8601778b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.504099 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 0ac7111d13b6460a9f4c77e94e09d417 (DEFAULT_TABLE table=default.Table_9cac798aeef2436f9a044ec54e72daef [id=e2b68fb34c3b48e8952ef6d4201fd93f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.504455 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 12aa6e3af0694429b45bef5af3592377 (DEFAULT_TABLE table=default.Table_36bfef90af1d48f78b2cce6a78527a44 [id=0065037e95714bb6ad48cbe647173541]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.504604 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 40bd202c1b834c949f9a5cee23b3486d (DEFAULT_TABLE table=default.Table_948cff022cff49c9b8148e67df44ca00 [id=fd2beefa0629466daea4da402b033347]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.504730 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 0ac7111d13b6460a9f4c77e94e09d417 (DEFAULT_TABLE table=default.Table_9cac798aeef2436f9a044ec54e72daef [id=e2b68fb34c3b48e8952ef6d4201fd93f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.504973 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 21b785277d604543bfa7816ed96d6181 (DEFAULT_TABLE table=default.Table_d7a6b578109241ccb05e5ffc8df713b5 [id=e3de2e5ea6484be7af68eb8601778b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.505224 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 9cec6294051b41868c9c7c7fadafa138 (DEFAULT_TABLE table=default.Table_fce1fc7d60384a6b86622ff2afae27b6 [id=12b1bbc770194455ab7292641980e8e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.505501 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0029b70ed3ff4db9a816a6190d36bf88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.505877 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40bd202c1b834c949f9a5cee23b3486d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.508282 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 12aa6e3af0694429b45bef5af3592377 (DEFAULT_TABLE table=default.Table_36bfef90af1d48f78b2cce6a78527a44 [id=0065037e95714bb6ad48cbe647173541]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.508642 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12aa6e3af0694429b45bef5af3592377. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.505375 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 9cec6294051b41868c9c7c7fadafa138 (DEFAULT_TABLE table=default.Table_fce1fc7d60384a6b86622ff2afae27b6 [id=12b1bbc770194455ab7292641980e8e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.510581 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21b785277d604543bfa7816ed96d6181. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.510660 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 0029b70ed3ff4db9a816a6190d36bf88 (DEFAULT_TABLE table=default.Table_450e3d88ee8f4a328418b279a354b9e2 [id=0f311c7d59a4471fbad8c2a0c59430f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.510933 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ac7111d13b6460a9f4c77e94e09d417. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.512145 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_29a176f38aaa46218fcd17638ef0ce72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.512696 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29a176f38aaa46218fcd17638ef0ce72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.513358 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cec6294051b41868c9c7c7fadafa138. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.514568 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12aa6e3af0694429b45bef5af3592377. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.515743 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 63662f175a1c450f8f67f37da971db37 (DEFAULT_TABLE table=default.Table_d7caf208b9864991a0e50ced803eb60d [id=4142b5bb00004c83afe20e6982409849]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.516245 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63662f175a1c450f8f67f37da971db37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.518514 9614 tablet_service.cc:1505] Processing CreateTablet for tablet f5da8d7ecccc4c3492f9d0d30ffea423 (DEFAULT_TABLE table=default.Table_50cddb609f3043f29b42899ad521fbee [id=63988803b0384c0b9766dccecf60804a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.519068 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_108afd04fe5d45c195ba81996dee61d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.519546 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_108afd04fe5d45c195ba81996dee61d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.521467 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0029b70ed3ff4db9a816a6190d36bf88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.523211 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_41a04598f12247fbac5f4103d2cbf22d" } modify_external_catalogs: true
I20251027 21:07:47.523414 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_41a04598f12247fbac5f4103d2cbf22d" } modify_external_catalogs: true
I20251027 21:07:47.526895 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet 3adbe5db3442462a8ae64a4b08129d69 with delete_type TABLET_DATA_DELETED (replaced by tablet 3218f18a797c40c9ad6ac64b2670cd19) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.529663 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_450e3d88ee8f4a328418b279a354b9e2" } modify_external_catalogs: true
I20251027 21:07:47.529846 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_450e3d88ee8f4a328418b279a354b9e2" } modify_external_catalogs: true
I20251027 21:07:47.532289 10041 tablet_replica.cc:333] T 3adbe5db3442462a8ae64a4b08129d69 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:47.532462 10041 raft_consensus.cc:2243] T 3adbe5db3442462a8ae64a4b08129d69 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.533011 10041 raft_consensus.cc:2272] T 3adbe5db3442462a8ae64a4b08129d69 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:47.534888 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 84305bcf21b64d2c8c1df015a2e9615c
I20251027 21:07:47.535995 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_be23b67279484b3db60ef67b1b157253"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:47.536091 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_84d985f1c9f345df85a4102d9c155a6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.536476 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be23b67279484b3db60ef67b1b157253 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:47.536584 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84d985f1c9f345df85a4102d9c155a6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.537766 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d81917295a3548f787986bf566037719"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.538211 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d81917295a3548f787986bf566037719 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.538887 9608 tablet_service.cc:1552] Processing DeleteTablet for tablet 84305bcf21b64d2c8c1df015a2e9615c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.540149 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 9cec6294051b41868c9c7c7fadafa138 (DEFAULT_TABLE table=default.Table_fce1fc7d60384a6b86622ff2afae27b6 [id=12b1bbc770194455ab7292641980e8e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.540639 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cec6294051b41868c9c7c7fadafa138. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.541592 10031 consensus_queue.cc:260] T 347ec88f0a314ea9af6a7dfbd42fe305 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:47.544382 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 84305bcf21b64d2c8c1df015a2e9615c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.545311 10031 ts_tablet_manager.cc:1434] T 347ec88f0a314ea9af6a7dfbd42fe305 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.057s user 0.001s sys 0.001s
W20251027 21:07:47.545464 9297 catalog_manager.cc:4952] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 84305bcf21b64d2c8c1df015a2e9615c because the tablet was not found. No further retry: Not found: Tablet not found: 84305bcf21b64d2c8c1df015a2e9615c
I20251027 21:07:47.545961 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 347ec88f0a314ea9af6a7dfbd42fe305 with delete_type TABLET_DATA_DELETED (replaced by tablet a52f41629a2744ef8db0d57e06f8e0a4) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.546257 10040 tablet_replica.cc:333] T 347ec88f0a314ea9af6a7dfbd42fe305 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:47.546442 10040 raft_consensus.cc:2243] T 347ec88f0a314ea9af6a7dfbd42fe305 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.546600 10040 raft_consensus.cc:2272] T 347ec88f0a314ea9af6a7dfbd42fe305 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:47.546618 9901 tablet_service.cc:1505] Processing CreateTablet for tablet 21b785277d604543bfa7816ed96d6181 (DEFAULT_TABLE table=default.Table_d7a6b578109241ccb05e5ffc8df713b5 [id=e3de2e5ea6484be7af68eb8601778b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.547794 10031 tablet_bootstrap.cc:492] T bc2b7a771a1647c290ed9c3e394c60fe P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:47.548218 10040 ts_tablet_manager.cc:1916] T 347ec88f0a314ea9af6a7dfbd42fe305 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:47.548740 9753 tablet_service.cc:1505] Processing CreateTablet for tablet d642da56a67449aa95932180bc5b2c5f (DEFAULT_TABLE table=default.Table_178e5d554faa4e119ac8f52da2d284d7 [id=05edeb649c544c2f899ed42a87cef2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.549218 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d642da56a67449aa95932180bc5b2c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.549703 10031 tablet_bootstrap.cc:654] T bc2b7a771a1647c290ed9c3e394c60fe P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.550613 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 0029b70ed3ff4db9a816a6190d36bf88
I20251027 21:07:47.551703 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 0029b70ed3ff4db9a816a6190d36bf88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.555068 9903 tablet_service.cc:1505] Processing CreateTablet for tablet 0ac7111d13b6460a9f4c77e94e09d417 (DEFAULT_TABLE table=default.Table_9cac798aeef2436f9a044ec54e72daef [id=e2b68fb34c3b48e8952ef6d4201fd93f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.555377 10041 ts_tablet_manager.cc:1916] T 3adbe5db3442462a8ae64a4b08129d69 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:47.555437 9608 tablet_service.cc:1505] Processing CreateTablet for tablet 63662f175a1c450f8f67f37da971db37 (DEFAULT_TABLE table=default.Table_d7caf208b9864991a0e50ced803eb60d [id=4142b5bb00004c83afe20e6982409849]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.556299 9607 tablet_service.cc:1505] Processing CreateTablet for tablet d642da56a67449aa95932180bc5b2c5f (DEFAULT_TABLE table=default.Table_178e5d554faa4e119ac8f52da2d284d7 [id=05edeb649c544c2f899ed42a87cef2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.556751 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 5c87a5450d354506852c4b20bf0541d2 (DEFAULT_TABLE table=default.Table_d1fb7f4ef09e4bfdac00a75f752f0873 [id=1cf515b159954979acd73a0b276e7503]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.556846 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet 0029b70ed3ff4db9a816a6190d36bf88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.559724 10040 ts_tablet_manager.cc:1929] T 347ec88f0a314ea9af6a7dfbd42fe305 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:47.559744 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c87a5450d354506852c4b20bf0541d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.559993 10040 log.cc:1199] T 347ec88f0a314ea9af6a7dfbd42fe305 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/347ec88f0a314ea9af6a7dfbd42fe305
I20251027 21:07:47.564345 9901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21b785277d604543bfa7816ed96d6181. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.565340 10040 ts_tablet_manager.cc:1950] T 347ec88f0a314ea9af6a7dfbd42fe305 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:47.566466 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet 84305bcf21b64d2c8c1df015a2e9615c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.566584 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_44e3f093c8d7476a98074b07828da3ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.567068 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44e3f093c8d7476a98074b07828da3ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.568718 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 347ec88f0a314ea9af6a7dfbd42fe305 (table default.Table_d233023476bd4092bc445f39c7dd0ec6 [id=7c1a1f82393b4e658ddb7b05bee043e9]) successfully deleted
I20251027 21:07:47.569694 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 5c87a5450d354506852c4b20bf0541d2 (DEFAULT_TABLE table=default.Table_d1fb7f4ef09e4bfdac00a75f752f0873 [id=1cf515b159954979acd73a0b276e7503]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.570417 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f9d4630f65ee49bf9969224da34b519f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:47.570794 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 84305bcf21b64d2c8c1df015a2e9615c (DEFAULT_TABLE table=default.Table_41a04598f12247fbac5f4103d2cbf22d [id=2e28a4d0f9204351b93335736a9abe18]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:47.570979 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9d4630f65ee49bf9969224da34b519f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.571904 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 84305bcf21b64d2c8c1df015a2e9615c (DEFAULT_TABLE table=default.Table_41a04598f12247fbac5f4103d2cbf22d [id=2e28a4d0f9204351b93335736a9abe18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.572299 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84305bcf21b64d2c8c1df015a2e9615c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.575829 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ac7111d13b6460a9f4c77e94e09d417. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.554934 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 40bd202c1b834c949f9a5cee23b3486d (DEFAULT_TABLE table=default.Table_948cff022cff49c9b8148e67df44ca00 [id=fd2beefa0629466daea4da402b033347]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.578924 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3886813b08244afa9a2d851978407392"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.579552 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3886813b08244afa9a2d851978407392 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.581481 9305 master_service.cc:954] ReplaceTablet: received request to replace tablet bd6230765d6448498132e24482100b39 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:47.583597 9904 tablet_service.cc:1505] Processing CreateTablet for tablet d642da56a67449aa95932180bc5b2c5f (DEFAULT_TABLE table=default.Table_178e5d554faa4e119ac8f52da2d284d7 [id=05edeb649c544c2f899ed42a87cef2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.584012 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d642da56a67449aa95932180bc5b2c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.587770 9896 tablet_service.cc:1552] Processing DeleteTablet for tablet 0029b70ed3ff4db9a816a6190d36bf88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:47.587791 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 0029b70ed3ff4db9a816a6190d36bf88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0029b70ed3ff4db9a816a6190d36bf88 already in progress: creating tablet
I20251027 21:07:47.601182 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 5c87a5450d354506852c4b20bf0541d2 (DEFAULT_TABLE table=default.Table_d1fb7f4ef09e4bfdac00a75f752f0873 [id=1cf515b159954979acd73a0b276e7503]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.601220 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 63662f175a1c450f8f67f37da971db37 (DEFAULT_TABLE table=default.Table_d7caf208b9864991a0e50ced803eb60d [id=4142b5bb00004c83afe20e6982409849]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.604929 9894 tablet_service.cc:1505] Processing CreateTablet for tablet 84305bcf21b64d2c8c1df015a2e9615c (DEFAULT_TABLE table=default.Table_41a04598f12247fbac5f4103d2cbf22d [id=2e28a4d0f9204351b93335736a9abe18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.605614 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40bd202c1b834c949f9a5cee23b3486d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.607702 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84305bcf21b64d2c8c1df015a2e9615c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.607997 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63662f175a1c450f8f67f37da971db37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.609263 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d642da56a67449aa95932180bc5b2c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.614203 9273 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu with pid 9420
I20251027 21:07:47.617874 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84305bcf21b64d2c8c1df015a2e9615c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.618381 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ac7111d13b6460a9f4c77e94e09d417. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.619769 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c87a5450d354506852c4b20bf0541d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.623302 10029 tablet_bootstrap.cc:492] T a4c022969ea448cf933d5220f7ae8f66 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:47.623504 10029 ts_tablet_manager.cc:1403] T a4c022969ea448cf933d5220f7ae8f66 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.137s user 0.003s sys 0.000s
I20251027 21:07:47.624428 10029 raft_consensus.cc:359] T a4c022969ea448cf933d5220f7ae8f66 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.627390 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0029b70ed3ff4db9a816a6190d36bf88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.628085 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40bd202c1b834c949f9a5cee23b3486d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.628342 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12aa6e3af0694429b45bef5af3592377. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.628438 10031 tablet_bootstrap.cc:492] T bc2b7a771a1647c290ed9c3e394c60fe P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:47.628600 10031 ts_tablet_manager.cc:1403] T bc2b7a771a1647c290ed9c3e394c60fe P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.081s user 0.016s sys 0.023s
I20251027 21:07:47.629333 10031 raft_consensus.cc:359] T bc2b7a771a1647c290ed9c3e394c60fe P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.629545 10031 raft_consensus.cc:385] T bc2b7a771a1647c290ed9c3e394c60fe P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.629632 10031 raft_consensus.cc:740] T bc2b7a771a1647c290ed9c3e394c60fe P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.630503 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c87a5450d354506852c4b20bf0541d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.631323 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63662f175a1c450f8f67f37da971db37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.631682 10031 consensus_queue.cc:260] T bc2b7a771a1647c290ed9c3e394c60fe P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.634224 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cec6294051b41868c9c7c7fadafa138. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.634825 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet bc2b7a771a1647c290ed9c3e394c60fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.636384 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5d5f327d728f4250841a9f4f10a2b533"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:47.635826 10031 ts_tablet_manager.cc:1434] T bc2b7a771a1647c290ed9c3e394c60fe P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
W20251027 21:07:47.636956 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d5f327d728f4250841a9f4f10a2b533 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:47.636384 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet bc2b7a771a1647c290ed9c3e394c60fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc2b7a771a1647c290ed9c3e394c60fe already in progress: creating tablet
I20251027 21:07:47.638484 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21b785277d604543bfa7816ed96d6181. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.639134 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet bc2b7a771a1647c290ed9c3e394c60fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.639227 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5da8d7ecccc4c3492f9d0d30ffea423. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.639448 10040 tablet_replica.cc:333] T bc2b7a771a1647c290ed9c3e394c60fe P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:47.639592 10040 raft_consensus.cc:2243] T bc2b7a771a1647c290ed9c3e394c60fe P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.639742 10040 raft_consensus.cc:2272] T bc2b7a771a1647c290ed9c3e394c60fe P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:47.640280 10031 tablet_bootstrap.cc:492] T 4bcd083a30a44ba898211eb769bc4252 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:47.640494 10040 ts_tablet_manager.cc:1916] T bc2b7a771a1647c290ed9c3e394c60fe P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:47.642061 10031 tablet_bootstrap.cc:654] T 4bcd083a30a44ba898211eb769bc4252 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.646097 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet bc2b7a771a1647c290ed9c3e394c60fe from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:47.646236 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet a4c022969ea448cf933d5220f7ae8f66 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:47.647541 10031 tablet_bootstrap.cc:492] T 4bcd083a30a44ba898211eb769bc4252 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:47.647665 10031 ts_tablet_manager.cc:1403] T 4bcd083a30a44ba898211eb769bc4252 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20251027 21:07:47.647909 10029 raft_consensus.cc:385] T a4c022969ea448cf933d5220f7ae8f66 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.648066 10029 raft_consensus.cc:740] T a4c022969ea448cf933d5220f7ae8f66 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.648321 10040 ts_tablet_manager.cc:1929] T bc2b7a771a1647c290ed9c3e394c60fe P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:47.648504 10040 log.cc:1199] T bc2b7a771a1647c290ed9c3e394c60fe P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/bc2b7a771a1647c290ed9c3e394c60fe
I20251027 21:07:47.648993 10040 ts_tablet_manager.cc:1950] T bc2b7a771a1647c290ed9c3e394c60fe P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:47.649730 10031 raft_consensus.cc:359] T 4bcd083a30a44ba898211eb769bc4252 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.649799 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet bc2b7a771a1647c290ed9c3e394c60fe (table default.Table_e46f2bc398e145209af5db2546ad90f7 [id=5280a5a662394d82b01929699833a1dc]) successfully deleted
I20251027 21:07:47.649905 10031 raft_consensus.cc:385] T 4bcd083a30a44ba898211eb769bc4252 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.649984 10031 raft_consensus.cc:740] T 4bcd083a30a44ba898211eb769bc4252 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.650277 10029 consensus_queue.cc:260] T a4c022969ea448cf933d5220f7ae8f66 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.650797 10029 ts_tablet_manager.cc:1434] T a4c022969ea448cf933d5220f7ae8f66 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20251027 21:07:47.651180 10031 consensus_queue.cc:260] T 4bcd083a30a44ba898211eb769bc4252 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.653180 10029 tablet_bootstrap.cc:492] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:47.654505 10031 ts_tablet_manager.cc:1434] T 4bcd083a30a44ba898211eb769bc4252 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20251027 21:07:47.654785 10031 tablet_bootstrap.cc:492] T a4c022969ea448cf933d5220f7ae8f66 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:47.655524 9304 master_service.cc:954] ReplaceTablet: received request to replace tablet 5c87a5450d354506852c4b20bf0541d2 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:47.656148 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b889b1ae5fb44f19a198a8ebabad297c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.656747 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b889b1ae5fb44f19a198a8ebabad297c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.657174 10031 tablet_bootstrap.cc:654] T a4c022969ea448cf933d5220f7ae8f66 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.658192 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bcd083a30a44ba898211eb769bc4252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.664228 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_fce1fc7d60384a6b86622ff2afae27b6" } new_table_name: "default.Table_5d8e4122af254070a2df7016e521a937" modify_external_catalogs: true
I20251027 21:07:47.665040 10040 tablet_replica.cc:333] T 4bcd083a30a44ba898211eb769bc4252 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:47.665524 10029 tablet_bootstrap.cc:654] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.665460 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_29a176f38aaa46218fcd17638ef0ce72" } new_table_name: "default.Table_5942b242afb64f95a82e2a0bafe674f0" modify_external_catalogs: true
I20251027 21:07:47.667086 10040 raft_consensus.cc:2243] T 4bcd083a30a44ba898211eb769bc4252 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.667461 9309 master_service.cc:954] ReplaceTablet: received request to replace tablet 988052533a1042d9bba5ed5711d09d9c from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:47.667292 10040 raft_consensus.cc:2272] T 4bcd083a30a44ba898211eb769bc4252 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:47.669569 10040 ts_tablet_manager.cc:1916] T 4bcd083a30a44ba898211eb769bc4252 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:47.674402 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_948cff022cff49c9b8148e67df44ca00" } new_table_name: "default.Table_f9b456b781864738a6fa0d5c4bfa50fd" modify_external_catalogs: true
I20251027 21:07:47.674676 9608 tablet_service.cc:1505] Processing CreateTablet for tablet ba8b125474bc42a5946e8f2dd718f7f4 (DEFAULT_TABLE table=default.Table_84d985f1c9f345df85a4102d9c155a6e [id=e7b7bdb2ea7f468c98bfe88f900a5be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.675146 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba8b125474bc42a5946e8f2dd718f7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.675782 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_96482998755b4c28840abc6ccea5d311" } new_table_name: "default.Table_df95aebbdcf94d8289b235722b7d2c8c" modify_external_catalogs: true
I20251027 21:07:47.668694 10031 tablet_bootstrap.cc:492] T a4c022969ea448cf933d5220f7ae8f66 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:47.678555 10031 ts_tablet_manager.cc:1403] T a4c022969ea448cf933d5220f7ae8f66 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.005s
I20251027 21:07:47.679049 10040 ts_tablet_manager.cc:1929] T 4bcd083a30a44ba898211eb769bc4252 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:47.679188 10040 log.cc:1199] T 4bcd083a30a44ba898211eb769bc4252 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/4bcd083a30a44ba898211eb769bc4252
I20251027 21:07:47.679535 10040 ts_tablet_manager.cc:1950] T 4bcd083a30a44ba898211eb769bc4252 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:47.679664 10031 raft_consensus.cc:359] T a4c022969ea448cf933d5220f7ae8f66 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.679911 10031 raft_consensus.cc:385] T a4c022969ea448cf933d5220f7ae8f66 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.680033 10031 raft_consensus.cc:740] T a4c022969ea448cf933d5220f7ae8f66 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.680527 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 6e5d52b7ec114be5ae68e57280c2eea8 (DEFAULT_TABLE table=default.Table_d81917295a3548f787986bf566037719 [id=709b33acbd184a218ec6b700150b58b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.680871 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e5d52b7ec114be5ae68e57280c2eea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.680924 9751 tablet_service.cc:1505] Processing CreateTablet for tablet b3dfef8b741c42f68b664d7e0eab49c6 (DEFAULT_TABLE table=default.Table_be23b67279484b3db60ef67b1b157253 [id=9e10359cf7a640e9b16f01096987ca59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.681098 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 78a0a8b05f10477c923ab565839141b8 (DEFAULT_TABLE table=default.Table_108afd04fe5d45c195ba81996dee61d9 [id=02839636442f4955b245a06a594634c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.681275 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3dfef8b741c42f68b664d7e0eab49c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.681602 9750 tablet_service.cc:1505] Processing CreateTablet for tablet cbada3946dab4739a8369032ff3b8a48 (DEFAULT_TABLE table=default.Table_29a176f38aaa46218fcd17638ef0ce72 [id=67b71acdadc64445bb2492e5d424cb2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.681921 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbada3946dab4739a8369032ff3b8a48. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:47.682540 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9cec6294051b41868c9c7c7fadafa138 consensus state has no leader
I20251027 21:07:47.682775 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_36bfef90af1d48f78b2cce6a78527a44" } modify_external_catalogs: true
I20251027 21:07:47.682832 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_be23b67279484b3db60ef67b1b157253" } modify_external_catalogs: true
I20251027 21:07:47.682956 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_36bfef90af1d48f78b2cce6a78527a44" } modify_external_catalogs: true
I20251027 21:07:47.683027 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_be23b67279484b3db60ef67b1b157253" } modify_external_catalogs: true
I20251027 21:07:47.685853 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a0a8b05f10477c923ab565839141b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.686520 9749 tablet_service.cc:1505] Processing CreateTablet for tablet ba8b125474bc42a5946e8f2dd718f7f4 (DEFAULT_TABLE table=default.Table_84d985f1c9f345df85a4102d9c155a6e [id=e7b7bdb2ea7f468c98bfe88f900a5be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.686916 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba8b125474bc42a5946e8f2dd718f7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.689232 9612 tablet_service.cc:1505] Processing CreateTablet for tablet cbada3946dab4739a8369032ff3b8a48 (DEFAULT_TABLE table=default.Table_29a176f38aaa46218fcd17638ef0ce72 [id=67b71acdadc64445bb2492e5d424cb2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.689606 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbada3946dab4739a8369032ff3b8a48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.691903 10029 tablet_bootstrap.cc:492] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
W20251027 21:07:47.692018 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cbada3946dab4739a8369032ff3b8a48 consensus state has no leader
I20251027 21:07:47.692126 10029 ts_tablet_manager.cc:1403] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20251027 21:07:47.693513 10029 raft_consensus.cc:359] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.693722 10029 raft_consensus.cc:385] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.693849 10029 raft_consensus.cc:740] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.694108 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 5c87a5450d354506852c4b20bf0541d2
I20251027 21:07:47.694123 10029 consensus_queue.cc:260] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.694312 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 988052533a1042d9bba5ed5711d09d9c
I20251027 21:07:47.694770 9304 catalog_manager.cc:6719] ReplaceTablet: tablet 5c87a5450d354506852c4b20bf0541d2 deleted and replaced by tablet 77b2d15e86b244708a2d07f88bf8278d
I20251027 21:07:47.694793 9309 catalog_manager.cc:6719] ReplaceTablet: tablet 988052533a1042d9bba5ed5711d09d9c deleted and replaced by tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe
I20251027 21:07:47.696094 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c87a5450d354506852c4b20bf0541d2 with delete_type TABLET_DATA_DELETED (replaced by tablet 77b2d15e86b244708a2d07f88bf8278d) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.696763 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f9d4630f65ee49bf9969224da34b519f" } new_table_name: "default.Table_2225c7fa6a5a4d6eb28c8d58e77b4397" modify_external_catalogs: true
I20251027 21:07:47.697185 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet a4c022969ea448cf933d5220f7ae8f66
I20251027 21:07:47.697489 10029 ts_tablet_manager.cc:1434] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251027 21:07:47.697640 9303 catalog_manager.cc:6719] ReplaceTablet: tablet a4c022969ea448cf933d5220f7ae8f66 deleted and replaced by tablet d8ef795c88da49b58082ba1264d83378
W20251027 21:07:47.698501 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 40bd202c1b834c949f9a5cee23b3486d consensus state has no leader
I20251027 21:07:47.699527 10029 tablet_bootstrap.cc:492] T 0cd4f5fa03d4417383df279da475345f P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:47.699710 9608 tablet_service.cc:1552] Processing DeleteTablet for tablet 988052533a1042d9bba5ed5711d09d9c with delete_type TABLET_DATA_DELETED (replaced by tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.702525 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 4bcd083a30a44ba898211eb769bc4252 (table default.Table_e6bc4379784d49cbae1c20888dc66289 [id=0b6aecd6b64c4f2e837bf7fe73166ab2]) successfully deleted
I20251027 21:07:47.702757 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 988052533a1042d9bba5ed5711d09d9c with delete_type TABLET_DATA_DELETED (replaced by tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.702809 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet a4c022969ea448cf933d5220f7ae8f66 with delete_type TABLET_DATA_DELETED (replaced by tablet d8ef795c88da49b58082ba1264d83378) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.703147 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_53596f51be6a408a8987da9e37e4d779" } modify_external_catalogs: true
I20251027 21:07:47.703336 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_53596f51be6a408a8987da9e37e4d779" } modify_external_catalogs: true
I20251027 21:07:47.703399 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c87a5450d354506852c4b20bf0541d2 with delete_type TABLET_DATA_DELETED (replaced by tablet 77b2d15e86b244708a2d07f88bf8278d) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.703963 10031 consensus_queue.cc:260] T a4c022969ea448cf933d5220f7ae8f66 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.704504 10029 tablet_bootstrap.cc:654] T 0cd4f5fa03d4417383df279da475345f P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.704944 10031 ts_tablet_manager.cc:1434] T a4c022969ea448cf933d5220f7ae8f66 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.026s user 0.000s sys 0.001s
W20251027 21:07:47.710705 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet a4c022969ea448cf933d5220f7ae8f66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4c022969ea448cf933d5220f7ae8f66 already in progress: creating tablet
W20251027 21:07:47.710978 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 988052533a1042d9bba5ed5711d09d9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 988052533a1042d9bba5ed5711d09d9c already in progress: creating tablet
W20251027 21:07:47.711217 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 5c87a5450d354506852c4b20bf0541d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c87a5450d354506852c4b20bf0541d2 already in progress: creating tablet
W20251027 21:07:47.711798 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d8ef795c88da49b58082ba1264d83378 has no consensus state
I20251027 21:07:47.713057 10031 tablet_bootstrap.cc:492] T 0cd4f5fa03d4417383df279da475345f P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:47.714617 9307 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet a878184a3a444761aff8730bdf43ccaf (table default.Table_1c94591b5a734f7a8ce72ed245351b55 [id=77687d6093f741d3905c660d0650ae13]). Expected version 1 got 0
I20251027 21:07:47.715103 10031 tablet_bootstrap.cc:654] T 0cd4f5fa03d4417383df279da475345f P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.718279 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet b3dfef8b741c42f68b664d7e0eab49c6
I20251027 21:07:47.718397 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 12aa6e3af0694429b45bef5af3592377
I20251027 21:07:47.719522 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet 12aa6e3af0694429b45bef5af3592377 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.719725 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 12aa6e3af0694429b45bef5af3592377 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.719780 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet b3dfef8b741c42f68b664d7e0eab49c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:47.720386 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 12aa6e3af0694429b45bef5af3592377 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12aa6e3af0694429b45bef5af3592377 already in progress: creating tablet
I20251027 21:07:47.720608 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet b3dfef8b741c42f68b664d7e0eab49c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:47.721141 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet b3dfef8b741c42f68b664d7e0eab49c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3dfef8b741c42f68b664d7e0eab49c6 already in progress: creating tablet
W20251027 21:07:47.722433 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3c9f64ee0544e59add4f73bca59df98 has no consensus state
I20251027 21:07:47.722538 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 783d845009d54d20b8e6626c3e73d2a7
I20251027 21:07:47.723351 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 783d845009d54d20b8e6626c3e73d2a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.723605 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet 783d845009d54d20b8e6626c3e73d2a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:47.723840 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 783d845009d54d20b8e6626c3e73d2a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 783d845009d54d20b8e6626c3e73d2a7 already in progress: creating tablet
I20251027 21:07:47.727685 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 6e5d52b7ec114be5ae68e57280c2eea8 (DEFAULT_TABLE table=default.Table_d81917295a3548f787986bf566037719 [id=709b33acbd184a218ec6b700150b58b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.729847 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet a4c022969ea448cf933d5220f7ae8f66 with delete_type TABLET_DATA_DELETED (replaced by tablet d8ef795c88da49b58082ba1264d83378) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.730115 10040 tablet_replica.cc:333] T a4c022969ea448cf933d5220f7ae8f66 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:47.730309 10040 raft_consensus.cc:2243] T a4c022969ea448cf933d5220f7ae8f66 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.730487 10040 raft_consensus.cc:2272] T a4c022969ea448cf933d5220f7ae8f66 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:47.730532 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e5d52b7ec114be5ae68e57280c2eea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.731169 10040 ts_tablet_manager.cc:1916] T a4c022969ea448cf933d5220f7ae8f66 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:47.736922 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 78a0a8b05f10477c923ab565839141b8 (DEFAULT_TABLE table=default.Table_108afd04fe5d45c195ba81996dee61d9 [id=02839636442f4955b245a06a594634c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.737370 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a0a8b05f10477c923ab565839141b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.737728 10040 ts_tablet_manager.cc:1929] T a4c022969ea448cf933d5220f7ae8f66 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:47.737840 10040 log.cc:1199] T a4c022969ea448cf933d5220f7ae8f66 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/a4c022969ea448cf933d5220f7ae8f66
I20251027 21:07:47.738173 10040 ts_tablet_manager.cc:1950] T a4c022969ea448cf933d5220f7ae8f66 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:47.739105 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet a4c022969ea448cf933d5220f7ae8f66 (table default.Table_df95aebbdcf94d8289b235722b7d2c8c [id=98143d42942742568f2c3a1d0fbcc100]) successfully deleted
I20251027 21:07:47.739663 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 6e5d52b7ec114be5ae68e57280c2eea8 (DEFAULT_TABLE table=default.Table_d81917295a3548f787986bf566037719 [id=709b33acbd184a218ec6b700150b58b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.740042 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e5d52b7ec114be5ae68e57280c2eea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.744012 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_8c58943248a74b5ab81a5b742eafc5b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.744593 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c58943248a74b5ab81a5b742eafc5b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.746061 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d1fb7f4ef09e4bfdac00a75f752f0873" } modify_external_catalogs: true
I20251027 21:07:47.746235 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d1fb7f4ef09e4bfdac00a75f752f0873" } modify_external_catalogs: true
W20251027 21:07:47.751108 9970 heartbeater.cc:646] Failed to heartbeat to 127.9.14.124:45071 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.9.14.124:45071 (error 108)
I20251027 21:07:47.752636 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_98c734ca00af4e01a7da18accd5f6c30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.753144 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98c734ca00af4e01a7da18accd5f6c30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.753093 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_02a777c596804daa888665c5e79ea115"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:47.753254 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_1764f51a79024aac9bcd15e3f03fd314"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.753605 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02a777c596804daa888665c5e79ea115 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:47.753955 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1764f51a79024aac9bcd15e3f03fd314 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.754899 9606 tablet_service.cc:1505] Processing CreateTablet for tablet b3dfef8b741c42f68b664d7e0eab49c6 (DEFAULT_TABLE table=default.Table_be23b67279484b3db60ef67b1b157253 [id=9e10359cf7a640e9b16f01096987ca59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.755273 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3dfef8b741c42f68b664d7e0eab49c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.758176 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0377267e434442dfb2e1cce27b28dd35" } modify_external_catalogs: true
I20251027 21:07:47.758432 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0377267e434442dfb2e1cce27b28dd35" } modify_external_catalogs: true
W20251027 21:07:47.758507 9820 heartbeater.cc:646] Failed to heartbeat to 127.9.14.124:45071 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.9.14.124:45071 (error 108)
W20251027 21:07:47.759184 9682 heartbeater.cc:646] Failed to heartbeat to 127.9.14.124:45071 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.9.14.124:45071 (error 108)
I20251027 21:07:47.767526 10043 ts_tablet_manager.cc:1929] T f5b6f80625f54d818d251a8150578491 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:47.767760 10043 log.cc:1199] T f5b6f80625f54d818d251a8150578491 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/f5b6f80625f54d818d251a8150578491
I20251027 21:07:47.768355 10043 ts_tablet_manager.cc:1950] T f5b6f80625f54d818d251a8150578491 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
W20251027 21:07:47.771292 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 0cd4f5fa03d4417383df279da475345f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cd4f5fa03d4417383df279da475345f already in progress: creating tablet
I20251027 21:07:47.776244 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4ea5cf4870a14209b590147d71a46a0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.777271 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ea5cf4870a14209b590147d71a46a0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.778956 9306 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 77b2d15e86b244708a2d07f88bf8278d
I20251027 21:07:47.787657 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet a4c022969ea448cf933d5220f7ae8f66 with delete_type TABLET_DATA_DELETED (replaced by tablet d8ef795c88da49b58082ba1264d83378) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.784350 9898 tablet_service.cc:1505] Processing CreateTablet for tablet ba8b125474bc42a5946e8f2dd718f7f4 (DEFAULT_TABLE table=default.Table_84d985f1c9f345df85a4102d9c155a6e [id=e7b7bdb2ea7f468c98bfe88f900a5be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.783211 9891 tablet_service.cc:1505] Processing CreateTablet for tablet b3dfef8b741c42f68b664d7e0eab49c6 (DEFAULT_TABLE table=default.Table_be23b67279484b3db60ef67b1b157253 [id=9e10359cf7a640e9b16f01096987ca59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.792119 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_52c53bafe66b4c66b99e5b26e4df7ef5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:47.792279 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6b913111f04c4f1abe4b7244040b07d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.793020 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b913111f04c4f1abe4b7244040b07d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.794574 10043 tablet_replica.cc:333] T bc2b7a771a1647c290ed9c3e394c60fe P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:47.794796 10043 raft_consensus.cc:2243] T bc2b7a771a1647c290ed9c3e394c60fe P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.794968 10043 raft_consensus.cc:2272] T bc2b7a771a1647c290ed9c3e394c60fe P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:07:47.795221 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52c53bafe66b4c66b99e5b26e4df7ef5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.796716 10043 ts_tablet_manager.cc:1916] T bc2b7a771a1647c290ed9c3e394c60fe P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:47.799846 10072 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 21:07:47.759753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.14.124:45071 (local address 127.9.14.65:40397)
1027 21:07:47.798766 (+ 39013us) negotiation.cc:107] Waiting for socket to connect
1027 21:07:47.798880 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":38912,"threads_started":1}
I20251027 21:07:47.802446 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 65fbcaa5447c48818fabfccac22c4048 (DEFAULT_TABLE table=default.Table_b889b1ae5fb44f19a198a8ebabad297c [id=4369b0d36cc14f48a5f41f214aa4fe4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.802838 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65fbcaa5447c48818fabfccac22c4048. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.803969 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba8b125474bc42a5946e8f2dd718f7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.804785 9901 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c87a5450d354506852c4b20bf0541d2 with delete_type TABLET_DATA_DELETED (replaced by tablet 77b2d15e86b244708a2d07f88bf8278d) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.806223 9888 tablet_service.cc:1552] Processing DeleteTablet for tablet b3dfef8b741c42f68b664d7e0eab49c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.806434 9886 tablet_service.cc:1552] Processing DeleteTablet for tablet 783d845009d54d20b8e6626c3e73d2a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.806618 9892 tablet_service.cc:1505] Processing CreateTablet for tablet cbada3946dab4739a8369032ff3b8a48 (DEFAULT_TABLE table=default.Table_29a176f38aaa46218fcd17638ef0ce72 [id=67b71acdadc64445bb2492e5d424cb2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.807138 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet 12aa6e3af0694429b45bef5af3592377 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:47.811846 10059 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 21:07:47.751626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.14.124:45071 (local address 127.9.14.67:58123)
1027 21:07:47.751838 (+ 212us) negotiation.cc:107] Waiting for socket to connect
1027 21:07:47.751855 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 21:07:47.751950 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 21:07:47.786890 (+ 34940us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":108}
I20251027 21:07:47.804026 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 988052533a1042d9bba5ed5711d09d9c with delete_type TABLET_DATA_DELETED (replaced by tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:47.816231 10073 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 21:07:47.760101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.14.124:45071 (local address 127.9.14.66:43365)
1027 21:07:47.795775 (+ 35674us) negotiation.cc:107] Waiting for socket to connect
1027 21:07:47.815230 (+ 19455us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":35551,"thread_start_us":34146,"threads_started":1}
I20251027 21:07:47.816648 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 65fbcaa5447c48818fabfccac22c4048 (DEFAULT_TABLE table=default.Table_b889b1ae5fb44f19a198a8ebabad297c [id=4369b0d36cc14f48a5f41f214aa4fe4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.817871 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbada3946dab4739a8369032ff3b8a48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.818776 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b
I20251027 21:07:47.819059 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3dfef8b741c42f68b664d7e0eab49c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.820963 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.802125 9608 tablet_service.cc:1552] Processing DeleteTablet for tablet a4c022969ea448cf933d5220f7ae8f66 with delete_type TABLET_DATA_DELETED (replaced by tablet d8ef795c88da49b58082ba1264d83378) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.802599 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 78a0a8b05f10477c923ab565839141b8 (DEFAULT_TABLE table=default.Table_108afd04fe5d45c195ba81996dee61d9 [id=02839636442f4955b245a06a594634c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.821818 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a0a8b05f10477c923ab565839141b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.822412 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 0c1a6a7616f24f69b83ec8a282746e92 (DEFAULT_TABLE table=default.Table_3886813b08244afa9a2d851978407392 [id=af6cb904f8cd4a929be7a978f9bc2001]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.822955 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c1a6a7616f24f69b83ec8a282746e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.832283 9608 tablet_service.cc:1505] Processing CreateTablet for tablet 0c1a6a7616f24f69b83ec8a282746e92 (DEFAULT_TABLE table=default.Table_3886813b08244afa9a2d851978407392 [id=af6cb904f8cd4a929be7a978f9bc2001]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.832952 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c1a6a7616f24f69b83ec8a282746e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.837229 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65fbcaa5447c48818fabfccac22c4048. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.837672 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 65fbcaa5447c48818fabfccac22c4048 (DEFAULT_TABLE table=default.Table_b889b1ae5fb44f19a198a8ebabad297c [id=4369b0d36cc14f48a5f41f214aa4fe4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.839426 9748 tablet_service.cc:1505] Processing CreateTablet for tablet e3c9f64ee0544e59add4f73bca59df98 (DEFAULT_TABLE table=default.Table_2225c7fa6a5a4d6eb28c8d58e77b4397 [id=0e1ad615b4494a6dbb8c1390c1b8416f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.839852 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3c9f64ee0544e59add4f73bca59df98. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.840689 9615 tablet_service.cc:1505] Processing CreateTablet for tablet e3c9f64ee0544e59add4f73bca59df98 (DEFAULT_TABLE table=default.Table_2225c7fa6a5a4d6eb28c8d58e77b4397 [id=0e1ad615b4494a6dbb8c1390c1b8416f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.841854 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.843174 9900 tablet_service.cc:1505] Processing CreateTablet for tablet 0c1a6a7616f24f69b83ec8a282746e92 (DEFAULT_TABLE table=default.Table_3886813b08244afa9a2d851978407392 [id=af6cb904f8cd4a929be7a978f9bc2001]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.843658 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c1a6a7616f24f69b83ec8a282746e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.846480 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.846858 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65fbcaa5447c48818fabfccac22c4048. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:47.847218 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b already in progress: creating tablet
I20251027 21:07:47.848655 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3c9f64ee0544e59add4f73bca59df98. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.855285 10031 tablet_bootstrap.cc:492] T 0cd4f5fa03d4417383df279da475345f P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:47.855496 10031 ts_tablet_manager.cc:1403] T 0cd4f5fa03d4417383df279da475345f P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.144s user 0.015s sys 0.013s
I20251027 21:07:47.856701 10031 raft_consensus.cc:359] T 0cd4f5fa03d4417383df279da475345f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.856984 10031 raft_consensus.cc:385] T 0cd4f5fa03d4417383df279da475345f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.857120 10031 raft_consensus.cc:740] T 0cd4f5fa03d4417383df279da475345f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.857209 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 69b1fb42428748239d8939d98428d8fe (DEFAULT_TABLE table=default.Table_5d5f327d728f4250841a9f4f10a2b533 [id=33dbf5fe986c415cb54fbf1b574651ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.857549 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69b1fb42428748239d8939d98428d8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.857525 10031 consensus_queue.cc:260] T 0cd4f5fa03d4417383df279da475345f P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.856101 9890 tablet_service.cc:1505] Processing CreateTablet for tablet e3c9f64ee0544e59add4f73bca59df98 (DEFAULT_TABLE table=default.Table_2225c7fa6a5a4d6eb28c8d58e77b4397 [id=0e1ad615b4494a6dbb8c1390c1b8416f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.858583 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3c9f64ee0544e59add4f73bca59df98. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.858896 10031 ts_tablet_manager.cc:1434] T 0cd4f5fa03d4417383df279da475345f P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:47.859153 10031 tablet_bootstrap.cc:492] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:47.861021 10029 tablet_bootstrap.cc:492] T 0cd4f5fa03d4417383df279da475345f P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:47.861235 10029 ts_tablet_manager.cc:1403] T 0cd4f5fa03d4417383df279da475345f P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.162s user 0.015s sys 0.015s
I20251027 21:07:47.861948 10029 raft_consensus.cc:359] T 0cd4f5fa03d4417383df279da475345f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.862211 10029 raft_consensus.cc:385] T 0cd4f5fa03d4417383df279da475345f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.862344 10029 raft_consensus.cc:740] T 0cd4f5fa03d4417383df279da475345f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.862648 10029 consensus_queue.cc:260] T 0cd4f5fa03d4417383df279da475345f P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.856534 9886 tablet_service.cc:1505] Processing CreateTablet for tablet 69b1fb42428748239d8939d98428d8fe (DEFAULT_TABLE table=default.Table_5d5f327d728f4250841a9f4f10a2b533 [id=33dbf5fe986c415cb54fbf1b574651ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.866640 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_35757fdc53ab433fb201f944c7906ff2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:47.867334 10043 ts_tablet_manager.cc:1929] T bc2b7a771a1647c290ed9c3e394c60fe P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251027 21:07:47.867344 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35757fdc53ab433fb201f944c7906ff2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.867472 10043 log.cc:1199] T bc2b7a771a1647c290ed9c3e394c60fe P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/bc2b7a771a1647c290ed9c3e394c60fe
I20251027 21:07:47.867693 10029 ts_tablet_manager.cc:1434] T 0cd4f5fa03d4417383df279da475345f P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20251027 21:07:47.867830 10043 ts_tablet_manager.cc:1950] T bc2b7a771a1647c290ed9c3e394c60fe P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:47.868784 10029 tablet_bootstrap.cc:492] T bd6230765d6448498132e24482100b39 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:47.872087 10029 tablet_bootstrap.cc:654] T bd6230765d6448498132e24482100b39 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.875581 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 44783760defd433a972c9cd83933249c (DEFAULT_TABLE table=default.Table_44e3f093c8d7476a98074b07828da3ab [id=30fbbddfaed646e3b2d80f788f774741]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.875984 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44783760defd433a972c9cd83933249c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.881094 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet 0cd4f5fa03d4417383df279da475345f from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:47.881377 9886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69b1fb42428748239d8939d98428d8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.882156 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cd4f5fa03d4417383df279da475345f with delete_type TABLET_DATA_DELETED (replaced by tablet d642da56a67449aa95932180bc5b2c5f) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.882387 10040 tablet_replica.cc:333] T 0cd4f5fa03d4417383df279da475345f P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:47.882591 10040 raft_consensus.cc:2243] T 0cd4f5fa03d4417383df279da475345f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.882748 10040 raft_consensus.cc:2272] T 0cd4f5fa03d4417383df279da475345f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:47.883870 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 69b1fb42428748239d8939d98428d8fe (DEFAULT_TABLE table=default.Table_5d5f327d728f4250841a9f4f10a2b533 [id=33dbf5fe986c415cb54fbf1b574651ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.884220 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69b1fb42428748239d8939d98428d8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.884529 10040 ts_tablet_manager.cc:1916] T 0cd4f5fa03d4417383df279da475345f P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:47.887171 10031 tablet_bootstrap.cc:654] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.888439 9306 master_service.cc:954] ReplaceTablet: received request to replace tablet 78a0a8b05f10477c923ab565839141b8 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:47.876865 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 44783760defd433a972c9cd83933249c (DEFAULT_TABLE table=default.Table_44e3f093c8d7476a98074b07828da3ab [id=30fbbddfaed646e3b2d80f788f774741]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.878392 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cd4f5fa03d4417383df279da475345f with delete_type TABLET_DATA_DELETED (replaced by tablet d642da56a67449aa95932180bc5b2c5f) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.889369 10032 tablet_bootstrap.cc:492] T bc2b7a771a1647c290ed9c3e394c60fe P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:47.889452 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 44783760defd433a972c9cd83933249c (DEFAULT_TABLE table=default.Table_44e3f093c8d7476a98074b07828da3ab [id=30fbbddfaed646e3b2d80f788f774741]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.889541 10032 ts_tablet_manager.cc:1403] T bc2b7a771a1647c290ed9c3e394c60fe P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.398s user 0.103s sys 0.118s
I20251027 21:07:47.890781 10041 ts_tablet_manager.cc:1929] T 3adbe5db3442462a8ae64a4b08129d69 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:47.890974 10041 log.cc:1199] T 3adbe5db3442462a8ae64a4b08129d69 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/3adbe5db3442462a8ae64a4b08129d69
I20251027 21:07:47.892755 10040 ts_tablet_manager.cc:1929] T 0cd4f5fa03d4417383df279da475345f P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:47.894245 10040 log.cc:1199] T 0cd4f5fa03d4417383df279da475345f P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/0cd4f5fa03d4417383df279da475345f
I20251027 21:07:47.894913 10040 ts_tablet_manager.cc:1950] T 0cd4f5fa03d4417383df279da475345f P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:47.895368 10032 raft_consensus.cc:359] T bc2b7a771a1647c290ed9c3e394c60fe P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.896687 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 0cd4f5fa03d4417383df279da475345f (table default.Table_178e5d554faa4e119ac8f52da2d284d7 [id=05edeb649c544c2f899ed42a87cef2c2]) successfully deleted
I20251027 21:07:47.897310 10032 raft_consensus.cc:385] T bc2b7a771a1647c290ed9c3e394c60fe P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.897768 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 78a0a8b05f10477c923ab565839141b8
I20251027 21:07:47.898093 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44783760defd433a972c9cd83933249c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.901079 10041 ts_tablet_manager.cc:1950] T 3adbe5db3442462a8ae64a4b08129d69 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:47.901346 10032 raft_consensus.cc:740] T bc2b7a771a1647c290ed9c3e394c60fe P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.902977 10032 consensus_queue.cc:260] T bc2b7a771a1647c290ed9c3e394c60fe P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.907299 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_780df15afd4d45d1bd65f9a2ede532c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.907449 9296 consensus_peers.cc:597] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 -> Peer 6e86ca5e83dd4df5b72f045e7b787e81 (127.9.14.124:45071): Couldn't send request to peer 6e86ca5e83dd4df5b72f045e7b787e81. Status: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 21:07:47.907408 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_98c734ca00af4e01a7da18accd5f6c30" } new_table_name: "default.Table_e68b56d3ddd94f308243ec6a03ea9e6e" modify_external_catalogs: true
W20251027 21:07:47.907899 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_780df15afd4d45d1bd65f9a2ede532c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.908485 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_84d985f1c9f345df85a4102d9c155a6e" } new_table_name: "default.Table_8c615d69dbaf418b9058374ee17309d5" modify_external_catalogs: true
I20251027 21:07:47.909303 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_02a777c596804daa888665c5e79ea115" } new_table_name: "default.Table_795e30871da94371958578747d1c898b" modify_external_catalogs: true
I20251027 21:07:47.909564 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 78a0a8b05f10477c923ab565839141b8 with delete_type TABLET_DATA_DELETED (replaced by tablet 1698ae8a37ad4560836b3b1daca1cf5e) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.910431 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_94fee319c82e4c53b9c5aa027e0c5718"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.911420 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94fee319c82e4c53b9c5aa027e0c5718 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.912377 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet 78a0a8b05f10477c923ab565839141b8 with delete_type TABLET_DATA_DELETED (replaced by tablet 1698ae8a37ad4560836b3b1daca1cf5e) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.912947 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 3adbe5db3442462a8ae64a4b08129d69 (table default.Table_40644b4bc09f4693866503d3b5fb7df7 [id=09c88f2f898844b88b70f5ed9e837841]) successfully deleted
I20251027 21:07:47.914551 9306 catalog_manager.cc:6719] ReplaceTablet: tablet 78a0a8b05f10477c923ab565839141b8 deleted and replaced by tablet 1698ae8a37ad4560836b3b1daca1cf5e
W20251027 21:07:47.915520 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 78a0a8b05f10477c923ab565839141b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78a0a8b05f10477c923ab565839141b8 already in progress: creating tablet
I20251027 21:07:47.916074 9306 master_service.cc:954] ReplaceTablet: received request to replace tablet 988052533a1042d9bba5ed5711d09d9c from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:47.916456 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet bd6230765d6448498132e24482100b39 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:47.920377 10032 ts_tablet_manager.cc:1434] T bc2b7a771a1647c290ed9c3e394c60fe P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20251027 21:07:47.920699 10032 tablet_bootstrap.cc:492] T a4c022969ea448cf933d5220f7ae8f66 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:07:47.922973 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 has no consensus state
W20251027 21:07:47.927947 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36e168cc44274dae9682b0c1329d4558 has no consensus state
W20251027 21:07:47.928011 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba8b125474bc42a5946e8f2dd718f7f4 consensus state has no leader
I20251027 21:07:47.929642 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44783760defd433a972c9cd83933249c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:47.932509 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 84305bcf21b64d2c8c1df015a2e9615c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84305bcf21b64d2c8c1df015a2e9615c already in progress: creating tablet
W20251027 21:07:47.932724 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 0029b70ed3ff4db9a816a6190d36bf88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0029b70ed3ff4db9a816a6190d36bf88 already in progress: creating tablet
I20251027 21:07:47.935796 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet bc2b7a771a1647c290ed9c3e394c60fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.938441 9308 master_service.cc:954] ReplaceTablet: received request to replace tablet 0c1a6a7616f24f69b83ec8a282746e92 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:47.938525 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d81917295a3548f787986bf566037719" } modify_external_catalogs: true
I20251027 21:07:47.938690 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d81917295a3548f787986bf566037719" } modify_external_catalogs: true
I20251027 21:07:47.939215 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4ea5cf4870a14209b590147d71a46a0f" } new_table_name: "default.Table_78302ea6dc094b199cfda33dd516a2ec" modify_external_catalogs: true
I20251027 21:07:47.939644 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_df95aebbdcf94d8289b235722b7d2c8c" } modify_external_catalogs: true
I20251027 21:07:47.939798 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_df95aebbdcf94d8289b235722b7d2c8c" } modify_external_catalogs: true
I20251027 21:07:47.940302 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d7a6b578109241ccb05e5ffc8df713b5" } new_table_name: "default.Table_59b38114c28e4a239c53d736dc162176" modify_external_catalogs: true
W20251027 21:07:47.941721 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 3218f18a797c40c9ad6ac64b2670cd19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3218f18a797c40c9ad6ac64b2670cd19 already in progress: creating tablet
W20251027 21:07:47.942075 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 84305bcf21b64d2c8c1df015a2e9615c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84305bcf21b64d2c8c1df015a2e9615c already in progress: creating tablet
W20251027 21:07:47.942320 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 0029b70ed3ff4db9a816a6190d36bf88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0029b70ed3ff4db9a816a6190d36bf88 already in progress: creating tablet
W20251027 21:07:47.942636 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 988052533a1042d9bba5ed5711d09d9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 988052533a1042d9bba5ed5711d09d9c already in progress: creating tablet
W20251027 21:07:47.942855 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 12aa6e3af0694429b45bef5af3592377 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12aa6e3af0694429b45bef5af3592377 already in progress: creating tablet
W20251027 21:07:47.943081 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet b3dfef8b741c42f68b664d7e0eab49c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3dfef8b741c42f68b664d7e0eab49c6 already in progress: creating tablet
W20251027 21:07:47.943280 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 783d845009d54d20b8e6626c3e73d2a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 783d845009d54d20b8e6626c3e73d2a7 already in progress: creating tablet
W20251027 21:07:47.943509 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b already in progress: creating tablet
I20251027 21:07:47.943531 10043 tablet_replica.cc:333] T a4c022969ea448cf933d5220f7ae8f66 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:47.943549 10032 tablet_bootstrap.cc:654] T a4c022969ea448cf933d5220f7ae8f66 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.943701 10043 raft_consensus.cc:2243] T a4c022969ea448cf933d5220f7ae8f66 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.943939 10043 raft_consensus.cc:2272] T a4c022969ea448cf933d5220f7ae8f66 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:47.944504 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d7caf208b9864991a0e50ced803eb60d" } modify_external_catalogs: true
I20251027 21:07:47.944600 10043 ts_tablet_manager.cc:1916] T a4c022969ea448cf933d5220f7ae8f66 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:47.944691 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d7caf208b9864991a0e50ced803eb60d" } modify_external_catalogs: true
I20251027 21:07:47.945236 10031 tablet_bootstrap.cc:492] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:47.945408 10031 ts_tablet_manager.cc:1403] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.086s user 0.002s sys 0.003s
W20251027 21:07:47.947808 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b1bdb2be5032415187c2826c14e75f07 consensus state has no leader
I20251027 21:07:47.948230 9753 tablet_service.cc:1505] Processing CreateTablet for tablet b1bdb2be5032415187c2826c14e75f07 (DEFAULT_TABLE table=default.Table_4ea5cf4870a14209b590147d71a46a0f [id=d7dcec1e03494fd5a219a97ea8b1ab1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.946208 10031 raft_consensus.cc:359] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.948626 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1bdb2be5032415187c2826c14e75f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.950845 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet 78a0a8b05f10477c923ab565839141b8 with delete_type TABLET_DATA_DELETED (replaced by tablet 1698ae8a37ad4560836b3b1daca1cf5e) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.951701 9608 tablet_service.cc:1505] Processing CreateTablet for tablet b1bdb2be5032415187c2826c14e75f07 (DEFAULT_TABLE table=default.Table_78302ea6dc094b199cfda33dd516a2ec [id=d7dcec1e03494fd5a219a97ea8b1ab1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.952944 9896 tablet_service.cc:1505] Processing CreateTablet for tablet b1bdb2be5032415187c2826c14e75f07 (DEFAULT_TABLE table=default.Table_78302ea6dc094b199cfda33dd516a2ec [id=d7dcec1e03494fd5a219a97ea8b1ab1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.952206 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1bdb2be5032415187c2826c14e75f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.954099 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1bdb2be5032415187c2826c14e75f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.955502 9749 tablet_service.cc:1505] Processing CreateTablet for tablet e5d8837ff3c44a66a1f933d80540ba0c (DEFAULT_TABLE table=default.Table_6b913111f04c4f1abe4b7244040b07d3 [id=7168e50c0645436b8c706086190b1736]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.955935 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d8837ff3c44a66a1f933d80540ba0c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:47.955951 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet a4c022969ea448cf933d5220f7ae8f66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4c022969ea448cf933d5220f7ae8f66 already in progress: creating tablet
W20251027 21:07:47.957326 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet b3dfef8b741c42f68b664d7e0eab49c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3dfef8b741c42f68b664d7e0eab49c6 already in progress: creating tablet
W20251027 21:07:47.958042 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 783d845009d54d20b8e6626c3e73d2a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 783d845009d54d20b8e6626c3e73d2a7 already in progress: creating tablet
I20251027 21:07:47.958268 10031 raft_consensus.cc:385] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.958398 10031 raft_consensus.cc:740] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.958657 10031 consensus_queue.cc:260] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:47.958822 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 12aa6e3af0694429b45bef5af3592377 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12aa6e3af0694429b45bef5af3592377 already in progress: creating tablet
W20251027 21:07:47.959702 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 988052533a1042d9bba5ed5711d09d9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 988052533a1042d9bba5ed5711d09d9c already in progress: creating tablet
W20251027 21:07:47.960543 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b already in progress: creating tablet
I20251027 21:07:47.960995 10031 ts_tablet_manager.cc:1434] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20251027 21:07:47.961093 10041 tablet_replica.cc:333] T bc2b7a771a1647c290ed9c3e394c60fe P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:47.961282 10041 raft_consensus.cc:2243] T bc2b7a771a1647c290ed9c3e394c60fe P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:47.961459 10041 raft_consensus.cc:2272] T bc2b7a771a1647c290ed9c3e394c60fe P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:47.962265 10031 tablet_bootstrap.cc:492] T a878184a3a444761aff8730bdf43ccaf P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:47.962756 9895 tablet_service.cc:1505] Processing CreateTablet for tablet e5d8837ff3c44a66a1f933d80540ba0c (DEFAULT_TABLE table=default.Table_6b913111f04c4f1abe4b7244040b07d3 [id=7168e50c0645436b8c706086190b1736]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.962924 10041 ts_tablet_manager.cc:1916] T bc2b7a771a1647c290ed9c3e394c60fe P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:47.963214 10029 tablet_bootstrap.cc:492] T bd6230765d6448498132e24482100b39 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:47.963469 10029 ts_tablet_manager.cc:1403] T bd6230765d6448498132e24482100b39 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.095s user 0.001s sys 0.004s
I20251027 21:07:47.963920 9615 tablet_service.cc:1505] Processing CreateTablet for tablet e5d8837ff3c44a66a1f933d80540ba0c (DEFAULT_TABLE table=default.Table_6b913111f04c4f1abe4b7244040b07d3 [id=7168e50c0645436b8c706086190b1736]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.964288 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d8837ff3c44a66a1f933d80540ba0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.964826 10029 raft_consensus.cc:359] T bd6230765d6448498132e24482100b39 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.965138 10029 raft_consensus.cc:385] T bd6230765d6448498132e24482100b39 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.965325 10029 raft_consensus.cc:740] T bd6230765d6448498132e24482100b39 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.965692 10029 consensus_queue.cc:260] T bd6230765d6448498132e24482100b39 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.963109 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d8837ff3c44a66a1f933d80540ba0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.968968 10032 tablet_bootstrap.cc:492] T a4c022969ea448cf933d5220f7ae8f66 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:47.969159 10032 ts_tablet_manager.cc:1403] T a4c022969ea448cf933d5220f7ae8f66 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.049s user 0.000s sys 0.008s
W20251027 21:07:47.969949 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 21b785277d604543bfa7816ed96d6181 consensus state has no leader
I20251027 21:07:47.969961 10032 raft_consensus.cc:359] T a4c022969ea448cf933d5220f7ae8f66 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.970211 10032 raft_consensus.cc:385] T a4c022969ea448cf933d5220f7ae8f66 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:47.970348 10032 raft_consensus.cc:740] T a4c022969ea448cf933d5220f7ae8f66 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:47.970671 10032 consensus_queue.cc:260] T a4c022969ea448cf933d5220f7ae8f66 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:47.974083 10032 ts_tablet_manager.cc:1434] T a4c022969ea448cf933d5220f7ae8f66 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251027 21:07:47.975293 10032 tablet_bootstrap.cc:492] T a878184a3a444761aff8730bdf43ccaf P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:47.978392 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 0c1a6a7616f24f69b83ec8a282746e92
I20251027 21:07:47.978613 10032 tablet_bootstrap.cc:654] T a878184a3a444761aff8730bdf43ccaf P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:47.978976 9308 catalog_manager.cc:6719] ReplaceTablet: tablet 0c1a6a7616f24f69b83ec8a282746e92 deleted and replaced by tablet 2d659e4d73ea4ff3975a4b58a794090a
I20251027 21:07:47.979295 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c1a6a7616f24f69b83ec8a282746e92 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d659e4d73ea4ff3975a4b58a794090a) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.979485 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 36e168cc44274dae9682b0c1329d4558 (DEFAULT_TABLE table=default.Table_795e30871da94371958578747d1c898b [id=9a111e9480b84a55b5226f0940d999c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.980223 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e168cc44274dae9682b0c1329d4558. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.981024 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 63662f175a1c450f8f67f37da971db37
I20251027 21:07:47.981746 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet 63662f175a1c450f8f67f37da971db37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:47.983305 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 (DEFAULT_TABLE table=default.Table_e68b56d3ddd94f308243ec6a03ea9e6e [id=0a94e88b69094ffe8be3367470be86da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.983717 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35e5b7bbab7f4ee9bbf23d913c0aee76. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.983649 9612 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c1a6a7616f24f69b83ec8a282746e92 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d659e4d73ea4ff3975a4b58a794090a) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.984290 9612 tablet_service.cc:1552] Processing DeleteTablet for tablet 63662f175a1c450f8f67f37da971db37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.985466 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_18e48b818fc040bfa4272f08e203dc3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.986057 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18e48b818fc040bfa4272f08e203dc3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.986881 9612 tablet_service.cc:1505] Processing CreateTablet for tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 (DEFAULT_TABLE table=default.Table_e68b56d3ddd94f308243ec6a03ea9e6e [id=0a94e88b69094ffe8be3367470be86da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.987205 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35e5b7bbab7f4ee9bbf23d913c0aee76. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.988448 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a356e5bd7b1c40a087ef83c7078e0381"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.988885 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a356e5bd7b1c40a087ef83c7078e0381 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.991204 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 36e168cc44274dae9682b0c1329d4558 (DEFAULT_TABLE table=default.Table_795e30871da94371958578747d1c898b [id=9a111e9480b84a55b5226f0940d999c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.991504 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f05923c6594b43378fa156d0ab2bd1da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.992058 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f05923c6594b43378fa156d0ab2bd1da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.992118 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e168cc44274dae9682b0c1329d4558. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.993767 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet bd6230765d6448498132e24482100b39 with delete_type TABLET_DATA_DELETED (replaced by tablet 84305bcf21b64d2c8c1df015a2e9615c) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:47.995157 10029 ts_tablet_manager.cc:1434] T bd6230765d6448498132e24482100b39 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.032s user 0.000s sys 0.002s
I20251027 21:07:47.975220 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.979483 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 36e168cc44274dae9682b0c1329d4558 (DEFAULT_TABLE table=default.Table_795e30871da94371958578747d1c898b [id=9a111e9480b84a55b5226f0940d999c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.979997 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c1a6a7616f24f69b83ec8a282746e92 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d659e4d73ea4ff3975a4b58a794090a) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.981998 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet 63662f175a1c450f8f67f37da971db37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:47.995747 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e168cc44274dae9682b0c1329d4558. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.985847 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 (DEFAULT_TABLE table=default.Table_e68b56d3ddd94f308243ec6a03ea9e6e [id=0a94e88b69094ffe8be3367470be86da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:47.996302 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35e5b7bbab7f4ee9bbf23d913c0aee76. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:47.997138 10043 ts_tablet_manager.cc:1929] T a4c022969ea448cf933d5220f7ae8f66 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:47.997267 10043 log.cc:1199] T a4c022969ea448cf933d5220f7ae8f66 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/a4c022969ea448cf933d5220f7ae8f66
I20251027 21:07:47.997619 10043 ts_tablet_manager.cc:1950] T a4c022969ea448cf933d5220f7ae8f66 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:47.997864 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_bee0cc21eb6f4fe9a9b2d4ab738a3fe9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:47.998620 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bee0cc21eb6f4fe9a9b2d4ab738a3fe9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:47.999863 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_108afd04fe5d45c195ba81996dee61d9" } modify_external_catalogs: true
I20251027 21:07:48.000125 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_108afd04fe5d45c195ba81996dee61d9" } modify_external_catalogs: true
I20251027 21:07:48.001844 10041 ts_tablet_manager.cc:1929] T bc2b7a771a1647c290ed9c3e394c60fe P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.001987 10041 log.cc:1199] T bc2b7a771a1647c290ed9c3e394c60fe P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/bc2b7a771a1647c290ed9c3e394c60fe
I20251027 21:07:48.002372 10041 ts_tablet_manager.cc:1950] T bc2b7a771a1647c290ed9c3e394c60fe P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
W20251027 21:07:48.002332 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 0c1a6a7616f24f69b83ec8a282746e92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c1a6a7616f24f69b83ec8a282746e92 already in progress: creating tablet
I20251027 21:07:48.002395 10040 tablet_replica.cc:333] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:48.002571 10040 raft_consensus.cc:2243] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251027 21:07:48.002655 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 63662f175a1c450f8f67f37da971db37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63662f175a1c450f8f67f37da971db37 already in progress: creating tablet
I20251027 21:07:48.002755 10040 raft_consensus.cc:2272] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.003412 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet bc2b7a771a1647c290ed9c3e394c60fe (table default.Table_e46f2bc398e145209af5db2546ad90f7 [id=5280a5a662394d82b01929699833a1dc]) successfully deleted
W20251027 21:07:48.003664 9298 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet ba8b125474bc42a5946e8f2dd718f7f4 (table default.Table_8c615d69dbaf418b9058374ee17309d5 [id=e7b7bdb2ea7f468c98bfe88f900a5be8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ba8b125474bc42a5946e8f2dd718f7f4 consensus state has no leader
I20251027 21:07:48.004491 10040 ts_tablet_manager.cc:1916] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:48.005945 9895 tablet_service.cc:1505] Processing CreateTablet for tablet cad3a3e910ac42399cf371ad82c89ff7 (DEFAULT_TABLE table=default.Table_35757fdc53ab433fb201f944c7906ff2 [id=0286a1b3137649eeaa5339cbdf97aa03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.007465 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cad3a3e910ac42399cf371ad82c89ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.007555 10031 tablet_bootstrap.cc:654] T a878184a3a444761aff8730bdf43ccaf P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.007411 9898 tablet_service.cc:1505] Processing CreateTablet for tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe (DEFAULT_TABLE table=default.Table_4c5433970bd64291a2b1289a84f35590 [id=a1b2d46db3764c7f97712a0c5f891881]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.008127 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.012094 9751 tablet_service.cc:1505] Processing CreateTablet for tablet 9c4369a10bf846fb94c9b71a70f21600 (DEFAULT_TABLE table=default.Table_1764f51a79024aac9bcd15e3f03fd314 [id=731981b3bb2c4078a597da73199e31d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.012132 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 9c4369a10bf846fb94c9b71a70f21600 (DEFAULT_TABLE table=default.Table_1764f51a79024aac9bcd15e3f03fd314 [id=731981b3bb2c4078a597da73199e31d8]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:48.012194 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 78a0a8b05f10477c923ab565839141b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78a0a8b05f10477c923ab565839141b8 already in progress: creating tablet
I20251027 21:07:48.012379 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet a4c022969ea448cf933d5220f7ae8f66 (table default.Table_df95aebbdcf94d8289b235722b7d2c8c [id=98143d42942742568f2c3a1d0fbcc100]) successfully deleted
I20251027 21:07:48.012492 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c4369a10bf846fb94c9b71a70f21600. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.012501 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c4369a10bf846fb94c9b71a70f21600. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:48.012455 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 0c1a6a7616f24f69b83ec8a282746e92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c1a6a7616f24f69b83ec8a282746e92 already in progress: creating tablet
W20251027 21:07:48.012687 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 63662f175a1c450f8f67f37da971db37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63662f175a1c450f8f67f37da971db37 already in progress: creating tablet
I20251027 21:07:48.013348 9748 tablet_service.cc:1505] Processing CreateTablet for tablet cad3a3e910ac42399cf371ad82c89ff7 (DEFAULT_TABLE table=default.Table_35757fdc53ab433fb201f944c7906ff2 [id=0286a1b3137649eeaa5339cbdf97aa03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.013625 9890 tablet_service.cc:1505] Processing CreateTablet for tablet d8ef795c88da49b58082ba1264d83378 (DEFAULT_TABLE table=default.Table_df95aebbdcf94d8289b235722b7d2c8c [id=98143d42942742568f2c3a1d0fbcc100]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.013777 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cad3a3e910ac42399cf371ad82c89ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.013984 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ef795c88da49b58082ba1264d83378. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.016256 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_429373058a504f3b859126b0a8077aab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.016767 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_429373058a504f3b859126b0a8077aab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.018464 9753 tablet_service.cc:1505] Processing CreateTablet for tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe (DEFAULT_TABLE table=default.Table_4c5433970bd64291a2b1289a84f35590 [id=a1b2d46db3764c7f97712a0c5f891881]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.018848 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.019184 10029 tablet_bootstrap.cc:492] T 988052533a1042d9bba5ed5711d09d9c P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:48.021042 10029 tablet_bootstrap.cc:654] T 988052533a1042d9bba5ed5711d09d9c P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.021726 9612 tablet_service.cc:1505] Processing CreateTablet for tablet d8ef795c88da49b58082ba1264d83378 (DEFAULT_TABLE table=default.Table_df95aebbdcf94d8289b235722b7d2c8c [id=98143d42942742568f2c3a1d0fbcc100]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.022158 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ef795c88da49b58082ba1264d83378. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.022364 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_35e8a94e410b4cf99228b2ec46855088"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.022867 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35e8a94e410b4cf99228b2ec46855088 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.024578 10032 tablet_bootstrap.cc:492] T a878184a3a444761aff8730bdf43ccaf P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:48.024745 10032 ts_tablet_manager.cc:1403] T a878184a3a444761aff8730bdf43ccaf P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.050s user 0.000s sys 0.006s
I20251027 21:07:48.025499 10032 raft_consensus.cc:359] T a878184a3a444761aff8730bdf43ccaf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.025674 10032 raft_consensus.cc:385] T a878184a3a444761aff8730bdf43ccaf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.025772 10032 raft_consensus.cc:740] T a878184a3a444761aff8730bdf43ccaf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.026062 10032 consensus_queue.cc:260] T a878184a3a444761aff8730bdf43ccaf P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.028332 9608 tablet_service.cc:1505] Processing CreateTablet for tablet cad3a3e910ac42399cf371ad82c89ff7 (DEFAULT_TABLE table=default.Table_35757fdc53ab433fb201f944c7906ff2 [id=0286a1b3137649eeaa5339cbdf97aa03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.028707 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cad3a3e910ac42399cf371ad82c89ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.028957 10043 tablet_replica.cc:333] T 0cd4f5fa03d4417383df279da475345f P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:48.029206 10043 raft_consensus.cc:2243] T 0cd4f5fa03d4417383df279da475345f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:48.029320 10032 ts_tablet_manager.cc:1434] T a878184a3a444761aff8730bdf43ccaf P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251027 21:07:48.030030 9750 tablet_service.cc:1505] Processing CreateTablet for tablet d8ef795c88da49b58082ba1264d83378 (DEFAULT_TABLE table=default.Table_df95aebbdcf94d8289b235722b7d2c8c [id=98143d42942742568f2c3a1d0fbcc100]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.030411 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ef795c88da49b58082ba1264d83378. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.030818 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 94a00ff844f0415fa419613d70a318e6 (DEFAULT_TABLE table=default.Table_8c58943248a74b5ab81a5b742eafc5b4 [id=f773d65a07a4473ab8b87cd82ac50bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.031145 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94a00ff844f0415fa419613d70a318e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.030918 10043 raft_consensus.cc:2272] T 0cd4f5fa03d4417383df279da475345f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.034804 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 94a00ff844f0415fa419613d70a318e6 (DEFAULT_TABLE table=default.Table_8c58943248a74b5ab81a5b742eafc5b4 [id=f773d65a07a4473ab8b87cd82ac50bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.035229 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94a00ff844f0415fa419613d70a318e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.035816 9612 tablet_service.cc:1505] Processing CreateTablet for tablet 94a00ff844f0415fa419613d70a318e6 (DEFAULT_TABLE table=default.Table_8c58943248a74b5ab81a5b742eafc5b4 [id=f773d65a07a4473ab8b87cd82ac50bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.035998 10032 tablet_bootstrap.cc:492] T bd6230765d6448498132e24482100b39 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:48.036336 10043 ts_tablet_manager.cc:1916] T 0cd4f5fa03d4417383df279da475345f P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:48.037438 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 61cd8ee8e1f8459eb7af5dd48c018e02 (DEFAULT_TABLE table=default.Table_52c53bafe66b4c66b99e5b26e4df7ef5 [id=27dc835f316f41e091be9b2b1f560bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.037833 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61cd8ee8e1f8459eb7af5dd48c018e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.037853 10032 tablet_bootstrap.cc:654] T bd6230765d6448498132e24482100b39 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.036469 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94a00ff844f0415fa419613d70a318e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.046435 10032 tablet_bootstrap.cc:492] T bd6230765d6448498132e24482100b39 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:48.046603 10032 ts_tablet_manager.cc:1403] T bd6230765d6448498132e24482100b39 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20251027 21:07:48.047299 10032 raft_consensus.cc:359] T bd6230765d6448498132e24482100b39 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.047488 10032 raft_consensus.cc:385] T bd6230765d6448498132e24482100b39 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.047562 10032 raft_consensus.cc:740] T bd6230765d6448498132e24482100b39 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.047748 10040 ts_tablet_manager.cc:1929] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.047777 10032 consensus_queue.cc:260] T bd6230765d6448498132e24482100b39 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.047890 10040 log.cc:1199] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/e9dd86d98cd94cfe9cc2a7b1ab1f979b
I20251027 21:07:48.048346 10040 ts_tablet_manager.cc:1950] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:48.049383 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b (table default.Table_0377267e434442dfb2e1cce27b28dd35 [id=7a31bdff8eab4483b0d88c59e7a2e914]) successfully deleted
I20251027 21:07:48.050515 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 61cd8ee8e1f8459eb7af5dd48c018e02 (DEFAULT_TABLE table=default.Table_52c53bafe66b4c66b99e5b26e4df7ef5 [id=27dc835f316f41e091be9b2b1f560bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.050953 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61cd8ee8e1f8459eb7af5dd48c018e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.052502 10045 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:48.052724 10045 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: This master's current role is: LEADER
I20251027 21:07:48.052670 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 61cd8ee8e1f8459eb7af5dd48c018e02 (DEFAULT_TABLE table=default.Table_52c53bafe66b4c66b99e5b26e4df7ef5 [id=27dc835f316f41e091be9b2b1f560bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.053136 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61cd8ee8e1f8459eb7af5dd48c018e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.057770 10032 ts_tablet_manager.cc:1434] T bd6230765d6448498132e24482100b39 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251027 21:07:48.061079 10032 tablet_bootstrap.cc:492] T 0cd4f5fa03d4417383df279da475345f P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:48.063025 10032 tablet_bootstrap.cc:654] T 0cd4f5fa03d4417383df279da475345f P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.064195 10039 raft_consensus.cc:493] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:48.064363 10039 raft_consensus.cc:515] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.065101 10039 leader_election.cc:290] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:48.065759 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a878184a3a444761aff8730bdf43ccaf" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:48.066009 9925 raft_consensus.cc:2468] T a878184a3a444761aff8730bdf43ccaf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:48.066556 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 9c4369a10bf846fb94c9b71a70f21600 (DEFAULT_TABLE table=default.Table_1764f51a79024aac9bcd15e3f03fd314 [id=731981b3bb2c4078a597da73199e31d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.066725 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a878184a3a444761aff8730bdf43ccaf" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:48.067495 9566 leader_election.cc:343] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:48.067889 9568 leader_election.cc:304] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:48.070966 10032 tablet_bootstrap.cc:492] T 0cd4f5fa03d4417383df279da475345f P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:48.071106 10032 ts_tablet_manager.cc:1403] T 0cd4f5fa03d4417383df279da475345f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20251027 21:07:48.071940 10032 raft_consensus.cc:359] T 0cd4f5fa03d4417383df279da475345f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.072101 10032 raft_consensus.cc:385] T 0cd4f5fa03d4417383df279da475345f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.072189 10032 raft_consensus.cc:740] T 0cd4f5fa03d4417383df279da475345f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.072422 10032 consensus_queue.cc:260] T 0cd4f5fa03d4417383df279da475345f P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.075527 10039 raft_consensus.cc:2804] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:48.075685 10039 raft_consensus.cc:493] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:48.075832 10039 raft_consensus.cc:3060] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:48.079885 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c4369a10bf846fb94c9b71a70f21600. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.082096 10039 raft_consensus.cc:515] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.082017 9606 tablet_service.cc:1505] Processing CreateTablet for tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe (DEFAULT_TABLE table=default.Table_4c5433970bd64291a2b1289a84f35590 [id=a1b2d46db3764c7f97712a0c5f891881]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.083735 10043 ts_tablet_manager.cc:1929] T 0cd4f5fa03d4417383df279da475345f P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.086560 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a878184a3a444761aff8730bdf43ccaf" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:07:48.087172 9566 leader_election.cc:343] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:48.087539 10032 ts_tablet_manager.cc:1434] T 0cd4f5fa03d4417383df279da475345f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251027 21:07:48.087790 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a878184a3a444761aff8730bdf43ccaf" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:07:48.092249 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b0f0589285ac4a56b7a2b23ace6248f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:48.092909 10039 leader_election.cc:290] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:07:48.092979 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0f0589285ac4a56b7a2b23ace6248f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.093107 10043 log.cc:1199] T 0cd4f5fa03d4417383df279da475345f P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/0cd4f5fa03d4417383df279da475345f
I20251027 21:07:48.093515 10043 ts_tablet_manager.cc:1950] T 0cd4f5fa03d4417383df279da475345f P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:48.094838 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet d8ef795c88da49b58082ba1264d83378
I20251027 21:07:48.095059 10043 tablet_replica.cc:333] T bd6230765d6448498132e24482100b39 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:48.095216 10043 raft_consensus.cc:2243] T bd6230765d6448498132e24482100b39 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:48.095381 10043 raft_consensus.cc:2272] T bd6230765d6448498132e24482100b39 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.095546 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.095578 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_50cddb609f3043f29b42899ad521fbee" } modify_external_catalogs: true
I20251027 21:07:48.095644 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_63c557c9df2c4f4888317b03f119c607"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:48.095796 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_50cddb609f3043f29b42899ad521fbee" } modify_external_catalogs: true
W20251027 21:07:48.096165 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63c557c9df2c4f4888317b03f119c607 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.101055 10043 ts_tablet_manager.cc:1916] T bd6230765d6448498132e24482100b39 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:48.101233 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 6e5d52b7ec114be5ae68e57280c2eea8
I20251027 21:07:48.101763 9306 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 1698ae8a37ad4560836b3b1daca1cf5e
I20251027 21:07:48.103394 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 0cd4f5fa03d4417383df279da475345f (table default.Table_178e5d554faa4e119ac8f52da2d284d7 [id=05edeb649c544c2f899ed42a87cef2c2]) successfully deleted
W20251027 21:07:48.107299 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 0c1a6a7616f24f69b83ec8a282746e92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c1a6a7616f24f69b83ec8a282746e92 already in progress: creating tablet
I20251027 21:07:48.106968 9608 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e5d52b7ec114be5ae68e57280c2eea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:48.107496 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 63662f175a1c450f8f67f37da971db37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63662f175a1c450f8f67f37da971db37 already in progress: creating tablet
I20251027 21:07:48.110069 10043 ts_tablet_manager.cc:1929] T bd6230765d6448498132e24482100b39 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.110265 10043 log.cc:1199] T bd6230765d6448498132e24482100b39 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/bd6230765d6448498132e24482100b39
I20251027 21:07:48.110782 10043 ts_tablet_manager.cc:1950] T bd6230765d6448498132e24482100b39 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:48.111251 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet d8ef795c88da49b58082ba1264d83378 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.113662 9309 master_service.cc:954] ReplaceTablet: received request to replace tablet 94a00ff844f0415fa419613d70a318e6 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:48.113631 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet bd6230765d6448498132e24482100b39 (table default.Table_41a04598f12247fbac5f4103d2cbf22d [id=2e28a4d0f9204351b93335736a9abe18]) successfully deleted
W20251027 21:07:48.113898 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 6e5d52b7ec114be5ae68e57280c2eea8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e5d52b7ec114be5ae68e57280c2eea8 already in progress: creating tablet
I20251027 21:07:48.114164 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e5d52b7ec114be5ae68e57280c2eea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:48.114270 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet d8ef795c88da49b58082ba1264d83378 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8ef795c88da49b58082ba1264d83378 already in progress: creating tablet
W20251027 21:07:48.115188 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 6e5d52b7ec114be5ae68e57280c2eea8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e5d52b7ec114be5ae68e57280c2eea8 already in progress: creating tablet
I20251027 21:07:48.117689 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet f5da8d7ecccc4c3492f9d0d30ffea423
I20251027 21:07:48.120144 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet f5da8d7ecccc4c3492f9d0d30ffea423 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:48.120663 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet f5da8d7ecccc4c3492f9d0d30ffea423 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5da8d7ecccc4c3492f9d0d30ffea423 already in progress: creating tablet
I20251027 21:07:48.125358 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet 69b1fb42428748239d8939d98428d8fe from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:48.128912 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet f5da8d7ecccc4c3492f9d0d30ffea423 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:48.129441 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet f5da8d7ecccc4c3492f9d0d30ffea423 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5da8d7ecccc4c3492f9d0d30ffea423 already in progress: creating tablet
I20251027 21:07:48.130234 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet d8ef795c88da49b58082ba1264d83378 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:48.130445 9308 master_service.cc:954] ReplaceTablet: received request to replace tablet b3dfef8b741c42f68b664d7e0eab49c6 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:48.130445 9306 master_service.cc:954] ReplaceTablet: received request to replace tablet 3218f18a797c40c9ad6ac64b2670cd19 from {username='slave'} at 127.0.0.1:41710
W20251027 21:07:48.131372 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet d8ef795c88da49b58082ba1264d83378 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8ef795c88da49b58082ba1264d83378 already in progress: creating tablet
I20251027 21:07:48.133190 10032 tablet_bootstrap.cc:492] T 988052533a1042d9bba5ed5711d09d9c P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:48.134264 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet a4c022969ea448cf933d5220f7ae8f66 with delete_type TABLET_DATA_DELETED (replaced by tablet d8ef795c88da49b58082ba1264d83378) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.134501 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e5d52b7ec114be5ae68e57280c2eea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.134703 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet d8ef795c88da49b58082ba1264d83378 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.134871 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet f5da8d7ecccc4c3492f9d0d30ffea423 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.135684 10032 tablet_bootstrap.cc:654] T 988052533a1042d9bba5ed5711d09d9c P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.137574 10041 tablet_replica.cc:333] T a4c022969ea448cf933d5220f7ae8f66 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:48.137756 10041 raft_consensus.cc:2243] T a4c022969ea448cf933d5220f7ae8f66 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:48.137940 10041 raft_consensus.cc:2272] T a4c022969ea448cf933d5220f7ae8f66 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.134217 9925 raft_consensus.cc:3060] T a878184a3a444761aff8730bdf43ccaf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:48.139179 10031 tablet_bootstrap.cc:492] T a878184a3a444761aff8730bdf43ccaf P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:48.139752 10031 ts_tablet_manager.cc:1403] T a878184a3a444761aff8730bdf43ccaf P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.178s user 0.010s sys 0.054s
I20251027 21:07:48.140115 10041 ts_tablet_manager.cc:1916] T a4c022969ea448cf933d5220f7ae8f66 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:48.141940 10031 raft_consensus.cc:359] T a878184a3a444761aff8730bdf43ccaf P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.142117 10031 raft_consensus.cc:385] T a878184a3a444761aff8730bdf43ccaf P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.142210 10031 raft_consensus.cc:740] T a878184a3a444761aff8730bdf43ccaf P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.142434 10031 consensus_queue.cc:260] T a878184a3a444761aff8730bdf43ccaf P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.142812 9925 raft_consensus.cc:2468] T a878184a3a444761aff8730bdf43ccaf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:48.144133 10031 ts_tablet_manager.cc:1434] T a878184a3a444761aff8730bdf43ccaf P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251027 21:07:48.144400 10031 tablet_bootstrap.cc:492] T bd6230765d6448498132e24482100b39 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:48.145815 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_94fee319c82e4c53b9c5aa027e0c5718" } new_table_name: "default.Table_a7f999b292d04332b831f454d4aa6743" modify_external_catalogs: true
I20251027 21:07:48.146217 10031 tablet_bootstrap.cc:654] T bd6230765d6448498132e24482100b39 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.146318 9568 leader_election.cc:304] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:48.146734 10039 raft_consensus.cc:2804] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:48.147245 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2225c7fa6a5a4d6eb28c8d58e77b4397" } new_table_name: "default.Table_79febe3afdb94e8781e2633f126402dc" modify_external_catalogs: true
I20251027 21:07:48.148705 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3b2e962c9ca9424dad8aaa7950efbf63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.149286 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b2e962c9ca9424dad8aaa7950efbf63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.154345 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_429373058a504f3b859126b0a8077aab" } new_table_name: "default.Table_23cfb97e14e443a1bc6763bc466837a6" modify_external_catalogs: true
I20251027 21:07:48.155158 10041 ts_tablet_manager.cc:1929] T a4c022969ea448cf933d5220f7ae8f66 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.155315 10041 log.cc:1199] T a4c022969ea448cf933d5220f7ae8f66 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/a4c022969ea448cf933d5220f7ae8f66
I20251027 21:07:48.155793 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_096b7cf08dbf49279d8d5c11b3b687ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.156316 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_096b7cf08dbf49279d8d5c11b3b687ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.156502 10041 ts_tablet_manager.cc:1950] T a4c022969ea448cf933d5220f7ae8f66 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:48.158744 10029 tablet_bootstrap.cc:492] T 988052533a1042d9bba5ed5711d09d9c P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:48.159139 10029 ts_tablet_manager.cc:1403] T 988052533a1042d9bba5ed5711d09d9c P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.140s user 0.022s sys 0.035s
I20251027 21:07:48.160442 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet a4c022969ea448cf933d5220f7ae8f66 (table default.Table_df95aebbdcf94d8289b235722b7d2c8c [id=98143d42942742568f2c3a1d0fbcc100]) successfully deleted
I20251027 21:07:48.161123 10039 raft_consensus.cc:697] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
W20251027 21:07:48.161265 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 6e5d52b7ec114be5ae68e57280c2eea8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e5d52b7ec114be5ae68e57280c2eea8 already in progress: creating tablet
I20251027 21:07:48.161561 10039 consensus_queue.cc:237] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.162226 10029 raft_consensus.cc:359] T 988052533a1042d9bba5ed5711d09d9c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.162508 10029 raft_consensus.cc:385] T 988052533a1042d9bba5ed5711d09d9c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.162606 10029 raft_consensus.cc:740] T 988052533a1042d9bba5ed5711d09d9c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
W20251027 21:07:48.163446 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet d8ef795c88da49b58082ba1264d83378 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8ef795c88da49b58082ba1264d83378 already in progress: creating tablet
W20251027 21:07:48.164832 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet f5da8d7ecccc4c3492f9d0d30ffea423 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5da8d7ecccc4c3492f9d0d30ffea423 already in progress: creating tablet
I20251027 21:07:48.166709 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b889b1ae5fb44f19a198a8ebabad297c" } new_table_name: "default.Table_c93601bc03f1438496c9610e8c23b5a6" modify_external_catalogs: true
I20251027 21:07:48.166050 10029 consensus_queue.cc:260] T 988052533a1042d9bba5ed5711d09d9c P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.167419 10029 ts_tablet_manager.cc:1434] T 988052533a1042d9bba5ed5711d09d9c P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251027 21:07:48.167793 10029 tablet_bootstrap.cc:492] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:48.169540 10032 tablet_bootstrap.cc:492] T 988052533a1042d9bba5ed5711d09d9c P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:48.169705 10032 ts_tablet_manager.cc:1403] T 988052533a1042d9bba5ed5711d09d9c P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.078s user 0.004s sys 0.004s
I20251027 21:07:48.170714 10032 raft_consensus.cc:359] T 988052533a1042d9bba5ed5711d09d9c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.170962 10032 raft_consensus.cc:385] T 988052533a1042d9bba5ed5711d09d9c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.171051 10032 raft_consensus.cc:740] T 988052533a1042d9bba5ed5711d09d9c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.171404 10032 consensus_queue.cc:260] T 988052533a1042d9bba5ed5711d09d9c P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.178807 10032 ts_tablet_manager.cc:1434] T 988052533a1042d9bba5ed5711d09d9c P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20251027 21:07:48.183122 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 33e41145c79b4465b8d564741174b8e0 (DEFAULT_TABLE table=default.Table_780df15afd4d45d1bd65f9a2ede532c3 [id=af5b5b94fcef4677affe7e7afaa7f023]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.183594 9308 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet a878184a3a444761aff8730bdf43ccaf (table default.Table_1c94591b5a734f7a8ce72ed245351b55 [id=77687d6093f741d3905c660d0650ae13]). Expected version 1 got 0
I20251027 21:07:48.183663 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33e41145c79b4465b8d564741174b8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.185411 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_35e8a94e410b4cf99228b2ec46855088" } new_table_name: "default.Table_eca1294e7d18437bad196fc2fc5dd3fc" modify_external_catalogs: true
I20251027 21:07:48.187063 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet a4c022969ea448cf933d5220f7ae8f66 with delete_type TABLET_DATA_DELETED (replaced by tablet d8ef795c88da49b58082ba1264d83378) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.187521 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 33e41145c79b4465b8d564741174b8e0 (DEFAULT_TABLE table=default.Table_780df15afd4d45d1bd65f9a2ede532c3 [id=af5b5b94fcef4677affe7e7afaa7f023]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.188004 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33e41145c79b4465b8d564741174b8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.188109 9306 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet a878184a3a444761aff8730bdf43ccaf (table default.Table_1c94591b5a734f7a8ce72ed245351b55 [id=77687d6093f741d3905c660d0650ae13]). Expected version 1 got 0
I20251027 21:07:48.188997 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_795e30871da94371958578747d1c898b" } modify_external_catalogs: true
I20251027 21:07:48.189172 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_795e30871da94371958578747d1c898b" } modify_external_catalogs: true
I20251027 21:07:48.190491 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 2d659e4d73ea4ff3975a4b58a794090a (DEFAULT_TABLE table=default.Table_3886813b08244afa9a2d851978407392 [id=af6cb904f8cd4a929be7a978f9bc2001]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.190950 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d659e4d73ea4ff3975a4b58a794090a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.193635 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 33e41145c79b4465b8d564741174b8e0 (DEFAULT_TABLE table=default.Table_780df15afd4d45d1bd65f9a2ede532c3 [id=af5b5b94fcef4677affe7e7afaa7f023]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.194000 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33e41145c79b4465b8d564741174b8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.194468 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 81e914ecc3c74b869e9fc3ad102887ec (DEFAULT_TABLE table=default.Table_94fee319c82e4c53b9c5aa027e0c5718 [id=a6803bcbb3c2427992f31518cc1de040]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.194918 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e914ecc3c74b869e9fc3ad102887ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.198473 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 2d659e4d73ea4ff3975a4b58a794090a (DEFAULT_TABLE table=default.Table_3886813b08244afa9a2d851978407392 [id=af6cb904f8cd4a929be7a978f9bc2001]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.198065 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cd4f5fa03d4417383df279da475345f with delete_type TABLET_DATA_DELETED (replaced by tablet d642da56a67449aa95932180bc5b2c5f) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.198825 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d659e4d73ea4ff3975a4b58a794090a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.198834 10041 tablet_replica.cc:333] T 0cd4f5fa03d4417383df279da475345f P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:48.199079 10041 raft_consensus.cc:2243] T 0cd4f5fa03d4417383df279da475345f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:48.199265 10041 raft_consensus.cc:2272] T 0cd4f5fa03d4417383df279da475345f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.200078 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet bd6230765d6448498132e24482100b39 with delete_type TABLET_DATA_DELETED (replaced by tablet 84305bcf21b64d2c8c1df015a2e9615c) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.200795 10041 ts_tablet_manager.cc:1916] T 0cd4f5fa03d4417383df279da475345f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:48.202740 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 81e914ecc3c74b869e9fc3ad102887ec (DEFAULT_TABLE table=default.Table_94fee319c82e4c53b9c5aa027e0c5718 [id=a6803bcbb3c2427992f31518cc1de040]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.203145 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e914ecc3c74b869e9fc3ad102887ec. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:48.204736 9299 catalog_manager.cc:4952] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet a4c022969ea448cf933d5220f7ae8f66 because the tablet was not found. No further retry: Not found: Tablet not found: a4c022969ea448cf933d5220f7ae8f66
W20251027 21:07:48.206409 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81e914ecc3c74b869e9fc3ad102887ec consensus state has no leader
I20251027 21:07:48.207163 9311 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet a878184a3a444761aff8730bdf43ccaf (table default.Table_1c94591b5a734f7a8ce72ed245351b55 [id=77687d6093f741d3905c660d0650ae13]). Expected version 1 got 0
I20251027 21:07:48.208894 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:48.209064 9612 tablet_service.cc:1505] Processing CreateTablet for tablet 81e914ecc3c74b869e9fc3ad102887ec (DEFAULT_TABLE table=default.Table_94fee319c82e4c53b9c5aa027e0c5718 [id=a6803bcbb3c2427992f31518cc1de040]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.209439 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e914ecc3c74b869e9fc3ad102887ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.209761 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 988052533a1042d9bba5ed5711d09d9c with delete_type TABLET_DATA_DELETED (replaced by tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:48.216290 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3c9f64ee0544e59add4f73bca59df98 consensus state has no leader
W20251027 21:07:48.223067 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8fcfa6b85e7b46d78a84947e9f11ae46 has no consensus state
I20251027 21:07:48.230475 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 2d659e4d73ea4ff3975a4b58a794090a (DEFAULT_TABLE table=default.Table_3886813b08244afa9a2d851978407392 [id=af6cb904f8cd4a929be7a978f9bc2001]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.230753 10031 tablet_bootstrap.cc:492] T bd6230765d6448498132e24482100b39 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:48.230870 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d659e4d73ea4ff3975a4b58a794090a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.230926 10031 ts_tablet_manager.cc:1403] T bd6230765d6448498132e24482100b39 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.087s user 0.036s sys 0.045s
I20251027 21:07:48.231864 10031 raft_consensus.cc:359] T bd6230765d6448498132e24482100b39 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.232069 10031 raft_consensus.cc:385] T bd6230765d6448498132e24482100b39 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.232138 10031 raft_consensus.cc:740] T bd6230765d6448498132e24482100b39 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.232419 10031 consensus_queue.cc:260] T bd6230765d6448498132e24482100b39 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.232892 10031 ts_tablet_manager.cc:1434] T bd6230765d6448498132e24482100b39 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:48.233222 10031 tablet_bootstrap.cc:492] T 988052533a1042d9bba5ed5711d09d9c P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:48.234838 10031 tablet_bootstrap.cc:654] T 988052533a1042d9bba5ed5711d09d9c P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.237367 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet bd6230765d6448498132e24482100b39 with delete_type TABLET_DATA_DELETED (replaced by tablet 84305bcf21b64d2c8c1df015a2e9615c) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:48.237716 10040 tablet_replica.cc:333] T bd6230765d6448498132e24482100b39 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:48.237901 10040 raft_consensus.cc:2243] T bd6230765d6448498132e24482100b39 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:48.238051 10040 raft_consensus.cc:2272] T bd6230765d6448498132e24482100b39 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.239538 10040 ts_tablet_manager.cc:1916] T bd6230765d6448498132e24482100b39 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:48.240480 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 65fbcaa5447c48818fabfccac22c4048 consensus state has no leader
I20251027 21:07:48.240814 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 94a00ff844f0415fa419613d70a318e6
I20251027 21:07:48.241247 9309 catalog_manager.cc:6719] ReplaceTablet: tablet 94a00ff844f0415fa419613d70a318e6 deleted and replaced by tablet efc0019840244d14bb95ae581793e74b
I20251027 21:07:48.241442 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet 94a00ff844f0415fa419613d70a318e6 with delete_type TABLET_DATA_DELETED (replaced by tablet efc0019840244d14bb95ae581793e74b) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.241976 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 94a00ff844f0415fa419613d70a318e6 with delete_type TABLET_DATA_DELETED (replaced by tablet efc0019840244d14bb95ae581793e74b) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:48.242446 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet 94a00ff844f0415fa419613d70a318e6 with delete_type TABLET_DATA_DELETED (replaced by tablet efc0019840244d14bb95ae581793e74b) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:48.243268 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 94a00ff844f0415fa419613d70a318e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94a00ff844f0415fa419613d70a318e6 already in progress: creating tablet
I20251027 21:07:48.244097 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 69b1fb42428748239d8939d98428d8fe
I20251027 21:07:48.244149 10041 ts_tablet_manager.cc:1929] T 0cd4f5fa03d4417383df279da475345f P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.244331 10041 log.cc:1199] T 0cd4f5fa03d4417383df279da475345f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/0cd4f5fa03d4417383df279da475345f
I20251027 21:07:48.244587 9307 catalog_manager.cc:6719] ReplaceTablet: tablet 69b1fb42428748239d8939d98428d8fe deleted and replaced by tablet 49b3d23798b740008c98cd0754ae33f4
I20251027 21:07:48.244693 10041 ts_tablet_manager.cc:1950] T 0cd4f5fa03d4417383df279da475345f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:48.244900 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet 69b1fb42428748239d8939d98428d8fe with delete_type TABLET_DATA_DELETED (replaced by tablet 49b3d23798b740008c98cd0754ae33f4) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.244982 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 36e168cc44274dae9682b0c1329d4558
I20251027 21:07:48.245066 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet 69b1fb42428748239d8939d98428d8fe with delete_type TABLET_DATA_DELETED (replaced by tablet 49b3d23798b740008c98cd0754ae33f4) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.245565 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet 36e168cc44274dae9682b0c1329d4558 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:48.245695 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 970fbf01a4c3406dafe1ea4cd005ea23 has no consensus state
W20251027 21:07:48.245786 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 69b1fb42428748239d8939d98428d8fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69b1fb42428748239d8939d98428d8fe already in progress: creating tablet
I20251027 21:07:48.245782 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 0cd4f5fa03d4417383df279da475345f (table default.Table_178e5d554faa4e119ac8f52da2d284d7 [id=05edeb649c544c2f899ed42a87cef2c2]) successfully deleted
I20251027 21:07:48.245997 10041 tablet_replica.cc:333] T bd6230765d6448498132e24482100b39 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:48.246507 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet 36e168cc44274dae9682b0c1329d4558 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:48.247193 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 36e168cc44274dae9682b0c1329d4558 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36e168cc44274dae9682b0c1329d4558 already in progress: creating tablet
I20251027 21:07:48.247852 10032 tablet_bootstrap.cc:492] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:48.250195 10032 tablet_bootstrap.cc:654] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.250872 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 69b1fb42428748239d8939d98428d8fe with delete_type TABLET_DATA_DELETED (replaced by tablet 49b3d23798b740008c98cd0754ae33f4) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:48.251065 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet 36e168cc44274dae9682b0c1329d4558 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:48.251271 10040 ts_tablet_manager.cc:1929] T bd6230765d6448498132e24482100b39 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.251504 10040 log.cc:1199] T bd6230765d6448498132e24482100b39 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/bd6230765d6448498132e24482100b39
I20251027 21:07:48.251942 10040 ts_tablet_manager.cc:1950] T bd6230765d6448498132e24482100b39 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:48.252882 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet bd6230765d6448498132e24482100b39 (table default.Table_41a04598f12247fbac5f4103d2cbf22d [id=2e28a4d0f9204351b93335736a9abe18]) successfully deleted
I20251027 21:07:48.258909 10029 tablet_bootstrap.cc:654] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.260174 10041 raft_consensus.cc:2243] T bd6230765d6448498132e24482100b39 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:48.260403 10041 raft_consensus.cc:2272] T bd6230765d6448498132e24482100b39 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.267349 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 970fbf01a4c3406dafe1ea4cd005ea23 (DEFAULT_TABLE table=default.Table_eca1294e7d18437bad196fc2fc5dd3fc [id=741c9ce04c9d47d9a016bce1f4403ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.267553 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 10a44aecfade42969454126b990eca24 (DEFAULT_TABLE table=default.Table_b0f0589285ac4a56b7a2b23ace6248f6 [id=43ebd8ba9e694d17a9def870c95d7aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.267732 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 970fbf01a4c3406dafe1ea4cd005ea23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.268051 9895 tablet_service.cc:1505] Processing CreateTablet for tablet f6249fc5c6174e8e81b9d02d598d5ebd (DEFAULT_TABLE table=default.Table_bee0cc21eb6f4fe9a9b2d4ab738a3fe9 [id=1cb32eea25114c6dbcb3b28044036c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.268375 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6249fc5c6174e8e81b9d02d598d5ebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.269740 9612 tablet_service.cc:1505] Processing CreateTablet for tablet 8fcfa6b85e7b46d78a84947e9f11ae46 (DEFAULT_TABLE table=default.Table_23cfb97e14e443a1bc6763bc466837a6 [id=e68b63cf420d49289a5bd1db81e1cbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.270282 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fcfa6b85e7b46d78a84947e9f11ae46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.270828 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10a44aecfade42969454126b990eca24. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.274447 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 970fbf01a4c3406dafe1ea4cd005ea23 (DEFAULT_TABLE table=default.Table_eca1294e7d18437bad196fc2fc5dd3fc [id=741c9ce04c9d47d9a016bce1f4403ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.274850 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 970fbf01a4c3406dafe1ea4cd005ea23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.278508 9608 tablet_service.cc:1505] Processing CreateTablet for tablet 10a44aecfade42969454126b990eca24 (DEFAULT_TABLE table=default.Table_b0f0589285ac4a56b7a2b23ace6248f6 [id=43ebd8ba9e694d17a9def870c95d7aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.278909 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10a44aecfade42969454126b990eca24. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.282480 9605 tablet_service.cc:1505] Processing CreateTablet for tablet f6249fc5c6174e8e81b9d02d598d5ebd (DEFAULT_TABLE table=default.Table_bee0cc21eb6f4fe9a9b2d4ab738a3fe9 [id=1cb32eea25114c6dbcb3b28044036c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.282874 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6249fc5c6174e8e81b9d02d598d5ebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.286531 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 601c74459d3a4a41b812fdcdf8b0a62c (DEFAULT_TABLE table=default.Table_63c557c9df2c4f4888317b03f119c607 [id=7c97543d81eb4a1fa0c81678857179ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.286652 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 601c74459d3a4a41b812fdcdf8b0a62c (DEFAULT_TABLE table=default.Table_63c557c9df2c4f4888317b03f119c607 [id=7c97543d81eb4a1fa0c81678857179ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.287144 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 601c74459d3a4a41b812fdcdf8b0a62c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.287031 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 601c74459d3a4a41b812fdcdf8b0a62c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.290454 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 8fcfa6b85e7b46d78a84947e9f11ae46 (DEFAULT_TABLE table=default.Table_23cfb97e14e443a1bc6763bc466837a6 [id=e68b63cf420d49289a5bd1db81e1cbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.290890 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fcfa6b85e7b46d78a84947e9f11ae46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.294077 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 970fbf01a4c3406dafe1ea4cd005ea23 (DEFAULT_TABLE table=default.Table_eca1294e7d18437bad196fc2fc5dd3fc [id=741c9ce04c9d47d9a016bce1f4403ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.294478 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 970fbf01a4c3406dafe1ea4cd005ea23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.294399 9748 tablet_service.cc:1505] Processing CreateTablet for tablet f6249fc5c6174e8e81b9d02d598d5ebd (DEFAULT_TABLE table=default.Table_bee0cc21eb6f4fe9a9b2d4ab738a3fe9 [id=1cb32eea25114c6dbcb3b28044036c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.294759 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6249fc5c6174e8e81b9d02d598d5ebd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:48.294713 9825 tablet.cc:2378] T a878184a3a444761aff8730bdf43ccaf P 77b0513f17c0472088901a1ce8b08218: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 21:07:48.295001 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 10a44aecfade42969454126b990eca24 (DEFAULT_TABLE table=default.Table_b0f0589285ac4a56b7a2b23ace6248f6 [id=43ebd8ba9e694d17a9def870c95d7aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.295347 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10a44aecfade42969454126b990eca24. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.298435 9751 tablet_service.cc:1505] Processing CreateTablet for tablet 601c74459d3a4a41b812fdcdf8b0a62c (DEFAULT_TABLE table=default.Table_63c557c9df2c4f4888317b03f119c607 [id=7c97543d81eb4a1fa0c81678857179ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.298743 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 601c74459d3a4a41b812fdcdf8b0a62c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.298835 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3b2e962c9ca9424dad8aaa7950efbf63" } modify_external_catalogs: true
I20251027 21:07:48.299032 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3b2e962c9ca9424dad8aaa7950efbf63" } modify_external_catalogs: true
I20251027 21:07:48.301060 10041 ts_tablet_manager.cc:1916] T bd6230765d6448498132e24482100b39 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:48.301977 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7a662e3d3c4b4f8cb68a606bdf279049"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.302493 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a662e3d3c4b4f8cb68a606bdf279049 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:48.303333 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 94a00ff844f0415fa419613d70a318e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94a00ff844f0415fa419613d70a318e6 already in progress: creating tablet
W20251027 21:07:48.303561 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 69b1fb42428748239d8939d98428d8fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69b1fb42428748239d8939d98428d8fe already in progress: creating tablet
W20251027 21:07:48.303810 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 36e168cc44274dae9682b0c1329d4558 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36e168cc44274dae9682b0c1329d4558 already in progress: creating tablet
I20251027 21:07:48.304121 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_758d1e73474647eaa8a15b6cff350af6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.304567 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_758d1e73474647eaa8a15b6cff350af6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.303644 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d98a9c0b61d649f2945cb6c199f70a64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.305779 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d98a9c0b61d649f2945cb6c199f70a64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.309202 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f05923c6594b43378fa156d0ab2bd1da" } modify_external_catalogs: true
I20251027 21:07:48.309365 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f05923c6594b43378fa156d0ab2bd1da" } modify_external_catalogs: true
I20251027 21:07:48.310549 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_fd19a2c8dc2f4164a0326865a21cd141"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.310933 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd19a2c8dc2f4164a0326865a21cd141 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.311085 10032 tablet_bootstrap.cc:492] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:48.311259 10032 ts_tablet_manager.cc:1403] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.125s user 0.003s sys 0.000s
I20251027 21:07:48.311944 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b534fdb9e3084d6f9656dfdbb0e01750"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:48.312161 10032 raft_consensus.cc:359] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:48.312340 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b534fdb9e3084d6f9656dfdbb0e01750 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.312546 10032 raft_consensus.cc:385] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.312671 10032 raft_consensus.cc:740] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.312951 10032 consensus_queue.cc:260] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.313326 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1c94591b5a734f7a8ce72ed245351b55" } modify_external_catalogs: true
I20251027 21:07:48.313488 10032 ts_tablet_manager.cc:1434] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:48.313472 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1c94591b5a734f7a8ce72ed245351b55" } modify_external_catalogs: true
I20251027 21:07:48.313818 10032 tablet_bootstrap.cc:492] T 783d845009d54d20b8e6626c3e73d2a7 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:48.315598 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b2dd87493a364598b5a702417e743dc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.316036 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2dd87493a364598b5a702417e743dc8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.320456 10029 tablet_bootstrap.cc:492] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:48.320652 10029 ts_tablet_manager.cc:1403] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.153s user 0.004s sys 0.000s
I20251027 21:07:48.321663 10029 raft_consensus.cc:359] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.321899 10029 raft_consensus.cc:385] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.322027 10029 raft_consensus.cc:740] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.322355 10029 consensus_queue.cc:260] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.323011 10029 ts_tablet_manager.cc:1434] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251027 21:07:48.323390 10029 tablet_bootstrap.cc:492] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:48.324965 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 8fcfa6b85e7b46d78a84947e9f11ae46 (DEFAULT_TABLE table=default.Table_23cfb97e14e443a1bc6763bc466837a6 [id=e68b63cf420d49289a5bd1db81e1cbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.325369 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fcfa6b85e7b46d78a84947e9f11ae46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.325647 10029 tablet_bootstrap.cc:654] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.329398 9896 tablet_service.cc:1505] Processing CreateTablet for tablet d89c3c588a1747d7875ebf45924fb890 (DEFAULT_TABLE table=default.Table_f05923c6594b43378fa156d0ab2bd1da [id=ad541dd12124493583f4fdba032b7ed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.332279 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 35e5b7bbab7f4ee9bbf23d913c0aee76
I20251027 21:07:48.332764 9311 catalog_manager.cc:6719] ReplaceTablet: tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 deleted and replaced by tablet e07b8aa0a41241d79254c4dae8738e79
I20251027 21:07:48.335706 9615 tablet_service.cc:1505] Processing CreateTablet for tablet d13d00353b6740b5bed5a153936d2059 (DEFAULT_TABLE table=default.Table_a356e5bd7b1c40a087ef83c7078e0381 [id=f2edcf33b0b74923b4ddbbc2419e14ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.336489 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d13d00353b6740b5bed5a153936d2059. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.337903 10029 tablet_bootstrap.cc:492] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:48.336129 9608 tablet_service.cc:1505] Processing CreateTablet for tablet d89c3c588a1747d7875ebf45924fb890 (DEFAULT_TABLE table=default.Table_f05923c6594b43378fa156d0ab2bd1da [id=ad541dd12124493583f4fdba032b7ed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.338066 10029 ts_tablet_manager.cc:1403] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20251027 21:07:48.338414 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d89c3c588a1747d7875ebf45924fb890. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.338487 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d89c3c588a1747d7875ebf45924fb890. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.338900 10029 raft_consensus.cc:359] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:48.339393 10029 raft_consensus.cc:385] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.339505 10029 raft_consensus.cc:740] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.339663 9612 tablet_service.cc:1552] Processing DeleteTablet for tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 with delete_type TABLET_DATA_DELETED (replaced by tablet e07b8aa0a41241d79254c4dae8738e79) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.339793 10029 consensus_queue.cc:260] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:48.341135 10029 ts_tablet_manager.cc:1434] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:07:48.341588 10029 tablet_bootstrap.cc:492] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
W20251027 21:07:48.342371 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 already in progress: creating tablet
I20251027 21:07:48.343878 10029 tablet_bootstrap.cc:654] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.344273 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 019a892bfdd54e62bc3560a341d5c2da (DEFAULT_TABLE table=default.Table_18e48b818fc040bfa4272f08e203dc3e [id=2b0d4e085a8e42879c67963231a24b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.344738 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019a892bfdd54e62bc3560a341d5c2da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.345537 9753 tablet_service.cc:1505] Processing CreateTablet for tablet d13d00353b6740b5bed5a153936d2059 (DEFAULT_TABLE table=default.Table_a356e5bd7b1c40a087ef83c7078e0381 [id=f2edcf33b0b74923b4ddbbc2419e14ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.346493 9746 tablet_service.cc:1552] Processing DeleteTablet for tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 with delete_type TABLET_DATA_DELETED (replaced by tablet e07b8aa0a41241d79254c4dae8738e79) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:48.346868 9752 tablet_service.cc:1505] Processing CreateTablet for tablet d89c3c588a1747d7875ebf45924fb890 (DEFAULT_TABLE table=default.Table_f05923c6594b43378fa156d0ab2bd1da [id=ad541dd12124493583f4fdba032b7ed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.347033 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 019a892bfdd54e62bc3560a341d5c2da (DEFAULT_TABLE table=default.Table_18e48b818fc040bfa4272f08e203dc3e [id=2b0d4e085a8e42879c67963231a24b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.347388 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019a892bfdd54e62bc3560a341d5c2da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.347798 9747 tablet_service.cc:1505] Processing CreateTablet for tablet 019a892bfdd54e62bc3560a341d5c2da (DEFAULT_TABLE table=default.Table_18e48b818fc040bfa4272f08e203dc3e [id=2b0d4e085a8e42879c67963231a24b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.349692 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 with delete_type TABLET_DATA_DELETED (replaced by tablet e07b8aa0a41241d79254c4dae8738e79) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.349920 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.354182 9895 tablet_service.cc:1505] Processing CreateTablet for tablet d13d00353b6740b5bed5a153936d2059 (DEFAULT_TABLE table=default.Table_a356e5bd7b1c40a087ef83c7078e0381 [id=f2edcf33b0b74923b4ddbbc2419e14ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.355583 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d13d00353b6740b5bed5a153936d2059. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.355796 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019a892bfdd54e62bc3560a341d5c2da. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:48.358520 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 already in progress: creating tablet
I20251027 21:07:48.358886 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d89c3c588a1747d7875ebf45924fb890. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.368505 10029 tablet_bootstrap.cc:492] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:48.368551 10039 raft_consensus.cc:493] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:48.368649 10029 ts_tablet_manager.cc:1403] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.000s
I20251027 21:07:48.368669 10039 raft_consensus.cc:515] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.369349 10039 leader_election.cc:290] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:48.369443 10029 raft_consensus.cc:359] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:48.369592 10029 raft_consensus.cc:385] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.369678 10029 raft_consensus.cc:740] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.369706 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d13d00353b6740b5bed5a153936d2059. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.369892 10029 consensus_queue.cc:260] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:48.370114 10031 tablet_bootstrap.cc:492] T 988052533a1042d9bba5ed5711d09d9c P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
W20251027 21:07:48.370303 9566 leader_election.cc:343] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Not found: Tablet not found: e9dd86d98cd94cfe9cc2a7b1ab1f979b
I20251027 21:07:48.370438 10031 ts_tablet_manager.cc:1403] T 988052533a1042d9bba5ed5711d09d9c P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.137s user 0.052s sys 0.047s
I20251027 21:07:48.371106 10031 raft_consensus.cc:359] T 988052533a1042d9bba5ed5711d09d9c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.371228 10029 ts_tablet_manager.cc:1434] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251027 21:07:48.371292 10031 raft_consensus.cc:385] T 988052533a1042d9bba5ed5711d09d9c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.371376 10031 raft_consensus.cc:740] T 988052533a1042d9bba5ed5711d09d9c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.371567 10031 consensus_queue.cc:260] T 988052533a1042d9bba5ed5711d09d9c P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.371994 10031 ts_tablet_manager.cc:1434] T 988052533a1042d9bba5ed5711d09d9c P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:07:48.372725 10029 tablet_bootstrap.cc:492] T 3218f18a797c40c9ad6ac64b2670cd19 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:48.374302 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9dd86d98cd94cfe9cc2a7b1ab1f979b" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:48.374436 10029 tablet_bootstrap.cc:654] T 3218f18a797c40c9ad6ac64b2670cd19 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.374498 9925 raft_consensus.cc:2468] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:48.374960 9568 leader_election.cc:304] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:48.375345 10039 raft_consensus.cc:2804] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:48.375365 10031 tablet_bootstrap.cc:492] T 87ec5a6b9b3c425da4e96cc6736d6310 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:48.377063 10031 tablet_bootstrap.cc:654] T 87ec5a6b9b3c425da4e96cc6736d6310 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.377113 10039 raft_consensus.cc:493] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:48.377255 10039 raft_consensus.cc:3060] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:48.380091 10031 tablet_bootstrap.cc:492] T 87ec5a6b9b3c425da4e96cc6736d6310 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:48.380277 10031 ts_tablet_manager.cc:1403] T 87ec5a6b9b3c425da4e96cc6736d6310 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20251027 21:07:48.381059 10031 raft_consensus.cc:359] T 87ec5a6b9b3c425da4e96cc6736d6310 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:48.381282 10031 raft_consensus.cc:385] T 87ec5a6b9b3c425da4e96cc6736d6310 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.381389 10031 raft_consensus.cc:740] T 87ec5a6b9b3c425da4e96cc6736d6310 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.381646 10031 consensus_queue.cc:260] T 87ec5a6b9b3c425da4e96cc6736d6310 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:48.382083 10031 ts_tablet_manager.cc:1434] T 87ec5a6b9b3c425da4e96cc6736d6310 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:48.382403 10031 tablet_bootstrap.cc:492] T 3218f18a797c40c9ad6ac64b2670cd19 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:48.382903 10032 tablet_bootstrap.cc:654] T 783d845009d54d20b8e6626c3e73d2a7 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.383885 10031 tablet_bootstrap.cc:654] T 3218f18a797c40c9ad6ac64b2670cd19 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.383886 10039 raft_consensus.cc:515] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.384766 10039 leader_election.cc:290] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:48.384945 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9dd86d98cd94cfe9cc2a7b1ab1f979b" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:07:48.385123 9925 raft_consensus.cc:3060] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:07:48.385139 9566 leader_election.cc:343] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Not found: Tablet not found: e9dd86d98cd94cfe9cc2a7b1ab1f979b
I20251027 21:07:48.386739 10041 ts_tablet_manager.cc:1929] T bd6230765d6448498132e24482100b39 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.386834 10041 log.cc:1199] T bd6230765d6448498132e24482100b39 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/bd6230765d6448498132e24482100b39
I20251027 21:07:48.387182 10041 ts_tablet_manager.cc:1950] T bd6230765d6448498132e24482100b39 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:48.387916 9925 raft_consensus.cc:2468] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:48.390498 10041 tablet_replica.cc:333] T 988052533a1042d9bba5ed5711d09d9c P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:48.390684 10041 raft_consensus.cc:2243] T 988052533a1042d9bba5ed5711d09d9c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:48.390820 10041 raft_consensus.cc:2272] T 988052533a1042d9bba5ed5711d09d9c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.391484 10041 ts_tablet_manager.cc:1916] T 988052533a1042d9bba5ed5711d09d9c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:48.392081 9568 leader_election.cc:304] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:48.393428 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet bd6230765d6448498132e24482100b39 (table default.Table_41a04598f12247fbac5f4103d2cbf22d [id=2e28a4d0f9204351b93335736a9abe18]) successfully deleted
I20251027 21:07:48.397475 10039 raft_consensus.cc:2804] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:48.398279 9308 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 674870d162dd443e9b095770a3d69ac9
I20251027 21:07:48.399101 10039 raft_consensus.cc:697] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:07:48.399420 10039 consensus_queue.cc:237] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.400857 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_03046ffdd6ca47e8aa4cde13574dc119"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.401515 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03046ffdd6ca47e8aa4cde13574dc119 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.402710 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_79febe3afdb94e8781e2633f126402dc" } modify_external_catalogs: true
I20251027 21:07:48.402922 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_79febe3afdb94e8781e2633f126402dc" } modify_external_catalogs: true
I20251027 21:07:48.404520 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet 61cd8ee8e1f8459eb7af5dd48c018e02 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:48.404953 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet d89c3c588a1747d7875ebf45924fb890
I20251027 21:07:48.405720 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4568794575f345c08a45afeb2d4f9648"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.406239 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4568794575f345c08a45afeb2d4f9648 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.407447 9312 master_service.cc:954] ReplaceTablet: received request to replace tablet ba8b125474bc42a5946e8f2dd718f7f4 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:48.409704 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet d89c3c588a1747d7875ebf45924fb890 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.410036 10031 tablet_bootstrap.cc:492] T 3218f18a797c40c9ad6ac64b2670cd19 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:48.410230 10031 ts_tablet_manager.cc:1403] T 3218f18a797c40c9ad6ac64b2670cd19 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.011s
I20251027 21:07:48.411108 9754 tablet_service.cc:1552] Processing DeleteTablet for tablet 988052533a1042d9bba5ed5711d09d9c with delete_type TABLET_DATA_DELETED (replaced by tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:48.411252 9747 tablet_service.cc:1552] Processing DeleteTablet for tablet d89c3c588a1747d7875ebf45924fb890 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:48.411361 10040 tablet_replica.cc:333] T 988052533a1042d9bba5ed5711d09d9c P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:48.411410 10031 raft_consensus.cc:359] T 3218f18a797c40c9ad6ac64b2670cd19 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:48.411557 10040 raft_consensus.cc:2243] T 988052533a1042d9bba5ed5711d09d9c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:48.411623 10031 raft_consensus.cc:385] T 3218f18a797c40c9ad6ac64b2670cd19 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.411710 10031 raft_consensus.cc:740] T 3218f18a797c40c9ad6ac64b2670cd19 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.411759 10040 raft_consensus.cc:2272] T 988052533a1042d9bba5ed5711d09d9c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.411963 10031 consensus_queue.cc:260] T 3218f18a797c40c9ad6ac64b2670cd19 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:48.412438 10031 ts_tablet_manager.cc:1434] T 3218f18a797c40c9ad6ac64b2670cd19 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:07:48.412567 10040 ts_tablet_manager.cc:1916] T 988052533a1042d9bba5ed5711d09d9c P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:48.412709 10031 tablet_bootstrap.cc:492] T 783d845009d54d20b8e6626c3e73d2a7 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:48.414218 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet a878184a3a444761aff8730bdf43ccaf
I20251027 21:07:48.414460 10031 tablet_bootstrap.cc:654] T 783d845009d54d20b8e6626c3e73d2a7 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.418798 9754 tablet_service.cc:1552] Processing DeleteTablet for tablet a878184a3a444761aff8730bdf43ccaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:48.419991 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_bee0cc21eb6f4fe9a9b2d4ab738a3fe9" } new_table_name: "default.Table_943417713b3b456ab4ca691d2fb168fd" modify_external_catalogs: true
I20251027 21:07:48.421464 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_274a6d6e3de14343aac38fb44fe222a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.421918 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_274a6d6e3de14343aac38fb44fe222a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.422473 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1764f51a79024aac9bcd15e3f03fd314" } new_table_name: "default.Table_ec86de5add82433b81e1060e76d559e3" modify_external_catalogs: true
I20251027 21:07:48.424240 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet a878184a3a444761aff8730bdf43ccaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.424510 10043 tablet_replica.cc:333] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:48.424597 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_63c557c9df2c4f4888317b03f119c607" } new_table_name: "default.Table_0ac2c74fb86c48459bad366bd8efccd5" modify_external_catalogs: true
I20251027 21:07:48.424755 10043 raft_consensus.cc:2243] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:07:48.425175 10043 pending_rounds.cc:70] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14: Trying to abort 1 pending ops.
I20251027 21:07:48.425395 10043 pending_rounds.cc:77] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510602395987968 op_type: NO_OP noop_request { }
I20251027 21:07:48.425568 10043 raft_consensus.cc:2889] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:07:48.425706 10043 raft_consensus.cc:2272] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.427099 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet d89c3c588a1747d7875ebf45924fb890 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.428280 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet a878184a3a444761aff8730bdf43ccaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.428840 10032 tablet_bootstrap.cc:492] T 783d845009d54d20b8e6626c3e73d2a7 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:48.428989 10032 ts_tablet_manager.cc:1403] T 783d845009d54d20b8e6626c3e73d2a7 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.115s user 0.003s sys 0.000s
I20251027 21:07:48.429562 10032 raft_consensus.cc:359] T 783d845009d54d20b8e6626c3e73d2a7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:48.429718 10032 raft_consensus.cc:385] T 783d845009d54d20b8e6626c3e73d2a7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.429795 10032 raft_consensus.cc:740] T 783d845009d54d20b8e6626c3e73d2a7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.430017 10032 consensus_queue.cc:260] T 783d845009d54d20b8e6626c3e73d2a7 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:48.430732 10043 ts_tablet_manager.cc:1916] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:48.433612 10032 ts_tablet_manager.cc:1434] T 783d845009d54d20b8e6626c3e73d2a7 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251027 21:07:48.434634 10032 tablet_bootstrap.cc:492] T 3218f18a797c40c9ad6ac64b2670cd19 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:48.436537 10043 ts_tablet_manager.cc:1929] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:07:48.436631 10043 log.cc:1199] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/a878184a3a444761aff8730bdf43ccaf
I20251027 21:07:48.437055 10043 ts_tablet_manager.cc:1950] T a878184a3a444761aff8730bdf43ccaf P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:48.438117 10032 tablet_bootstrap.cc:654] T 3218f18a797c40c9ad6ac64b2670cd19 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.439934 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet a878184a3a444761aff8730bdf43ccaf (table default.Table_1c94591b5a734f7a8ce72ed245351b55 [id=77687d6093f741d3905c660d0650ae13]) successfully deleted
I20251027 21:07:48.440361 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet ba8b125474bc42a5946e8f2dd718f7f4
W20251027 21:07:48.440618 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet d89c3c588a1747d7875ebf45924fb890 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d89c3c588a1747d7875ebf45924fb890 already in progress: creating tablet
I20251027 21:07:48.440836 9312 catalog_manager.cc:6719] ReplaceTablet: tablet ba8b125474bc42a5946e8f2dd718f7f4 deleted and replaced by tablet 77797eea3e9c4f8099dc9d58f116200b
I20251027 21:07:48.441190 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet ba8b125474bc42a5946e8f2dd718f7f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 77797eea3e9c4f8099dc9d58f116200b) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:48.441856 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet ba8b125474bc42a5946e8f2dd718f7f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba8b125474bc42a5946e8f2dd718f7f4 already in progress: creating tablet
W20251027 21:07:48.442553 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f6249fc5c6174e8e81b9d02d598d5ebd consensus state has no leader
I20251027 21:07:48.442572 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet ba8b125474bc42a5946e8f2dd718f7f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 77797eea3e9c4f8099dc9d58f116200b) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.443274 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c28df5bcb6c0432791ff79f424fcdca2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.443737 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c28df5bcb6c0432791ff79f424fcdca2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.445891 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_59b38114c28e4a239c53d736dc162176" } new_table_name: "default.Table_f1908c85b5144e90a366ca29ba5d1b9b" modify_external_catalogs: true
I20251027 21:07:48.447885 10040 ts_tablet_manager.cc:1929] T 988052533a1042d9bba5ed5711d09d9c P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.448004 10040 log.cc:1199] T 988052533a1042d9bba5ed5711d09d9c P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/988052533a1042d9bba5ed5711d09d9c
I20251027 21:07:48.448364 10040 ts_tablet_manager.cc:1950] T 988052533a1042d9bba5ed5711d09d9c P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:48.448975 10040 tablet_replica.cc:333] T a878184a3a444761aff8730bdf43ccaf P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:48.449172 10040 raft_consensus.cc:2243] T a878184a3a444761aff8730bdf43ccaf P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:48.449337 10040 raft_consensus.cc:2272] T a878184a3a444761aff8730bdf43ccaf P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:07:48.449982 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c4369a10bf846fb94c9b71a70f21600 consensus state has no leader
I20251027 21:07:48.450793 10040 ts_tablet_manager.cc:1916] T a878184a3a444761aff8730bdf43ccaf P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:48.450819 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 988052533a1042d9bba5ed5711d09d9c (table default.Table_4c5433970bd64291a2b1289a84f35590 [id=a1b2d46db3764c7f97712a0c5f891881]) successfully deleted
W20251027 21:07:48.451077 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet d89c3c588a1747d7875ebf45924fb890 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d89c3c588a1747d7875ebf45924fb890 already in progress: creating tablet
W20251027 21:07:48.451515 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 601c74459d3a4a41b812fdcdf8b0a62c consensus state has no leader
I20251027 21:07:48.451854 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e3c9f64ee0544e59add4f73bca59df98
I20251027 21:07:48.452980 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b2dd87493a364598b5a702417e743dc8" } new_table_name: "default.Table_d9744e5483fd46a581cff3e8e787a6bd" modify_external_catalogs: true
I20251027 21:07:48.454325 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4c5433970bd64291a2b1289a84f35590" } modify_external_catalogs: true
I20251027 21:07:48.454494 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4c5433970bd64291a2b1289a84f35590" } modify_external_catalogs: true
I20251027 21:07:48.456161 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet e3c9f64ee0544e59add4f73bca59df98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.457290 10040 ts_tablet_manager.cc:1929] T a878184a3a444761aff8730bdf43ccaf P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.457422 10040 log.cc:1199] T a878184a3a444761aff8730bdf43ccaf P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/a878184a3a444761aff8730bdf43ccaf
I20251027 21:07:48.457769 10040 ts_tablet_manager.cc:1950] T a878184a3a444761aff8730bdf43ccaf P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:48.459323 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet 5c87a5450d354506852c4b20bf0541d2 from {username='slave'} at 127.0.0.1:41710
W20251027 21:07:48.460057 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet e3c9f64ee0544e59add4f73bca59df98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3c9f64ee0544e59add4f73bca59df98 already in progress: creating tablet
I20251027 21:07:48.460426 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet a878184a3a444761aff8730bdf43ccaf (table default.Table_1c94591b5a734f7a8ce72ed245351b55 [id=77687d6093f741d3905c660d0650ae13]) successfully deleted
I20251027 21:07:48.460578 9747 tablet_service.cc:1552] Processing DeleteTablet for tablet e3c9f64ee0544e59add4f73bca59df98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:48.461791 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet e3c9f64ee0544e59add4f73bca59df98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3c9f64ee0544e59add4f73bca59df98 already in progress: creating tablet
W20251027 21:07:48.462270 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 21b785277d604543bfa7816ed96d6181 consensus state has no leader
W20251027 21:07:48.464442 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9248f458baca409aa3505cc0db321b34 has no consensus state
I20251027 21:07:48.465970 9754 tablet_service.cc:1552] Processing DeleteTablet for tablet ba8b125474bc42a5946e8f2dd718f7f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 77797eea3e9c4f8099dc9d58f116200b) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:48.466606 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet ba8b125474bc42a5946e8f2dd718f7f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba8b125474bc42a5946e8f2dd718f7f4 already in progress: creating tablet
I20251027 21:07:48.466722 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet e3c9f64ee0544e59add4f73bca59df98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.467684 9309 master_service.cc:954] ReplaceTablet: received request to replace tablet 5c87a5450d354506852c4b20bf0541d2 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:48.470835 10031 tablet_bootstrap.cc:492] T 783d845009d54d20b8e6626c3e73d2a7 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:48.471019 10031 ts_tablet_manager.cc:1403] T 783d845009d54d20b8e6626c3e73d2a7 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.058s user 0.021s sys 0.011s
I20251027 21:07:48.472090 10031 raft_consensus.cc:359] T 783d845009d54d20b8e6626c3e73d2a7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:48.472285 10031 raft_consensus.cc:385] T 783d845009d54d20b8e6626c3e73d2a7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.472370 10031 raft_consensus.cc:740] T 783d845009d54d20b8e6626c3e73d2a7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.472605 10031 consensus_queue.cc:260] T 783d845009d54d20b8e6626c3e73d2a7 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:48.472893 10032 tablet_bootstrap.cc:492] T 3218f18a797c40c9ad6ac64b2670cd19 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:48.472831 9747 tablet_service.cc:1505] Processing CreateTablet for tablet efc0019840244d14bb95ae581793e74b (DEFAULT_TABLE table=default.Table_8c58943248a74b5ab81a5b742eafc5b4 [id=f773d65a07a4473ab8b87cd82ac50bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.473035 10031 ts_tablet_manager.cc:1434] T 783d845009d54d20b8e6626c3e73d2a7 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 21:07:48.473196 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efc0019840244d14bb95ae581793e74b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.473016 10032 ts_tablet_manager.cc:1403] T 3218f18a797c40c9ad6ac64b2670cd19 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.002s
I20251027 21:07:48.474700 10032 raft_consensus.cc:359] T 3218f18a797c40c9ad6ac64b2670cd19 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:48.474892 10032 raft_consensus.cc:385] T 3218f18a797c40c9ad6ac64b2670cd19 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.474977 10032 raft_consensus.cc:740] T 3218f18a797c40c9ad6ac64b2670cd19 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.475253 10032 consensus_queue.cc:260] T 3218f18a797c40c9ad6ac64b2670cd19 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:48.475364 10031 tablet_bootstrap.cc:492] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:48.475759 10032 ts_tablet_manager.cc:1434] T 3218f18a797c40c9ad6ac64b2670cd19 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:07:48.476008 10032 tablet_bootstrap.cc:492] T 87ec5a6b9b3c425da4e96cc6736d6310 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:48.477474 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 2b247829a11349b6835a2f18906e20a9 (DEFAULT_TABLE table=default.Table_096b7cf08dbf49279d8d5c11b3b687ab [id=20b788f5a2d14b31a58e1b29fa954951]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.477758 10032 tablet_bootstrap.cc:654] T 87ec5a6b9b3c425da4e96cc6736d6310 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.478360 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 783d845009d54d20b8e6626c3e73d2a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.478574 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 2b247829a11349b6835a2f18906e20a9 (DEFAULT_TABLE table=default.Table_096b7cf08dbf49279d8d5c11b3b687ab [id=20b788f5a2d14b31a58e1b29fa954951]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.478770 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b247829a11349b6835a2f18906e20a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.478971 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b247829a11349b6835a2f18906e20a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.482379 10031 tablet_bootstrap.cc:654] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.482398 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 3218f18a797c40c9ad6ac64b2670cd19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.483811 9895 tablet_service.cc:1505] Processing CreateTablet for tablet efc0019840244d14bb95ae581793e74b (DEFAULT_TABLE table=default.Table_8c58943248a74b5ab81a5b742eafc5b4 [id=f773d65a07a4473ab8b87cd82ac50bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.484663 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 49b3d23798b740008c98cd0754ae33f4 (DEFAULT_TABLE table=default.Table_5d5f327d728f4250841a9f4f10a2b533 [id=33dbf5fe986c415cb54fbf1b574651ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.485006 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49b3d23798b740008c98cd0754ae33f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.486274 10032 tablet_bootstrap.cc:492] T 87ec5a6b9b3c425da4e96cc6736d6310 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:48.486420 10032 ts_tablet_manager.cc:1403] T 87ec5a6b9b3c425da4e96cc6736d6310 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.001s
I20251027 21:07:48.488018 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 2b247829a11349b6835a2f18906e20a9 (DEFAULT_TABLE table=default.Table_096b7cf08dbf49279d8d5c11b3b687ab [id=20b788f5a2d14b31a58e1b29fa954951]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.488507 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b247829a11349b6835a2f18906e20a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.491052 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 49b3d23798b740008c98cd0754ae33f4 (DEFAULT_TABLE table=default.Table_5d5f327d728f4250841a9f4f10a2b533 [id=33dbf5fe986c415cb54fbf1b574651ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.492717 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe
I20251027 21:07:48.493019 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 61cd8ee8e1f8459eb7af5dd48c018e02
I20251027 21:07:48.493471 9307 catalog_manager.cc:6719] ReplaceTablet: tablet 61cd8ee8e1f8459eb7af5dd48c018e02 deleted and replaced by tablet 49d9e5c5960a4534a6603875006e6266
I20251027 21:07:48.493711 9615 tablet_service.cc:1505] Processing CreateTablet for tablet efc0019840244d14bb95ae581793e74b (DEFAULT_TABLE table=default.Table_8c58943248a74b5ab81a5b742eafc5b4 [id=f773d65a07a4473ab8b87cd82ac50bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.494210 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efc0019840244d14bb95ae581793e74b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.494451 9612 tablet_service.cc:1552] Processing DeleteTablet for tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.495329 9612 tablet_service.cc:1552] Processing DeleteTablet for tablet 61cd8ee8e1f8459eb7af5dd48c018e02 with delete_type TABLET_DATA_DELETED (replaced by tablet 49d9e5c5960a4534a6603875006e6266) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:48.496230 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe already in progress: creating tablet
W20251027 21:07:48.496438 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 61cd8ee8e1f8459eb7af5dd48c018e02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61cd8ee8e1f8459eb7af5dd48c018e02 already in progress: creating tablet
I20251027 21:07:48.496868 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efc0019840244d14bb95ae581793e74b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.497341 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49b3d23798b740008c98cd0754ae33f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.498528 9608 tablet_service.cc:1505] Processing CreateTablet for tablet 49b3d23798b740008c98cd0754ae33f4 (DEFAULT_TABLE table=default.Table_5d5f327d728f4250841a9f4f10a2b533 [id=33dbf5fe986c415cb54fbf1b574651ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.498932 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49b3d23798b740008c98cd0754ae33f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.501669 10039 raft_consensus.cc:493] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:48.501859 10039 raft_consensus.cc:515] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:48.502415 10041 ts_tablet_manager.cc:1929] T 988052533a1042d9bba5ed5711d09d9c P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.502534 10041 log.cc:1199] T 988052533a1042d9bba5ed5711d09d9c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/988052533a1042d9bba5ed5711d09d9c
I20251027 21:07:48.502679 10039 leader_election.cc:290] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:48.502877 10041 ts_tablet_manager.cc:1950] T 988052533a1042d9bba5ed5711d09d9c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:48.504097 10041 tablet_replica.cc:333] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:48.504248 10041 raft_consensus.cc:2243] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:48.504388 10041 raft_consensus.cc:2272] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.504884 9273 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu
/tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.14.124:45071
--webserver_interface=127.9.14.124
--webserver_port=41907
--builtin_ntp_servers=127.9.14.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.14.126:34057,127.9.14.125:46313,127.9.14.124:45071
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251027 21:07:48.505018 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "783d845009d54d20b8e6626c3e73d2a7" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:48.505312 9775 raft_consensus.cc:2468] T 783d845009d54d20b8e6626c3e73d2a7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:48.506032 9566 leader_election.cc:304] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:07:48.510967 10041 ts_tablet_manager.cc:1916] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:48.515450 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:48.515712 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 61cd8ee8e1f8459eb7af5dd48c018e02 with delete_type TABLET_DATA_DELETED (replaced by tablet 49d9e5c5960a4534a6603875006e6266) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:48.516032 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 988052533a1042d9bba5ed5711d09d9c with delete_type TABLET_DATA_DELETED (replaced by tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:48.516332 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 3218f18a797c40c9ad6ac64b2670cd19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:48.516454 10029 tablet_bootstrap.cc:492] T 3218f18a797c40c9ad6ac64b2670cd19 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:48.516600 10029 ts_tablet_manager.cc:1403] T 3218f18a797c40c9ad6ac64b2670cd19 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.144s user 0.010s sys 0.002s
I20251027 21:07:48.516757 10040 tablet_replica.cc:333] T 3218f18a797c40c9ad6ac64b2670cd19 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:48.516983 10040 raft_consensus.cc:2243] T 3218f18a797c40c9ad6ac64b2670cd19 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:48.517226 10040 raft_consensus.cc:2272] T 3218f18a797c40c9ad6ac64b2670cd19 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.517352 10029 raft_consensus.cc:359] T 3218f18a797c40c9ad6ac64b2670cd19 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:48.517539 10029 raft_consensus.cc:385] T 3218f18a797c40c9ad6ac64b2670cd19 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.517596 10029 raft_consensus.cc:740] T 3218f18a797c40c9ad6ac64b2670cd19 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.520187 10039 raft_consensus.cc:2804] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:48.520311 10039 raft_consensus.cc:493] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:48.520289 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "783d845009d54d20b8e6626c3e73d2a7" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:48.520439 10039 raft_consensus.cc:3060] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:48.520476 9925 raft_consensus.cc:2468] T 783d845009d54d20b8e6626c3e73d2a7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:48.521057 10040 ts_tablet_manager.cc:1916] T 3218f18a797c40c9ad6ac64b2670cd19 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:48.523136 10039 raft_consensus.cc:515] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:48.523738 10039 leader_election.cc:290] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:48.523882 10029 consensus_queue.cc:260] T 3218f18a797c40c9ad6ac64b2670cd19 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:48.524202 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "783d845009d54d20b8e6626c3e73d2a7" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:48.524369 9775 raft_consensus.cc:3060] T 783d845009d54d20b8e6626c3e73d2a7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:48.524479 10029 ts_tablet_manager.cc:1434] T 3218f18a797c40c9ad6ac64b2670cd19 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251027 21:07:48.524791 10029 tablet_bootstrap.cc:492] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:48.527506 9775 raft_consensus.cc:2468] T 783d845009d54d20b8e6626c3e73d2a7 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:48.527910 9566 leader_election.cc:304] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:07:48.528132 10039 raft_consensus.cc:2804] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:48.528393 10039 raft_consensus.cc:697] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:07:48.528584 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "783d845009d54d20b8e6626c3e73d2a7" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:07:48.528663 10039 consensus_queue.cc:237] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:48.528756 9925 raft_consensus.cc:3060] T 783d845009d54d20b8e6626c3e73d2a7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:07:48.530682 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe already in progress: creating tablet
W20251027 21:07:48.530970 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 61cd8ee8e1f8459eb7af5dd48c018e02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61cd8ee8e1f8459eb7af5dd48c018e02 already in progress: creating tablet
W20251027 21:07:48.531181 9297 catalog_manager.cc:4952] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 988052533a1042d9bba5ed5711d09d9c because the tablet was not found. No further retry: Not found: Tablet not found: 988052533a1042d9bba5ed5711d09d9c
I20251027 21:07:48.531437 9925 raft_consensus.cc:2468] T 783d845009d54d20b8e6626c3e73d2a7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
W20251027 21:07:48.532155 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 94a00ff844f0415fa419613d70a318e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94a00ff844f0415fa419613d70a318e6 already in progress: creating tablet
W20251027 21:07:48.532424 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 69b1fb42428748239d8939d98428d8fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69b1fb42428748239d8939d98428d8fe already in progress: creating tablet
I20251027 21:07:48.532471 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 61cd8ee8e1f8459eb7af5dd48c018e02 with delete_type TABLET_DATA_DELETED (replaced by tablet 49d9e5c5960a4534a6603875006e6266) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:48.532630 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 36e168cc44274dae9682b0c1329d4558 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36e168cc44274dae9682b0c1329d4558 already in progress: creating tablet
I20251027 21:07:48.532672 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:48.532845 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 already in progress: creating tablet
I20251027 21:07:48.537817 10029 tablet_bootstrap.cc:654] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.538158 10041 ts_tablet_manager.cc:1929] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.538350 10041 log.cc:1199] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/e9dd86d98cd94cfe9cc2a7b1ab1f979b
I20251027 21:07:48.538797 10041 ts_tablet_manager.cc:1950] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:48.539124 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.539271 9612 tablet_service.cc:1552] Processing DeleteTablet for tablet 988052533a1042d9bba5ed5711d09d9c with delete_type TABLET_DATA_DELETED (replaced by tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.539361 10043 tablet_replica.cc:333] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:48.539523 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 783d845009d54d20b8e6626c3e73d2a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.539548 10043 raft_consensus.cc:2243] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:07:48.539743 10043 pending_rounds.cc:70] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14: Trying to abort 1 pending ops.
I20251027 21:07:48.539809 10043 pending_rounds.cc:77] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510603366576128 op_type: NO_OP noop_request { }
I20251027 21:07:48.539885 10043 raft_consensus.cc:2889] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:07:48.539970 10043 raft_consensus.cc:2272] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.539960 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b (table default.Table_0377267e434442dfb2e1cce27b28dd35 [id=7a31bdff8eab4483b0d88c59e7a2e914]) successfully deleted
I20251027 21:07:48.543594 10043 ts_tablet_manager.cc:1916] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:48.544122 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet d89c3c588a1747d7875ebf45924fb890 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d89c3c588a1747d7875ebf45924fb890 already in progress: creating tablet
I20251027 21:07:48.544396 10041 tablet_replica.cc:333] T a878184a3a444761aff8730bdf43ccaf P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:48.544579 10041 raft_consensus.cc:2243] T a878184a3a444761aff8730bdf43ccaf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:48.544731 10041 raft_consensus.cc:2272] T a878184a3a444761aff8730bdf43ccaf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.545204 10041 ts_tablet_manager.cc:1916] T a878184a3a444761aff8730bdf43ccaf P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:48.546387 9296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20251027 21:07:48.551606 10043 ts_tablet_manager.cc:1929] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:07:48.551803 10043 log.cc:1199] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/e9dd86d98cd94cfe9cc2a7b1ab1f979b
I20251027 21:07:48.552186 10043 ts_tablet_manager.cc:1950] T e9dd86d98cd94cfe9cc2a7b1ab1f979b P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:48.553068 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet e9dd86d98cd94cfe9cc2a7b1ab1f979b (table default.Table_0377267e434442dfb2e1cce27b28dd35 [id=7a31bdff8eab4483b0d88c59e7a2e914]) successfully deleted
I20251027 21:07:48.553398 10043 tablet_replica.cc:333] T 988052533a1042d9bba5ed5711d09d9c P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:48.553619 10043 raft_consensus.cc:2243] T 988052533a1042d9bba5ed5711d09d9c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:48.553792 10043 raft_consensus.cc:2272] T 988052533a1042d9bba5ed5711d09d9c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.554391 9747 tablet_service.cc:1552] Processing DeleteTablet for tablet 783d845009d54d20b8e6626c3e73d2a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:48.487095 10032 raft_consensus.cc:359] T 87ec5a6b9b3c425da4e96cc6736d6310 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:48.554852 10032 raft_consensus.cc:385] T 87ec5a6b9b3c425da4e96cc6736d6310 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.554975 10032 raft_consensus.cc:740] T 87ec5a6b9b3c425da4e96cc6736d6310 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.555295 10032 consensus_queue.cc:260] T 87ec5a6b9b3c425da4e96cc6736d6310 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:48.557523 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet f6249fc5c6174e8e81b9d02d598d5ebd from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:48.558105 10032 ts_tablet_manager.cc:1434] T 87ec5a6b9b3c425da4e96cc6736d6310 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.072s user 0.002s sys 0.000s
I20251027 21:07:48.558486 10032 tablet_bootstrap.cc:492] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:48.560194 10032 tablet_bootstrap.cc:654] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.560889 10040 ts_tablet_manager.cc:1929] T 3218f18a797c40c9ad6ac64b2670cd19 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.561048 10040 log.cc:1199] T 3218f18a797c40c9ad6ac64b2670cd19 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/3218f18a797c40c9ad6ac64b2670cd19
I20251027 21:07:48.561486 10040 ts_tablet_manager.cc:1950] T 3218f18a797c40c9ad6ac64b2670cd19 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:48.557288 10041 ts_tablet_manager.cc:1929] T a878184a3a444761aff8730bdf43ccaf P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.561691 10041 log.cc:1199] T a878184a3a444761aff8730bdf43ccaf P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/a878184a3a444761aff8730bdf43ccaf
I20251027 21:07:48.562080 10041 ts_tablet_manager.cc:1950] T a878184a3a444761aff8730bdf43ccaf P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:48.562095 10043 ts_tablet_manager.cc:1916] T 988052533a1042d9bba5ed5711d09d9c P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:48.563151 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet a878184a3a444761aff8730bdf43ccaf (table default.Table_1c94591b5a734f7a8ce72ed245351b55 [id=77687d6093f741d3905c660d0650ae13]) successfully deleted
I20251027 21:07:48.565333 9887 tablet_service.cc:1505] Processing CreateTablet for tablet cb1686fc7c3a4996aa4462884d30c41f (DEFAULT_TABLE table=default.Table_c28df5bcb6c0432791ff79f424fcdca2 [id=3f2856bb858142989104b3f1d9e7f510]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.565688 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb1686fc7c3a4996aa4462884d30c41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.569202 10029 tablet_bootstrap.cc:492] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:48.569341 10029 ts_tablet_manager.cc:1403] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s
I20251027 21:07:48.569658 9895 tablet_service.cc:1505] Processing CreateTablet for tablet c49775486ff24686b0f45c47e2757aba (DEFAULT_TABLE table=default.Table_b534fdb9e3084d6f9656dfdbb0e01750 [id=dcefc291ce454343964e99be8876837c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.570080 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c49775486ff24686b0f45c47e2757aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.570088 10029 raft_consensus.cc:359] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:48.570274 10029 raft_consensus.cc:385] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.570358 10029 raft_consensus.cc:740] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.570549 10029 consensus_queue.cc:260] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:48.571623 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet f6249fc5c6174e8e81b9d02d598d5ebd
I20251027 21:07:48.571976 9311 catalog_manager.cc:6719] ReplaceTablet: tablet f6249fc5c6174e8e81b9d02d598d5ebd deleted and replaced by tablet 7d8825b80cec4ba68da077ca3b8d89db
I20251027 21:07:48.572894 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 1e5f5b57f5d948f3b2352451ccaacb09 (DEFAULT_TABLE table=default.Table_7a662e3d3c4b4f8cb68a606bdf279049 [id=7f83f7c0f9524147bb778a06dd233a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.573123 9608 tablet_service.cc:1505] Processing CreateTablet for tablet cb1686fc7c3a4996aa4462884d30c41f (DEFAULT_TABLE table=default.Table_c28df5bcb6c0432791ff79f424fcdca2 [id=3f2856bb858142989104b3f1d9e7f510]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.573299 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e5f5b57f5d948f3b2352451ccaacb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.573460 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb1686fc7c3a4996aa4462884d30c41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.574484 9606 tablet_service.cc:1505] Processing CreateTablet for tablet c49775486ff24686b0f45c47e2757aba (DEFAULT_TABLE table=default.Table_b534fdb9e3084d6f9656dfdbb0e01750 [id=dcefc291ce454343964e99be8876837c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.574848 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c49775486ff24686b0f45c47e2757aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.575379 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 1e5f5b57f5d948f3b2352451ccaacb09 (DEFAULT_TABLE table=default.Table_7a662e3d3c4b4f8cb68a606bdf279049 [id=7f83f7c0f9524147bb778a06dd233a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.575676 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e5f5b57f5d948f3b2352451ccaacb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.577544 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet f6249fc5c6174e8e81b9d02d598d5ebd with delete_type TABLET_DATA_DELETED (replaced by tablet 7d8825b80cec4ba68da077ca3b8d89db) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.577667 10029 ts_tablet_manager.cc:1434] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251027 21:07:48.577934 10029 tablet_bootstrap.cc:492] T d642da56a67449aa95932180bc5b2c5f P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:48.579352 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 9248f458baca409aa3505cc0db321b34 (DEFAULT_TABLE table=default.Table_d9744e5483fd46a581cff3e8e787a6bd [id=64df643709144959945c55df034b77a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.579699 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9248f458baca409aa3505cc0db321b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.579795 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet f6249fc5c6174e8e81b9d02d598d5ebd with delete_type TABLET_DATA_DELETED (replaced by tablet 7d8825b80cec4ba68da077ca3b8d89db) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.580269 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 9248f458baca409aa3505cc0db321b34 (DEFAULT_TABLE table=default.Table_d9744e5483fd46a581cff3e8e787a6bd [id=64df643709144959945c55df034b77a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.580693 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9248f458baca409aa3505cc0db321b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.587890 9609 tablet_service.cc:1505] Processing CreateTablet for tablet e07b8aa0a41241d79254c4dae8738e79 (DEFAULT_TABLE table=default.Table_e68b56d3ddd94f308243ec6a03ea9e6e [id=0a94e88b69094ffe8be3367470be86da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.588286 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e07b8aa0a41241d79254c4dae8738e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.593182 10029 tablet_bootstrap.cc:654] T d642da56a67449aa95932180bc5b2c5f P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.595845 9898 tablet_service.cc:1505] Processing CreateTablet for tablet e07b8aa0a41241d79254c4dae8738e79 (DEFAULT_TABLE table=default.Table_e68b56d3ddd94f308243ec6a03ea9e6e [id=0a94e88b69094ffe8be3367470be86da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.596304 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e07b8aa0a41241d79254c4dae8738e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.599179 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 1776590d8387494da5dfbd08db17f4b1 (DEFAULT_TABLE table=default.Table_758d1e73474647eaa8a15b6cff350af6 [id=8ed572891f9b41cba8ed984911fa6b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.599267 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 1776590d8387494da5dfbd08db17f4b1 (DEFAULT_TABLE table=default.Table_758d1e73474647eaa8a15b6cff350af6 [id=8ed572891f9b41cba8ed984911fa6b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.599613 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1776590d8387494da5dfbd08db17f4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.599596 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1776590d8387494da5dfbd08db17f4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.602820 9892 tablet_service.cc:1505] Processing CreateTablet for tablet cfce0e5c837743ea969374244163be3e (DEFAULT_TABLE table=default.Table_4568794575f345c08a45afeb2d4f9648 [id=13a25b18846c4580a337b91cee116a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.603220 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfce0e5c837743ea969374244163be3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.604632 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 77797eea3e9c4f8099dc9d58f116200b (DEFAULT_TABLE table=default.Table_8c615d69dbaf418b9058374ee17309d5 [id=e7b7bdb2ea7f468c98bfe88f900a5be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.604972 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77797eea3e9c4f8099dc9d58f116200b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.606484 9612 tablet_service.cc:1505] Processing CreateTablet for tablet 533f231e3ff64531ae2331409a961425 (DEFAULT_TABLE table=default.Table_fd19a2c8dc2f4164a0326865a21cd141 [id=b45a015e747e4f0db54899a28ab97a7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.606891 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533f231e3ff64531ae2331409a961425. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.608218 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 533f231e3ff64531ae2331409a961425 (DEFAULT_TABLE table=default.Table_fd19a2c8dc2f4164a0326865a21cd141 [id=b45a015e747e4f0db54899a28ab97a7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.608570 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533f231e3ff64531ae2331409a961425. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.609308 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 77797eea3e9c4f8099dc9d58f116200b (DEFAULT_TABLE table=default.Table_8c615d69dbaf418b9058374ee17309d5 [id=e7b7bdb2ea7f468c98bfe88f900a5be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.609622 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77797eea3e9c4f8099dc9d58f116200b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.612748 9610 tablet_service.cc:1505] Processing CreateTablet for tablet a9cc64ad85334daa9a09538e24ebd8a5 (DEFAULT_TABLE table=default.Table_03046ffdd6ca47e8aa4cde13574dc119 [id=8a893ef2396748f08f737fddfb76fd65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.613090 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9cc64ad85334daa9a09538e24ebd8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.615733 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ec86de5add82433b81e1060e76d559e3" } modify_external_catalogs: true
I20251027 21:07:48.615935 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ec86de5add82433b81e1060e76d559e3" } modify_external_catalogs: true
I20251027 21:07:48.616411 9886 tablet_service.cc:1505] Processing CreateTablet for tablet 779b7f63b0fa4f9ab47cf5799b3e085d (DEFAULT_TABLE table=default.Table_d98a9c0b61d649f2945cb6c199f70a64 [id=6c62d93feb804d6aa1d442f8458b7ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.616717 9886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779b7f63b0fa4f9ab47cf5799b3e085d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.617282 9613 tablet_service.cc:1505] Processing CreateTablet for tablet 779b7f63b0fa4f9ab47cf5799b3e085d (DEFAULT_TABLE table=default.Table_d98a9c0b61d649f2945cb6c199f70a64 [id=6c62d93feb804d6aa1d442f8458b7ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.617615 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779b7f63b0fa4f9ab47cf5799b3e085d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.618435 9900 tablet_service.cc:1505] Processing CreateTablet for tablet a9cc64ad85334daa9a09538e24ebd8a5 (DEFAULT_TABLE table=default.Table_03046ffdd6ca47e8aa4cde13574dc119 [id=8a893ef2396748f08f737fddfb76fd65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.618815 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9cc64ad85334daa9a09538e24ebd8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.622241 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 9c4369a10bf846fb94c9b71a70f21600
I20251027 21:07:48.623752 9611 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c4369a10bf846fb94c9b71a70f21600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.623966 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_053064f4a69d499ba2862255e92fbaab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.624701 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_053064f4a69d499ba2862255e92fbaab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.625895 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b23313ec4d7b4e32a4283dd404578616"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.626361 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b23313ec4d7b4e32a4283dd404578616 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.626330 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4568794575f345c08a45afeb2d4f9648" } modify_external_catalogs: true
I20251027 21:07:48.627108 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4568794575f345c08a45afeb2d4f9648" } modify_external_catalogs: true
I20251027 21:07:48.627458 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c4369a10bf846fb94c9b71a70f21600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.627868 9894 tablet_service.cc:1505] Processing CreateTablet for tablet 7f7ad06a15c04efbb701634e14904351 (DEFAULT_TABLE table=default.Table_274a6d6e3de14343aac38fb44fe222a2 [id=2bc39b5bc02645ef830eee14e92707db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.628155 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f7ad06a15c04efbb701634e14904351. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.629016 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3da5a86d9f104d1ba39726b5a6dcec8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:48.629453 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_42431224890f43859abe3dedfae4db51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.630014 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42431224890f43859abe3dedfae4db51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.630179 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d2bf88aca4cc44df99d13783ea2722f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:48.630358 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2fd9ca76e8a0406cbfd32f91038ccc0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.630673 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2bf88aca4cc44df99d13783ea2722f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:48.630769 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fd9ca76e8a0406cbfd32f91038ccc0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.631424 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_178e5d554faa4e119ac8f52da2d284d7" } modify_external_catalogs: true
I20251027 21:07:48.631637 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_178e5d554faa4e119ac8f52da2d284d7" } modify_external_catalogs: true
W20251027 21:07:48.629455 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3da5a86d9f104d1ba39726b5a6dcec8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.641095 9611 tablet_service.cc:1552] Processing DeleteTablet for tablet 3218f18a797c40c9ad6ac64b2670cd19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.644142 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet cfce0e5c837743ea969374244163be3e
I20251027 21:07:48.645529 9611 tablet_service.cc:1552] Processing DeleteTablet for tablet cfce0e5c837743ea969374244163be3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.647588 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3886813b08244afa9a2d851978407392" } new_table_name: "default.Table_908c9569154d403eb8aeab2a65da7f67" modify_external_catalogs: true
I20251027 21:07:48.647698 9611 tablet_service.cc:1552] Processing DeleteTablet for tablet 63662f175a1c450f8f67f37da971db37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.648883 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6abfd304277c48bfb3018f4e211e13b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.649314 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6abfd304277c48bfb3018f4e211e13b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.654634 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 7f7ad06a15c04efbb701634e14904351 (DEFAULT_TABLE table=default.Table_274a6d6e3de14343aac38fb44fe222a2 [id=2bc39b5bc02645ef830eee14e92707db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.654726 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet cfce0e5c837743ea969374244163be3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.655180 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f7ad06a15c04efbb701634e14904351. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:48.663781 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet ba8b125474bc42a5946e8f2dd718f7f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba8b125474bc42a5946e8f2dd718f7f4 already in progress: creating tablet
W20251027 21:07:48.664443 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet e3c9f64ee0544e59add4f73bca59df98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3c9f64ee0544e59add4f73bca59df98 already in progress: creating tablet
I20251027 21:07:48.666872 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet d642da56a67449aa95932180bc5b2c5f
I20251027 21:07:48.668650 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet d642da56a67449aa95932180bc5b2c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.673849 10043 ts_tablet_manager.cc:1929] T 988052533a1042d9bba5ed5711d09d9c P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.674342 10041 tablet_replica.cc:333] T 783d845009d54d20b8e6626c3e73d2a7 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:48.676997 10041 raft_consensus.cc:2243] T 783d845009d54d20b8e6626c3e73d2a7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:48.677292 10041 raft_consensus.cc:2272] T 783d845009d54d20b8e6626c3e73d2a7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.678122 10041 ts_tablet_manager.cc:1916] T 783d845009d54d20b8e6626c3e73d2a7 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:48.679379 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_780df15afd4d45d1bd65f9a2ede532c3" } new_table_name: "default.Table_e4d0e9336a834d6d8ab29ac0b1de7ace" modify_external_catalogs: true
I20251027 21:07:48.679497 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_35757fdc53ab433fb201f944c7906ff2" } new_table_name: "default.Table_b84197018ff44b40b7219e21846339f2" modify_external_catalogs: true
I20251027 21:07:48.680614 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet d642da56a67449aa95932180bc5b2c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.680963 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_1e51d9a2db704a699f7bb8a2f6b3a4f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:48.681185 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8c615d69dbaf418b9058374ee17309d5" } new_table_name: "default.Table_9ddfcdda61854c59a1295af022b0ddc0" modify_external_catalogs: true
W20251027 21:07:48.681589 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e51d9a2db704a699f7bb8a2f6b3a4f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.682296 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d98a9c0b61d649f2945cb6c199f70a64" } modify_external_catalogs: true
I20251027 21:07:48.682477 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d98a9c0b61d649f2945cb6c199f70a64" } modify_external_catalogs: true
I20251027 21:07:48.682519 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_eca1294e7d18437bad196fc2fc5dd3fc" } new_table_name: "default.Table_530197464d684950839a65bdf56549b8" modify_external_catalogs: true
I20251027 21:07:48.682300 9305 master_service.cc:954] ReplaceTablet: received request to replace tablet 019a892bfdd54e62bc3560a341d5c2da from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:48.683792 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d2ff301b3e2b4602a2483859973a02d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.684139 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2ff301b3e2b4602a2483859973a02d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.675055 10029 tablet_bootstrap.cc:492] T d642da56a67449aa95932180bc5b2c5f P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:48.685005 10043 log.cc:1199] T 988052533a1042d9bba5ed5711d09d9c P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/988052533a1042d9bba5ed5711d09d9c
I20251027 21:07:48.685160 10029 ts_tablet_manager.cc:1403] T d642da56a67449aa95932180bc5b2c5f P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.107s user 0.017s sys 0.005s
I20251027 21:07:48.686005 10029 raft_consensus.cc:359] T d642da56a67449aa95932180bc5b2c5f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.565896 9754 tablet_service.cc:1505] Processing CreateTablet for tablet c49775486ff24686b0f45c47e2757aba (DEFAULT_TABLE table=default.Table_b534fdb9e3084d6f9656dfdbb0e01750 [id=dcefc291ce454343964e99be8876837c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.686357 10029 raft_consensus.cc:385] T d642da56a67449aa95932180bc5b2c5f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.686494 10029 raft_consensus.cc:740] T d642da56a67449aa95932180bc5b2c5f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.686573 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c49775486ff24686b0f45c47e2757aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.686754 10029 consensus_queue.cc:260] T d642da56a67449aa95932180bc5b2c5f P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.687322 10029 ts_tablet_manager.cc:1434] T d642da56a67449aa95932180bc5b2c5f P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:48.687348 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 3218f18a797c40c9ad6ac64b2670cd19 (table default.Table_40644b4bc09f4693866503d3b5fb7df7 [id=09c88f2f898844b88b70f5ed9e837841]) successfully deleted
I20251027 21:07:48.687641 10029 tablet_bootstrap.cc:492] T 84305bcf21b64d2c8c1df015a2e9615c P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:48.689332 10029 tablet_bootstrap.cc:654] T 84305bcf21b64d2c8c1df015a2e9615c P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:48.689388 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2d659e4d73ea4ff3975a4b58a794090a consensus state has no leader
I20251027 21:07:48.690294 10039 raft_consensus.cc:493] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:48.690423 10039 raft_consensus.cc:515] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:48.691030 10039 leader_election.cc:290] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:48.692140 10040 tablet_replica.cc:333] T 783d845009d54d20b8e6626c3e73d2a7 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:48.692194 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63662f175a1c450f8f67f37da971db37" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:48.692323 10040 raft_consensus.cc:2243] T 783d845009d54d20b8e6626c3e73d2a7 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:48.566641 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 1e5f5b57f5d948f3b2352451ccaacb09 (DEFAULT_TABLE table=default.Table_7a662e3d3c4b4f8cb68a606bdf279049 [id=7f83f7c0f9524147bb778a06dd233a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.692343 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d5ad2155dc3543579943b617c2fdb9fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:48.692493 10040 raft_consensus.cc:2272] T 783d845009d54d20b8e6626c3e73d2a7 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.692623 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e5f5b57f5d948f3b2352451ccaacb09. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:48.692764 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5ad2155dc3543579943b617c2fdb9fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.692930 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63662f175a1c450f8f67f37da971db37" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
W20251027 21:07:48.693450 9566 leader_election.cc:343] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:07:48.693785 9568 leader_election.cc:343] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:48.693917 9568 leader_election.cc:304] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:48.694284 10039 raft_consensus.cc:2749] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:48.694458 10043 ts_tablet_manager.cc:1950] T 988052533a1042d9bba5ed5711d09d9c P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:48.695540 10043 tablet_replica.cc:333] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:48.695624 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 988052533a1042d9bba5ed5711d09d9c (table default.Table_4c5433970bd64291a2b1289a84f35590 [id=a1b2d46db3764c7f97712a0c5f891881]) successfully deleted
I20251027 21:07:48.695708 10043 raft_consensus.cc:2243] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:07:48.695995 10043 pending_rounds.cc:70] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14: Trying to abort 1 pending ops.
I20251027 21:07:48.696064 10043 pending_rounds.cc:77] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510603898429440 op_type: NO_OP noop_request { }
I20251027 21:07:48.696209 10043 raft_consensus.cc:2889] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:07:48.696321 10043 raft_consensus.cc:2272] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.567271 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 9248f458baca409aa3505cc0db321b34 (DEFAULT_TABLE table=default.Table_d9744e5483fd46a581cff3e8e787a6bd [id=64df643709144959945c55df034b77a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.698278 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9248f458baca409aa3505cc0db321b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.567905 9747 tablet_service.cc:1505] Processing CreateTablet for tablet cb1686fc7c3a4996aa4462884d30c41f (DEFAULT_TABLE table=default.Table_c28df5bcb6c0432791ff79f424fcdca2 [id=3f2856bb858142989104b3f1d9e7f510]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.702612 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb1686fc7c3a4996aa4462884d30c41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.702863 10029 tablet_bootstrap.cc:492] T 84305bcf21b64d2c8c1df015a2e9615c P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:48.702988 10029 ts_tablet_manager.cc:1403] T 84305bcf21b64d2c8c1df015a2e9615c P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.002s
I20251027 21:07:48.703778 10029 raft_consensus.cc:359] T 84305bcf21b64d2c8c1df015a2e9615c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.704185 10029 raft_consensus.cc:385] T 84305bcf21b64d2c8c1df015a2e9615c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.704289 10029 raft_consensus.cc:740] T 84305bcf21b64d2c8c1df015a2e9615c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.572535 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet f6249fc5c6174e8e81b9d02d598d5ebd with delete_type TABLET_DATA_DELETED (replaced by tablet 7d8825b80cec4ba68da077ca3b8d89db) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:48.592607 9752 tablet_service.cc:1505] Processing CreateTablet for tablet e07b8aa0a41241d79254c4dae8738e79 (DEFAULT_TABLE table=default.Table_e68b56d3ddd94f308243ec6a03ea9e6e [id=0a94e88b69094ffe8be3367470be86da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.704530 10029 consensus_queue.cc:260] T 84305bcf21b64d2c8c1df015a2e9615c P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.704802 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e07b8aa0a41241d79254c4dae8738e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.705143 10029 ts_tablet_manager.cc:1434] T 84305bcf21b64d2c8c1df015a2e9615c P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:48.705420 10029 tablet_bootstrap.cc:492] T 5c87a5450d354506852c4b20bf0541d2 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:48.707144 10029 tablet_bootstrap.cc:654] T 5c87a5450d354506852c4b20bf0541d2 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.709968 10029 tablet_bootstrap.cc:492] T 5c87a5450d354506852c4b20bf0541d2 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:48.594954 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 1776590d8387494da5dfbd08db17f4b1 (DEFAULT_TABLE table=default.Table_758d1e73474647eaa8a15b6cff350af6 [id=8ed572891f9b41cba8ed984911fa6b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.710111 10029 ts_tablet_manager.cc:1403] T 5c87a5450d354506852c4b20bf0541d2 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20251027 21:07:48.710364 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1776590d8387494da5dfbd08db17f4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.602095 9608 tablet_service.cc:1505] Processing CreateTablet for tablet cfce0e5c837743ea969374244163be3e (DEFAULT_TABLE table=default.Table_4568794575f345c08a45afeb2d4f9648 [id=13a25b18846c4580a337b91cee116a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.710871 10029 raft_consensus.cc:359] T 5c87a5450d354506852c4b20bf0541d2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:48.711057 10029 raft_consensus.cc:385] T 5c87a5450d354506852c4b20bf0541d2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.711129 10029 raft_consensus.cc:740] T 5c87a5450d354506852c4b20bf0541d2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.711395 10029 consensus_queue.cc:260] T 5c87a5450d354506852c4b20bf0541d2 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:48.711997 10029 ts_tablet_manager.cc:1434] T 5c87a5450d354506852c4b20bf0541d2 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:07:48.601043 9749 tablet_service.cc:1505] Processing CreateTablet for tablet cfce0e5c837743ea969374244163be3e (DEFAULT_TABLE table=default.Table_4568794575f345c08a45afeb2d4f9648 [id=13a25b18846c4580a337b91cee116a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.603013 9751 tablet_service.cc:1505] Processing CreateTablet for tablet 533f231e3ff64531ae2331409a961425 (DEFAULT_TABLE table=default.Table_fd19a2c8dc2f4164a0326865a21cd141 [id=b45a015e747e4f0db54899a28ab97a7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.606040 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 77797eea3e9c4f8099dc9d58f116200b (DEFAULT_TABLE table=default.Table_8c615d69dbaf418b9058374ee17309d5 [id=e7b7bdb2ea7f468c98bfe88f900a5be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.606942 9745 tablet_service.cc:1505] Processing CreateTablet for tablet a9cc64ad85334daa9a09538e24ebd8a5 (DEFAULT_TABLE table=default.Table_03046ffdd6ca47e8aa4cde13574dc119 [id=8a893ef2396748f08f737fddfb76fd65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.614889 9744 tablet_service.cc:1505] Processing CreateTablet for tablet 779b7f63b0fa4f9ab47cf5799b3e085d (DEFAULT_TABLE table=default.Table_d98a9c0b61d649f2945cb6c199f70a64 [id=6c62d93feb804d6aa1d442f8458b7ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.616986 9743 tablet_service.cc:1505] Processing CreateTablet for tablet 7f7ad06a15c04efbb701634e14904351 (DEFAULT_TABLE table=default.Table_274a6d6e3de14343aac38fb44fe222a2 [id=2bc39b5bc02645ef830eee14e92707db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.713582 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfce0e5c837743ea969374244163be3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.623598 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c4369a10bf846fb94c9b71a70f21600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:48.648247 9741 tablet_service.cc:1552] Processing DeleteTablet for tablet cfce0e5c837743ea969374244163be3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:48.667734 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet d642da56a67449aa95932180bc5b2c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:48.715137 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77797eea3e9c4f8099dc9d58f116200b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.715782 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533f231e3ff64531ae2331409a961425. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.716365 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779b7f63b0fa4f9ab47cf5799b3e085d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.716992 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9cc64ad85334daa9a09538e24ebd8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.716975 9740 tablet_service.cc:1505] Processing CreateTablet for tablet ae124514964440c5b88b16ac9f8a2567 (DEFAULT_TABLE table=default.Table_053064f4a69d499ba2862255e92fbaab [id=83ec44dba5c444aba48846811a5f5fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.717535 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae124514964440c5b88b16ac9f8a2567. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.717772 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f7ad06a15c04efbb701634e14904351. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.718034 9741 tablet_service.cc:1505] Processing CreateTablet for tablet 94e94cd50c91429d8da9804149501c86 (DEFAULT_TABLE table=default.Table_b23313ec4d7b4e32a4283dd404578616 [id=6a545a959eb74d0eb589f1be92a0bc0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.718557 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94e94cd50c91429d8da9804149501c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.718978 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet 44783760defd433a972c9cd83933249c from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:48.720165 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 7d8825b80cec4ba68da077ca3b8d89db (DEFAULT_TABLE table=default.Table_943417713b3b456ab4ca691d2fb168fd [id=1cb32eea25114c6dbcb3b28044036c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.721016 9604 tablet_service.cc:1505] Processing CreateTablet for tablet 49d9e5c5960a4534a6603875006e6266 (DEFAULT_TABLE table=default.Table_52c53bafe66b4c66b99e5b26e4df7ef5 [id=27dc835f316f41e091be9b2b1f560bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.723774 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 7d8825b80cec4ba68da077ca3b8d89db (DEFAULT_TABLE table=default.Table_943417713b3b456ab4ca691d2fb168fd [id=1cb32eea25114c6dbcb3b28044036c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.724231 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 94e94cd50c91429d8da9804149501c86 (DEFAULT_TABLE table=default.Table_b23313ec4d7b4e32a4283dd404578616 [id=6a545a959eb74d0eb589f1be92a0bc0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.724677 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 49d9e5c5960a4534a6603875006e6266 (DEFAULT_TABLE table=default.Table_52c53bafe66b4c66b99e5b26e4df7ef5 [id=27dc835f316f41e091be9b2b1f560bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.727205 10032 tablet_bootstrap.cc:492] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:48.727399 10032 ts_tablet_manager.cc:1403] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.169s user 0.001s sys 0.003s
I20251027 21:07:48.728232 10032 raft_consensus.cc:359] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.728482 10032 raft_consensus.cc:385] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.728621 10032 raft_consensus.cc:740] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.728911 10032 consensus_queue.cc:260] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.729642 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d8825b80cec4ba68da077ca3b8d89db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.729758 10029 tablet_bootstrap.cc:492] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:48.730958 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49d9e5c5960a4534a6603875006e6266. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:48.731832 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cad3a3e910ac42399cf371ad82c89ff7 consensus state has no leader
I20251027 21:07:48.732198 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94e94cd50c91429d8da9804149501c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.732424 10029 tablet_bootstrap.cc:654] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.733430 10032 ts_tablet_manager.cc:1434] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:48.733476 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfce0e5c837743ea969374244163be3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.733809 10032 tablet_bootstrap.cc:492] T 9cec6294051b41868c9c7c7fadafa138 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:48.736419 10032 tablet_bootstrap.cc:654] T 9cec6294051b41868c9c7c7fadafa138 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.737437 9886 tablet_service.cc:1505] Processing CreateTablet for tablet 49d9e5c5960a4534a6603875006e6266 (DEFAULT_TABLE table=default.Table_52c53bafe66b4c66b99e5b26e4df7ef5 [id=27dc835f316f41e091be9b2b1f560bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.737901 9886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49d9e5c5960a4534a6603875006e6266. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.747852 10032 tablet_bootstrap.cc:492] T 9cec6294051b41868c9c7c7fadafa138 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:48.748369 10032 ts_tablet_manager.cc:1403] T 9cec6294051b41868c9c7c7fadafa138 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20251027 21:07:48.751996 10032 raft_consensus.cc:359] T 9cec6294051b41868c9c7c7fadafa138 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.753129 10032 raft_consensus.cc:385] T 9cec6294051b41868c9c7c7fadafa138 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.753759 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 94e94cd50c91429d8da9804149501c86 (DEFAULT_TABLE table=default.Table_b23313ec4d7b4e32a4283dd404578616 [id=6a545a959eb74d0eb589f1be92a0bc0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.754565 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94e94cd50c91429d8da9804149501c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.757733 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 7d8825b80cec4ba68da077ca3b8d89db (DEFAULT_TABLE table=default.Table_943417713b3b456ab4ca691d2fb168fd [id=1cb32eea25114c6dbcb3b28044036c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.758237 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d8825b80cec4ba68da077ca3b8d89db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.758822 9895 tablet_service.cc:1505] Processing CreateTablet for tablet ae124514964440c5b88b16ac9f8a2567 (DEFAULT_TABLE table=default.Table_053064f4a69d499ba2862255e92fbaab [id=83ec44dba5c444aba48846811a5f5fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.760823 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae124514964440c5b88b16ac9f8a2567. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.753532 10032 raft_consensus.cc:740] T 9cec6294051b41868c9c7c7fadafa138 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.771802 10032 consensus_queue.cc:260] T 9cec6294051b41868c9c7c7fadafa138 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.772418 10032 ts_tablet_manager.cc:1434] T 9cec6294051b41868c9c7c7fadafa138 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20251027 21:07:48.772894 10032 tablet_bootstrap.cc:492] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:48.791656 10041 ts_tablet_manager.cc:1929] T 783d845009d54d20b8e6626c3e73d2a7 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.793227 10041 log.cc:1199] T 783d845009d54d20b8e6626c3e73d2a7 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/783d845009d54d20b8e6626c3e73d2a7
I20251027 21:07:48.794103 10032 tablet_bootstrap.cc:654] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.796663 10041 ts_tablet_manager.cc:1950] T 783d845009d54d20b8e6626c3e73d2a7 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:48.799468 9615 tablet_service.cc:1505] Processing CreateTablet for tablet ae124514964440c5b88b16ac9f8a2567 (DEFAULT_TABLE table=default.Table_053064f4a69d499ba2862255e92fbaab [id=83ec44dba5c444aba48846811a5f5fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:48.800530 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49d9e5c5960a4534a6603875006e6266. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.800565 10031 tablet_bootstrap.cc:492] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:48.800760 10031 ts_tablet_manager.cc:1403] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.327s user 0.085s sys 0.089s
I20251027 21:07:48.801537 10031 raft_consensus.cc:359] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.801704 10031 raft_consensus.cc:385] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.801790 10031 raft_consensus.cc:740] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.802055 10031 consensus_queue.cc:260] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.807354 10031 ts_tablet_manager.cc:1434] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251027 21:07:48.807678 10031 tablet_bootstrap.cc:492] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:48.808358 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d8825b80cec4ba68da077ca3b8d89db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.813208 10040 ts_tablet_manager.cc:1916] T 783d845009d54d20b8e6626c3e73d2a7 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:48.818477 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 783d845009d54d20b8e6626c3e73d2a7 (table default.Table_53596f51be6a408a8987da9e37e4d779 [id=117c8f5066e149db8482c2601b040e17]) successfully deleted
I20251027 21:07:48.818784 10041 tablet_replica.cc:333] T 3218f18a797c40c9ad6ac64b2670cd19 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:48.818964 10041 raft_consensus.cc:2243] T 3218f18a797c40c9ad6ac64b2670cd19 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:48.819132 10041 raft_consensus.cc:2272] T 3218f18a797c40c9ad6ac64b2670cd19 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.821333 10041 ts_tablet_manager.cc:1916] T 3218f18a797c40c9ad6ac64b2670cd19 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:48.821630 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 33e41145c79b4465b8d564741174b8e0 consensus state has no leader
I20251027 21:07:48.828007 10039 raft_consensus.cc:493] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:48.828424 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae124514964440c5b88b16ac9f8a2567. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:48.829193 10041 ts_tablet_manager.cc:1929] T 3218f18a797c40c9ad6ac64b2670cd19 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.829324 10041 log.cc:1199] T 3218f18a797c40c9ad6ac64b2670cd19 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/3218f18a797c40c9ad6ac64b2670cd19
I20251027 21:07:48.829646 10041 ts_tablet_manager.cc:1950] T 3218f18a797c40c9ad6ac64b2670cd19 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:48.829213 10039 raft_consensus.cc:515] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:48.831465 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87ec5a6b9b3c425da4e96cc6736d6310" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:48.831669 9925 raft_consensus.cc:2468] T 87ec5a6b9b3c425da4e96cc6736d6310 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:48.820122 10031 tablet_bootstrap.cc:654] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.831190 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87ec5a6b9b3c425da4e96cc6736d6310" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:48.834187 9568 leader_election.cc:304] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:07:48.834369 9775 raft_consensus.cc:2468] T 87ec5a6b9b3c425da4e96cc6736d6310 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:48.834841 10039 leader_election.cc:290] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:48.835223 10032 tablet_bootstrap.cc:492] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:48.835367 10032 ts_tablet_manager.cc:1403] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.004s
I20251027 21:07:48.835981 10089 raft_consensus.cc:2804] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:48.836117 10089 raft_consensus.cc:493] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:48.836160 10032 raft_consensus.cc:359] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.836339 10032 raft_consensus.cc:385] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.836431 10032 raft_consensus.cc:740] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.836959 10089 raft_consensus.cc:3060] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:48.837303 10032 consensus_queue.cc:260] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.838096 10032 ts_tablet_manager.cc:1434] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251027 21:07:48.838433 10032 tablet_bootstrap.cc:492] T f5da8d7ecccc4c3492f9d0d30ffea423 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:48.840029 10032 tablet_bootstrap.cc:654] T f5da8d7ecccc4c3492f9d0d30ffea423 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.841324 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 3218f18a797c40c9ad6ac64b2670cd19 (table default.Table_40644b4bc09f4693866503d3b5fb7df7 [id=09c88f2f898844b88b70f5ed9e837841]) successfully deleted
I20251027 21:07:48.843962 10040 ts_tablet_manager.cc:1929] T 783d845009d54d20b8e6626c3e73d2a7 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.844167 10040 log.cc:1199] T 783d845009d54d20b8e6626c3e73d2a7 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/783d845009d54d20b8e6626c3e73d2a7
W20251027 21:07:48.844300 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 61cd8ee8e1f8459eb7af5dd48c018e02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61cd8ee8e1f8459eb7af5dd48c018e02 already in progress: creating tablet
W20251027 21:07:48.844388 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 77797eea3e9c4f8099dc9d58f116200b consensus state has no leader
W20251027 21:07:48.844991 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe already in progress: creating tablet
W20251027 21:07:48.845664 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet f6249fc5c6174e8e81b9d02d598d5ebd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6249fc5c6174e8e81b9d02d598d5ebd already in progress: creating tablet
I20251027 21:07:48.846467 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet 2d659e4d73ea4ff3975a4b58a794090a from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:48.846719 10043 ts_tablet_manager.cc:1916] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:48.847381 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 9c4369a10bf846fb94c9b71a70f21600 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c4369a10bf846fb94c9b71a70f21600 already in progress: creating tablet
W20251027 21:07:48.847980 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet cfce0e5c837743ea969374244163be3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfce0e5c837743ea969374244163be3e already in progress: creating tablet
W20251027 21:07:48.848492 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet d642da56a67449aa95932180bc5b2c5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d642da56a67449aa95932180bc5b2c5f already in progress: creating tablet
I20251027 21:07:48.847453 9611 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c87a5450d354506852c4b20bf0541d2 with delete_type TABLET_DATA_DELETED (replaced by tablet 77b2d15e86b244708a2d07f88bf8278d) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.849059 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet 84305bcf21b64d2c8c1df015a2e9615c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.849867 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0ac2c74fb86c48459bad366bd8efccd5" } modify_external_catalogs: true
I20251027 21:07:48.850026 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0ac2c74fb86c48459bad366bd8efccd5" } modify_external_catalogs: true
I20251027 21:07:48.852244 10089 raft_consensus.cc:515] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:48.853680 10089 leader_election.cc:290] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:48.853766 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87ec5a6b9b3c425da4e96cc6736d6310" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:07:48.853945 9925 raft_consensus.cc:3060] T 87ec5a6b9b3c425da4e96cc6736d6310 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:48.854195 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87ec5a6b9b3c425da4e96cc6736d6310" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:48.854362 9775 raft_consensus.cc:3060] T 87ec5a6b9b3c425da4e96cc6736d6310 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:48.856976 9925 raft_consensus.cc:2468] T 87ec5a6b9b3c425da4e96cc6736d6310 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:48.857229 9775 raft_consensus.cc:2468] T 87ec5a6b9b3c425da4e96cc6736d6310 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:48.857681 9566 leader_election.cc:304] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
W20251027 21:07:48.858884 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 970fbf01a4c3406dafe1ea4cd005ea23 consensus state has no leader
I20251027 21:07:48.859129 9306 master_service.cc:954] ReplaceTablet: received request to replace tablet 1776590d8387494da5dfbd08db17f4b1 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:48.859601 10031 tablet_bootstrap.cc:492] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:48.859802 10031 ts_tablet_manager.cc:1403] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.007s
I20251027 21:07:48.860749 10031 raft_consensus.cc:359] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.860997 10031 raft_consensus.cc:385] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.861387 10031 raft_consensus.cc:740] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.861622 10031 consensus_queue.cc:260] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.862092 10031 ts_tablet_manager.cc:1434] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:48.863742 10089 raft_consensus.cc:2804] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:48.863900 10031 tablet_bootstrap.cc:492] T 12aa6e3af0694429b45bef5af3592377 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:48.864033 10089 raft_consensus.cc:697] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:07:48.864342 10089 consensus_queue.cc:237] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:48.866047 10031 tablet_bootstrap.cc:654] T 12aa6e3af0694429b45bef5af3592377 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.866904 10050 raft_consensus.cc:493] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:48.867067 10050 raft_consensus.cc:515] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.867719 10050 leader_election.cc:290] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:48.868597 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0029b70ed3ff4db9a816a6190d36bf88" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:48.868556 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0029b70ed3ff4db9a816a6190d36bf88" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:48.869040 9859 leader_election.cc:343] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:07:48.869464 9860 leader_election.cc:343] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:48.869678 9860 leader_election.cc:304] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:48.870157 10050 raft_consensus.cc:2749] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:48.875257 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet b1bdb2be5032415187c2826c14e75f07 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:48.876886 10043 ts_tablet_manager.cc:1929] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:07:48.879626 10032 tablet_bootstrap.cc:492] T f5da8d7ecccc4c3492f9d0d30ffea423 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:48.879808 10032 ts_tablet_manager.cc:1403] T f5da8d7ecccc4c3492f9d0d30ffea423 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.010s
I20251027 21:07:48.880509 10032 raft_consensus.cc:359] T f5da8d7ecccc4c3492f9d0d30ffea423 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.880673 10032 raft_consensus.cc:385] T f5da8d7ecccc4c3492f9d0d30ffea423 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.880728 10032 raft_consensus.cc:740] T f5da8d7ecccc4c3492f9d0d30ffea423 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.881713 10032 consensus_queue.cc:260] T f5da8d7ecccc4c3492f9d0d30ffea423 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.882197 10032 ts_tablet_manager.cc:1434] T f5da8d7ecccc4c3492f9d0d30ffea423 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:48.884449 10032 tablet_bootstrap.cc:492] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:48.885324 10031 tablet_bootstrap.cc:492] T 12aa6e3af0694429b45bef5af3592377 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:48.885473 10031 ts_tablet_manager.cc:1403] T 12aa6e3af0694429b45bef5af3592377 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.003s
I20251027 21:07:48.886137 10032 tablet_bootstrap.cc:654] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.886391 10040 ts_tablet_manager.cc:1950] T 783d845009d54d20b8e6626c3e73d2a7 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:48.886359 10031 raft_consensus.cc:359] T 12aa6e3af0694429b45bef5af3592377 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.886559 10031 raft_consensus.cc:385] T 12aa6e3af0694429b45bef5af3592377 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.886660 10031 raft_consensus.cc:740] T 12aa6e3af0694429b45bef5af3592377 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.886909 10031 consensus_queue.cc:260] T 12aa6e3af0694429b45bef5af3592377 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.887538 10043 log.cc:1199] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/783d845009d54d20b8e6626c3e73d2a7
I20251027 21:07:48.887660 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 783d845009d54d20b8e6626c3e73d2a7 (table default.Table_53596f51be6a408a8987da9e37e4d779 [id=117c8f5066e149db8482c2601b040e17]) successfully deleted
I20251027 21:07:48.887351 10031 ts_tablet_manager.cc:1434] T 12aa6e3af0694429b45bef5af3592377 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:48.888041 10029 tablet_bootstrap.cc:492] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:48.888168 10029 ts_tablet_manager.cc:1403] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.159s user 0.050s sys 0.033s
I20251027 21:07:48.889647 10029 raft_consensus.cc:359] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.889813 10029 raft_consensus.cc:385] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.889890 10029 raft_consensus.cc:740] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.890076 10029 consensus_queue.cc:260] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:48.890866 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet f6249fc5c6174e8e81b9d02d598d5ebd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6249fc5c6174e8e81b9d02d598d5ebd already in progress: creating tablet
I20251027 21:07:48.891063 10029 ts_tablet_manager.cc:1434] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:07:48.890640 9310 catalog_manager.cc:5649] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } }
I20251027 21:07:48.891351 10029 tablet_bootstrap.cc:492] T f5da8d7ecccc4c3492f9d0d30ffea423 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:48.892190 10043 ts_tablet_manager.cc:1950] T 783d845009d54d20b8e6626c3e73d2a7 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:48.892262 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 779b7f63b0fa4f9ab47cf5799b3e085d
I20251027 21:07:48.892978 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_943417713b3b456ab4ca691d2fb168fd" } modify_external_catalogs: true
I20251027 21:07:48.893151 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_943417713b3b456ab4ca691d2fb168fd" } modify_external_catalogs: true
I20251027 21:07:48.893528 10043 tablet_replica.cc:333] T 3218f18a797c40c9ad6ac64b2670cd19 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:48.893694 10043 raft_consensus.cc:2243] T 3218f18a797c40c9ad6ac64b2670cd19 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:48.893847 10043 raft_consensus.cc:2272] T 3218f18a797c40c9ad6ac64b2670cd19 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.894838 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 783d845009d54d20b8e6626c3e73d2a7 (table default.Table_53596f51be6a408a8987da9e37e4d779 [id=117c8f5066e149db8482c2601b040e17]) successfully deleted
W20251027 21:07:48.895061 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet f6249fc5c6174e8e81b9d02d598d5ebd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6249fc5c6174e8e81b9d02d598d5ebd already in progress: creating tablet
I20251027 21:07:48.895184 10029 tablet_bootstrap.cc:654] T f5da8d7ecccc4c3492f9d0d30ffea423 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:48.895291 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 9c4369a10bf846fb94c9b71a70f21600 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c4369a10bf846fb94c9b71a70f21600 already in progress: creating tablet
I20251027 21:07:48.895763 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet 779b7f63b0fa4f9ab47cf5799b3e085d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.896476 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 779b7f63b0fa4f9ab47cf5799b3e085d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.896838 10031 tablet_bootstrap.cc:492] T 40bd202c1b834c949f9a5cee23b3486d P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:07:48.897194 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 779b7f63b0fa4f9ab47cf5799b3e085d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 779b7f63b0fa4f9ab47cf5799b3e085d already in progress: creating tablet
I20251027 21:07:48.899497 10031 tablet_bootstrap.cc:654] T 40bd202c1b834c949f9a5cee23b3486d P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:48.899489 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 9c4369a10bf846fb94c9b71a70f21600 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c4369a10bf846fb94c9b71a70f21600 already in progress: creating tablet
W20251027 21:07:48.899767 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet cfce0e5c837743ea969374244163be3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfce0e5c837743ea969374244163be3e already in progress: creating tablet
W20251027 21:07:48.899966 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet d642da56a67449aa95932180bc5b2c5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d642da56a67449aa95932180bc5b2c5f already in progress: creating tablet
I20251027 21:07:48.902876 10043 ts_tablet_manager.cc:1916] T 3218f18a797c40c9ad6ac64b2670cd19 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:48.903606 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 779b7f63b0fa4f9ab47cf5799b3e085d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:48.905720 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 779b7f63b0fa4f9ab47cf5799b3e085d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 779b7f63b0fa4f9ab47cf5799b3e085d already in progress: creating tablet
I20251027 21:07:48.914779 10032 tablet_bootstrap.cc:492] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:48.915002 10032 ts_tablet_manager.cc:1403] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.000s
I20251027 21:07:48.916000 10032 raft_consensus.cc:359] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.916245 10032 raft_consensus.cc:385] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.916375 10032 raft_consensus.cc:740] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.916656 10032 consensus_queue.cc:260] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.917241 10032 ts_tablet_manager.cc:1434] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:48.917575 10032 tablet_bootstrap.cc:492] T 78a0a8b05f10477c923ab565839141b8 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:48.919955 10032 tablet_bootstrap.cc:654] T 78a0a8b05f10477c923ab565839141b8 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.925750 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet b1bdb2be5032415187c2826c14e75f07
I20251027 21:07:48.926473 9311 catalog_manager.cc:6719] ReplaceTablet: tablet b1bdb2be5032415187c2826c14e75f07 deleted and replaced by tablet f1aa5f7f76cb4a1ba15f016942f17067
I20251027 21:07:48.926843 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c93601bc03f1438496c9610e8c23b5a6" } modify_external_catalogs: true
I20251027 21:07:48.927047 10031 tablet_bootstrap.cc:492] T 40bd202c1b834c949f9a5cee23b3486d P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:48.927037 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c93601bc03f1438496c9610e8c23b5a6" } modify_external_catalogs: true
I20251027 21:07:48.927210 10031 ts_tablet_manager.cc:1403] T 40bd202c1b834c949f9a5cee23b3486d P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.006s
I20251027 21:07:48.927620 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet b1bdb2be5032415187c2826c14e75f07 with delete_type TABLET_DATA_DELETED (replaced by tablet f1aa5f7f76cb4a1ba15f016942f17067) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.927870 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet b1bdb2be5032415187c2826c14e75f07 with delete_type TABLET_DATA_DELETED (replaced by tablet f1aa5f7f76cb4a1ba15f016942f17067) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:48.927973 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet b1bdb2be5032415187c2826c14e75f07 with delete_type TABLET_DATA_DELETED (replaced by tablet f1aa5f7f76cb4a1ba15f016942f17067) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.928009 10031 raft_consensus.cc:359] T 40bd202c1b834c949f9a5cee23b3486d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.928174 10031 raft_consensus.cc:385] T 40bd202c1b834c949f9a5cee23b3486d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.928244 10031 raft_consensus.cc:740] T 40bd202c1b834c949f9a5cee23b3486d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.928205 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_84fa0625821f4deeb3422622babb5429"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:48.928521 10031 consensus_queue.cc:260] T 40bd202c1b834c949f9a5cee23b3486d P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:48.928691 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84fa0625821f4deeb3422622babb5429 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:48.928874 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet b1bdb2be5032415187c2826c14e75f07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1bdb2be5032415187c2826c14e75f07 already in progress: creating tablet
I20251027 21:07:48.929123 10031 ts_tablet_manager.cc:1434] T 40bd202c1b834c949f9a5cee23b3486d P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20251027 21:07:48.929315 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet b1bdb2be5032415187c2826c14e75f07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1bdb2be5032415187c2826c14e75f07 already in progress: creating tablet
I20251027 21:07:48.929430 10031 tablet_bootstrap.cc:492] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:48.929769 10029 tablet_bootstrap.cc:492] T f5da8d7ecccc4c3492f9d0d30ffea423 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:48.929912 10029 ts_tablet_manager.cc:1403] T f5da8d7ecccc4c3492f9d0d30ffea423 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.017s
I20251027 21:07:48.931069 10031 tablet_bootstrap.cc:654] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.931744 10029 raft_consensus.cc:359] T f5da8d7ecccc4c3492f9d0d30ffea423 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.931962 10029 raft_consensus.cc:385] T f5da8d7ecccc4c3492f9d0d30ffea423 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.932039 10029 raft_consensus.cc:740] T f5da8d7ecccc4c3492f9d0d30ffea423 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.932246 10029 consensus_queue.cc:260] T f5da8d7ecccc4c3492f9d0d30ffea423 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.937175 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 7d8825b80cec4ba68da077ca3b8d89db
I20251027 21:07:48.938122 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d8825b80cec4ba68da077ca3b8d89db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:48.938263 10043 ts_tablet_manager.cc:1929] T 3218f18a797c40c9ad6ac64b2670cd19 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.938386 10043 log.cc:1199] T 3218f18a797c40c9ad6ac64b2670cd19 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/3218f18a797c40c9ad6ac64b2670cd19
W20251027 21:07:48.938794 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 7d8825b80cec4ba68da077ca3b8d89db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d8825b80cec4ba68da077ca3b8d89db already in progress: creating tablet
I20251027 21:07:48.939337 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_741ffacfeaff46058d7e21e5bf7736fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.939884 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_741ffacfeaff46058d7e21e5bf7736fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.940690 10032 tablet_bootstrap.cc:492] T 78a0a8b05f10477c923ab565839141b8 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:48.940858 10032 ts_tablet_manager.cc:1403] T 78a0a8b05f10477c923ab565839141b8 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20251027 21:07:48.941604 10032 raft_consensus.cc:359] T 78a0a8b05f10477c923ab565839141b8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.941787 10032 raft_consensus.cc:385] T 78a0a8b05f10477c923ab565839141b8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.941840 10032 raft_consensus.cc:740] T 78a0a8b05f10477c923ab565839141b8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.942067 10032 consensus_queue.cc:260] T 78a0a8b05f10477c923ab565839141b8 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.942509 10032 ts_tablet_manager.cc:1434] T 78a0a8b05f10477c923ab565839141b8 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:48.942786 10032 tablet_bootstrap.cc:492] T d642da56a67449aa95932180bc5b2c5f P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:48.944242 10032 tablet_bootstrap.cc:654] T d642da56a67449aa95932180bc5b2c5f P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.946375 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet 12aa6e3af0694429b45bef5af3592377 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.948074 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d8825b80cec4ba68da077ca3b8d89db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:48.950268 10043 ts_tablet_manager.cc:1950] T 3218f18a797c40c9ad6ac64b2670cd19 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:48.951313 10043 tablet_replica.cc:333] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
W20251027 21:07:48.952454 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 7d8825b80cec4ba68da077ca3b8d89db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d8825b80cec4ba68da077ca3b8d89db already in progress: creating tablet
I20251027 21:07:48.953480 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 3218f18a797c40c9ad6ac64b2670cd19 (table default.Table_40644b4bc09f4693866503d3b5fb7df7 [id=09c88f2f898844b88b70f5ed9e837841]) successfully deleted
I20251027 21:07:48.953697 10029 ts_tablet_manager.cc:1434] T f5da8d7ecccc4c3492f9d0d30ffea423 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20251027 21:07:48.954113 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d8825b80cec4ba68da077ca3b8d89db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:48.954185 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet cfce0e5c837743ea969374244163be3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfce0e5c837743ea969374244163be3e already in progress: creating tablet
I20251027 21:07:48.953719 10043 raft_consensus.cc:2243] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:48.954811 10029 tablet_bootstrap.cc:492] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:48.956672 10029 tablet_bootstrap.cc:654] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.956852 10043 raft_consensus.cc:2272] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.956833 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0a8b47a441114631917879e9fe99d612"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.957374 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a8b47a441114631917879e9fe99d612 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.959244 10031 tablet_bootstrap.cc:492] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:48.959388 10031 ts_tablet_manager.cc:1403] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20251027 21:07:48.959714 10032 tablet_bootstrap.cc:492] T d642da56a67449aa95932180bc5b2c5f P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:48.959841 10032 ts_tablet_manager.cc:1403] T d642da56a67449aa95932180bc5b2c5f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.008s
I20251027 21:07:48.960259 10031 raft_consensus.cc:359] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.960453 10031 raft_consensus.cc:385] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.960539 10031 raft_consensus.cc:740] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.960772 10031 consensus_queue.cc:260] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.960906 10043 ts_tablet_manager.cc:1916] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:48.961262 10031 ts_tablet_manager.cc:1434] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:07:48.961619 10031 tablet_bootstrap.cc:492] T f5da8d7ecccc4c3492f9d0d30ffea423 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:48.961509 10032 raft_consensus.cc:359] T d642da56a67449aa95932180bc5b2c5f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.961691 10032 raft_consensus.cc:385] T d642da56a67449aa95932180bc5b2c5f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.961762 10032 raft_consensus.cc:740] T d642da56a67449aa95932180bc5b2c5f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.961971 10032 consensus_queue.cc:260] T d642da56a67449aa95932180bc5b2c5f P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.962478 10032 ts_tablet_manager.cc:1434] T d642da56a67449aa95932180bc5b2c5f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:07:48.962787 10032 tablet_bootstrap.cc:492] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:48.963169 10031 tablet_bootstrap.cc:654] T f5da8d7ecccc4c3492f9d0d30ffea423 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.964820 10032 tablet_bootstrap.cc:654] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.967221 10050 raft_consensus.cc:493] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:48.967370 10050 raft_consensus.cc:515] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.967962 10050 leader_election.cc:290] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:48.968529 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e5d52b7ec114be5ae68e57280c2eea8" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:48.969066 9859 leader_election.cc:343] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:48.969820 10031 tablet_bootstrap.cc:492] T f5da8d7ecccc4c3492f9d0d30ffea423 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:48.969995 10031 ts_tablet_manager.cc:1403] T f5da8d7ecccc4c3492f9d0d30ffea423 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20251027 21:07:48.970487 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e5d52b7ec114be5ae68e57280c2eea8" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:48.970800 10032 tablet_bootstrap.cc:492] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:48.970715 10031 raft_consensus.cc:359] T f5da8d7ecccc4c3492f9d0d30ffea423 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.970891 10031 raft_consensus.cc:385] T f5da8d7ecccc4c3492f9d0d30ffea423 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.970937 10032 ts_tablet_manager.cc:1403] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20251027 21:07:48.970963 10031 raft_consensus.cc:740] T f5da8d7ecccc4c3492f9d0d30ffea423 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.970952 10043 ts_tablet_manager.cc:1929] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.971074 10043 log.cc:1199] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/63662f175a1c450f8f67f37da971db37
I20251027 21:07:48.971174 10031 consensus_queue.cc:260] T f5da8d7ecccc4c3492f9d0d30ffea423 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.971423 10043 ts_tablet_manager.cc:1950] T 63662f175a1c450f8f67f37da971db37 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:48.971720 10031 ts_tablet_manager.cc:1434] T f5da8d7ecccc4c3492f9d0d30ffea423 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251027 21:07:48.971660 9860 leader_election.cc:343] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:48.971843 9860 leader_election.cc:304] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:48.971931 10031 tablet_bootstrap.cc:492] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:48.972162 10050 raft_consensus.cc:2749] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:48.973300 10031 tablet_bootstrap.cc:654] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.971678 10032 raft_consensus.cc:359] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.974913 10032 raft_consensus.cc:385] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.974982 10043 tablet_replica.cc:333] T d642da56a67449aa95932180bc5b2c5f P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:48.975078 10032 raft_consensus.cc:740] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.975193 10043 raft_consensus.cc:2243] T d642da56a67449aa95932180bc5b2c5f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:48.975244 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 63662f175a1c450f8f67f37da971db37 (table default.Table_d7caf208b9864991a0e50ced803eb60d [id=4142b5bb00004c83afe20e6982409849]) successfully deleted
I20251027 21:07:48.975373 10043 raft_consensus.cc:2272] T d642da56a67449aa95932180bc5b2c5f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:48.976137 10032 consensus_queue.cc:260] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.976583 10043 ts_tablet_manager.cc:1916] T d642da56a67449aa95932180bc5b2c5f P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:48.976740 10032 ts_tablet_manager.cc:1434] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251027 21:07:48.980104 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3922d39d16744b058b9a6116c7bc539f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:48.980773 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3922d39d16744b058b9a6116c7bc539f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:48.980880 10032 tablet_bootstrap.cc:492] T 84305bcf21b64d2c8c1df015a2e9615c P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:48.982509 10032 tablet_bootstrap.cc:654] T 84305bcf21b64d2c8c1df015a2e9615c P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:48.983336 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet f5da8d7ecccc4c3492f9d0d30ffea423 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:48.984472 10029 tablet_bootstrap.cc:492] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:48.984616 10029 ts_tablet_manager.cc:1403] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.007s
I20251027 21:07:48.986671 10029 raft_consensus.cc:359] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.986865 10029 raft_consensus.cc:385] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.986940 10029 raft_consensus.cc:740] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.987178 10029 consensus_queue.cc:260] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.987476 10043 ts_tablet_manager.cc:1929] T d642da56a67449aa95932180bc5b2c5f P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:48.987586 10043 log.cc:1199] T d642da56a67449aa95932180bc5b2c5f P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/d642da56a67449aa95932180bc5b2c5f
I20251027 21:07:48.987915 10043 ts_tablet_manager.cc:1950] T d642da56a67449aa95932180bc5b2c5f P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:48.988255 10049 raft_consensus.cc:493] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:48.988440 10029 ts_tablet_manager.cc:1434] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251027 21:07:48.988435 10049 raft_consensus.cc:515] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.989328 10031 tablet_bootstrap.cc:492] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:48.989460 10031 ts_tablet_manager.cc:1403] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.003s
I20251027 21:07:48.990265 10049 leader_election.cc:290] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:48.990191 10031 raft_consensus.cc:359] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:48.990459 10031 raft_consensus.cc:385] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.990578 10031 raft_consensus.cc:740] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.991389 10032 tablet_bootstrap.cc:492] T 84305bcf21b64d2c8c1df015a2e9615c P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:48.991509 10032 ts_tablet_manager.cc:1403] T 84305bcf21b64d2c8c1df015a2e9615c P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.002s
I20251027 21:07:48.992288 10032 raft_consensus.cc:359] T 84305bcf21b64d2c8c1df015a2e9615c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.992457 10032 raft_consensus.cc:385] T 84305bcf21b64d2c8c1df015a2e9615c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:48.992540 10032 raft_consensus.cc:740] T 84305bcf21b64d2c8c1df015a2e9615c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:48.996392 10031 consensus_queue.cc:260] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:48.998530 10032 consensus_queue.cc:260] T 84305bcf21b64d2c8c1df015a2e9615c P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:48.999037 10032 ts_tablet_manager.cc:1434] T 84305bcf21b64d2c8c1df015a2e9615c P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251027 21:07:48.999301 10032 tablet_bootstrap.cc:492] T 12aa6e3af0694429b45bef5af3592377 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:49.000933 10032 tablet_bootstrap.cc:654] T 12aa6e3af0694429b45bef5af3592377 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.001253 9312 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 21b785277d604543bfa7816ed96d6181 (table default.Table_f1908c85b5144e90a366ca29ba5d1b9b [id=e3de2e5ea6484be7af68eb8601778b85]). Expected version 2 got 0
I20251027 21:07:49.001729 9304 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 21b785277d604543bfa7816ed96d6181 (table default.Table_f1908c85b5144e90a366ca29ba5d1b9b [id=e3de2e5ea6484be7af68eb8601778b85]). Expected version 2 got 0
I20251027 21:07:49.003880 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 995284883d834bbb9ef583b736ca8402 (DEFAULT_TABLE table=default.Table_6abfd304277c48bfb3018f4e211e13b3 [id=186c9cce53d74040b9799ea386fdffd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.004302 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 995284883d834bbb9ef583b736ca8402. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.005666 10043 tablet_replica.cc:333] T 5c87a5450d354506852c4b20bf0541d2 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:49.006065 10029 tablet_bootstrap.cc:492] T ba8b125474bc42a5946e8f2dd718f7f4 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:49.010898 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 995284883d834bbb9ef583b736ca8402 (DEFAULT_TABLE table=default.Table_6abfd304277c48bfb3018f4e211e13b3 [id=186c9cce53d74040b9799ea386fdffd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.011461 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 995284883d834bbb9ef583b736ca8402. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.014475 10032 tablet_bootstrap.cc:492] T 12aa6e3af0694429b45bef5af3592377 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:49.014636 10032 ts_tablet_manager.cc:1403] T 12aa6e3af0694429b45bef5af3592377 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.003s
I20251027 21:07:49.015239 10032 raft_consensus.cc:359] T 12aa6e3af0694429b45bef5af3592377 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.015426 10032 raft_consensus.cc:385] T 12aa6e3af0694429b45bef5af3592377 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.015522 10032 raft_consensus.cc:740] T 12aa6e3af0694429b45bef5af3592377 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.015755 10032 consensus_queue.cc:260] T 12aa6e3af0694429b45bef5af3592377 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.016235 10032 ts_tablet_manager.cc:1434] T 12aa6e3af0694429b45bef5af3592377 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:07:49.016507 10032 tablet_bootstrap.cc:492] T 69b1fb42428748239d8939d98428d8fe P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:49.016453 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet d642da56a67449aa95932180bc5b2c5f (table default.Table_178e5d554faa4e119ac8f52da2d284d7 [id=05edeb649c544c2f899ed42a87cef2c2]) successfully deleted
I20251027 21:07:49.016498 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_12986d58c67145d69cc562041489fbc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.017010 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12986d58c67145d69cc562041489fbc4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.018272 10032 tablet_bootstrap.cc:654] T 69b1fb42428748239d8939d98428d8fe P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.021973 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 019a892bfdd54e62bc3560a341d5c2da
I20251027 21:07:49.022455 9305 catalog_manager.cc:6719] ReplaceTablet: tablet 019a892bfdd54e62bc3560a341d5c2da deleted and replaced by tablet ccaf5c2f8e544842880ba8ba27300ae6
I20251027 21:07:49.024150 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9cac798aeef2436f9a044ec54e72daef" } new_table_name: "default.Table_e65d5d66ce8747a9a1a2abfa04cd2855" modify_external_catalogs: true
I20251027 21:07:49.024474 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 019a892bfdd54e62bc3560a341d5c2da with delete_type TABLET_DATA_DELETED (replaced by tablet ccaf5c2f8e544842880ba8ba27300ae6) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:49.025211 9297 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 970fbf01a4c3406dafe1ea4cd005ea23 (table default.Table_530197464d684950839a65bdf56549b8 [id=741c9ce04c9d47d9a016bce1f4403ec7], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 970fbf01a4c3406dafe1ea4cd005ea23 consensus state has no leader
I20251027 21:07:49.025769 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2d758ba690f8478ba7f983e8fd770219"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:49.026269 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_673503358b54481c8877034c26349796"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:49.027038 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 1776590d8387494da5dfbd08db17f4b1
W20251027 21:07:49.027050 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d758ba690f8478ba7f983e8fd770219 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.027506 9306 catalog_manager.cc:6719] ReplaceTablet: tablet 1776590d8387494da5dfbd08db17f4b1 deleted and replaced by tablet 7aa8c538b6c849a28a28ffe706306891
I20251027 21:07:49.028394 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_78302ea6dc094b199cfda33dd516a2ec" } new_table_name: "default.Table_3e0f707f8e2844bfaf937150b747a935" modify_external_catalogs: true
W20251027 21:07:49.028534 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_673503358b54481c8877034c26349796 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:49.029306 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 019a892bfdd54e62bc3560a341d5c2da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 019a892bfdd54e62bc3560a341d5c2da already in progress: creating tablet
I20251027 21:07:49.029862 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 1776590d8387494da5dfbd08db17f4b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 7aa8c538b6c849a28a28ffe706306891) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.030063 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1e51d9a2db704a699f7bb8a2f6b3a4f3" } modify_external_catalogs: true
I20251027 21:07:49.030346 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1e51d9a2db704a699f7bb8a2f6b3a4f3" } modify_external_catalogs: true
W20251027 21:07:49.031021 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 1776590d8387494da5dfbd08db17f4b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1776590d8387494da5dfbd08db17f4b1 already in progress: creating tablet
I20251027 21:07:49.034080 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5942b242afb64f95a82e2a0bafe674f0" } new_table_name: "default.Table_4f6664b7160e4549a50589454ff10313" modify_external_catalogs: true
I20251027 21:07:49.041926 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 601c74459d3a4a41b812fdcdf8b0a62c
W20251027 21:07:49.042863 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ac7111d13b6460a9f4c77e94e09d417 consensus state has no leader
I20251027 21:07:49.043349 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 44783760defd433a972c9cd83933249c
I20251027 21:07:49.043813 9307 catalog_manager.cc:6719] ReplaceTablet: tablet 44783760defd433a972c9cd83933249c deleted and replaced by tablet ffe1182e8f4e4a44b9a12e0a5192989f
I20251027 21:07:49.045771 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 2d659e4d73ea4ff3975a4b58a794090a
W20251027 21:07:49.046085 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1aa5f7f76cb4a1ba15f016942f17067 has no consensus state
I20251027 21:07:49.046185 9303 catalog_manager.cc:6719] ReplaceTablet: tablet 2d659e4d73ea4ff3975a4b58a794090a deleted and replaced by tablet 6bf9c2a78aac42528d67846dc58adc40
I20251027 21:07:49.048043 10043 raft_consensus.cc:2243] T 5c87a5450d354506852c4b20bf0541d2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:49.049348 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7a662e3d3c4b4f8cb68a606bdf279049" } new_table_name: "default.Table_f0792045d49041578f5ff6459c99f77e" modify_external_catalogs: true
I20251027 21:07:49.050904 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 995284883d834bbb9ef583b736ca8402 (DEFAULT_TABLE table=default.Table_6abfd304277c48bfb3018f4e211e13b3 [id=186c9cce53d74040b9799ea386fdffd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.051322 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 995284883d834bbb9ef583b736ca8402. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.052498 10043 raft_consensus.cc:2272] T 5c87a5450d354506852c4b20bf0541d2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:49.053169 10043 ts_tablet_manager.cc:1916] T 5c87a5450d354506852c4b20bf0541d2 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:49.054039 9306 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 0ac7111d13b6460a9f4c77e94e09d417 (table default.Table_e65d5d66ce8747a9a1a2abfa04cd2855 [id=e2b68fb34c3b48e8952ef6d4201fd93f]). Expected version 1 got 0
I20251027 21:07:49.056169 9886 tablet_service.cc:1552] Processing DeleteTablet for tablet 44783760defd433a972c9cd83933249c with delete_type TABLET_DATA_DELETED (replaced by tablet ffe1182e8f4e4a44b9a12e0a5192989f) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.056386 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 601c74459d3a4a41b812fdcdf8b0a62c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.056465 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d659e4d73ea4ff3975a4b58a794090a with delete_type TABLET_DATA_DELETED (replaced by tablet 6bf9c2a78aac42528d67846dc58adc40) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.056867 9892 tablet_service.cc:1505] Processing CreateTablet for tablet d94c695320f24a7f9776c493e462ed24 (DEFAULT_TABLE table=default.Table_3da5a86d9f104d1ba39726b5a6dcec8a [id=da40d7d63ce348dd8a275cdd54f8f018]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.057258 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d94c695320f24a7f9776c493e462ed24. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:49.057281 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 44783760defd433a972c9cd83933249c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44783760defd433a972c9cd83933249c already in progress: creating tablet
W20251027 21:07:49.057523 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 601c74459d3a4a41b812fdcdf8b0a62c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 601c74459d3a4a41b812fdcdf8b0a62c already in progress: creating tablet
I20251027 21:07:49.057802 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e0dee9209ca14dd98065b0384855d953"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.057740 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 2d659e4d73ea4ff3975a4b58a794090a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d659e4d73ea4ff3975a4b58a794090a already in progress: creating tablet
I20251027 21:07:49.060685 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9ddfcdda61854c59a1295af022b0ddc0" } new_table_name: "default.Table_16dd2e0479ea434dbbd0b900b340fc9f" modify_external_catalogs: true
I20251027 21:07:49.060736 10031 ts_tablet_manager.cc:1434] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
W20251027 21:07:49.062094 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0dee9209ca14dd98065b0384855d953 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.062378 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 583fba7c750e4ac7b768da447002c211 (DEFAULT_TABLE table=default.Table_2fd9ca76e8a0406cbfd32f91038ccc0e [id=14daa3e199fe4fa3ba2de06dd6be4330]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.062783 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 583fba7c750e4ac7b768da447002c211. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.063468 9308 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 0ac7111d13b6460a9f4c77e94e09d417 (table default.Table_e65d5d66ce8747a9a1a2abfa04cd2855 [id=e2b68fb34c3b48e8952ef6d4201fd93f]). Expected version 1 got 0
I20251027 21:07:49.063771 9308 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 9cec6294051b41868c9c7c7fadafa138 (table default.Table_5d8e4122af254070a2df7016e521a937 [id=12b1bbc770194455ab7292641980e8e1]). Expected version 2 got 0
I20251027 21:07:49.063962 9308 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 21b785277d604543bfa7816ed96d6181 (table default.Table_f1908c85b5144e90a366ca29ba5d1b9b [id=e3de2e5ea6484be7af68eb8601778b85]). Expected version 2 got 0
I20251027 21:07:49.065961 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 78a0a8b05f10477c923ab565839141b8 with delete_type TABLET_DATA_DELETED (replaced by tablet 1698ae8a37ad4560836b3b1daca1cf5e) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.066449 10041 tablet_replica.cc:333] T 78a0a8b05f10477c923ab565839141b8 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:49.066637 10041 raft_consensus.cc:2243] T 78a0a8b05f10477c923ab565839141b8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:49.066815 10041 raft_consensus.cc:2272] T 78a0a8b05f10477c923ab565839141b8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:49.067098 9896 tablet_service.cc:1552] Processing DeleteTablet for tablet 0029b70ed3ff4db9a816a6190d36bf88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.067167 9886 tablet_service.cc:1552] Processing DeleteTablet for tablet f5da8d7ecccc4c3492f9d0d30ffea423 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.067348 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet d642da56a67449aa95932180bc5b2c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.067391 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e5d52b7ec114be5ae68e57280c2eea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.067943 10041 ts_tablet_manager.cc:1916] T 78a0a8b05f10477c923ab565839141b8 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:49.068537 9899 tablet_service.cc:1505] Processing CreateTablet for tablet b87e809ff897417c88e5edc19e2d591e (DEFAULT_TABLE table=default.Table_d2bf88aca4cc44df99d13783ea2722f8 [id=7de1278fe67640ff8540392837b2eeae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.068655 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet 019a892bfdd54e62bc3560a341d5c2da with delete_type TABLET_DATA_DELETED (replaced by tablet ccaf5c2f8e544842880ba8ba27300ae6) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:49.068890 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b87e809ff897417c88e5edc19e2d591e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.069556 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet 601c74459d3a4a41b812fdcdf8b0a62c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:49.070361 9608 tablet_service.cc:1552] Processing DeleteTablet for tablet 44783760defd433a972c9cd83933249c with delete_type TABLET_DATA_DELETED (replaced by tablet ffe1182e8f4e4a44b9a12e0a5192989f) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:49.071116 9608 tablet_service.cc:1505] Processing CreateTablet for tablet b87e809ff897417c88e5edc19e2d591e (DEFAULT_TABLE table=default.Table_d2bf88aca4cc44df99d13783ea2722f8 [id=7de1278fe67640ff8540392837b2eeae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.071847 10089 raft_consensus.cc:493] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:49.071985 10089 raft_consensus.cc:515] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.072747 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d659e4d73ea4ff3975a4b58a794090a with delete_type TABLET_DATA_DELETED (replaced by tablet 6bf9c2a78aac42528d67846dc58adc40) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:49.071857 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet 1776590d8387494da5dfbd08db17f4b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 7aa8c538b6c849a28a28ffe706306891) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:49.073022 10089 leader_election.cc:290] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:49.073196 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b87e809ff897417c88e5edc19e2d591e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.073375 10031 tablet_bootstrap.cc:492] T 0ac7111d13b6460a9f4c77e94e09d417 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:49.074225 9745 tablet_service.cc:1552] Processing DeleteTablet for tablet 601c74459d3a4a41b812fdcdf8b0a62c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.074458 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 1776590d8387494da5dfbd08db17f4b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 7aa8c538b6c849a28a28ffe706306891) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.074827 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 019a892bfdd54e62bc3560a341d5c2da with delete_type TABLET_DATA_DELETED (replaced by tablet ccaf5c2f8e544842880ba8ba27300ae6) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.077261 10029 tablet_bootstrap.cc:654] T ba8b125474bc42a5946e8f2dd718f7f4 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.077378 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b0d840604cdc476b9a84b5c4cc69e369"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.077893 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0d840604cdc476b9a84b5c4cc69e369 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.078255 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 44783760defd433a972c9cd83933249c with delete_type TABLET_DATA_DELETED (replaced by tablet ffe1182e8f4e4a44b9a12e0a5192989f) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.078429 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d659e4d73ea4ff3975a4b58a794090a with delete_type TABLET_DATA_DELETED (replaced by tablet 6bf9c2a78aac42528d67846dc58adc40) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.079137 9610 tablet_service.cc:1505] Processing CreateTablet for tablet d94c695320f24a7f9776c493e462ed24 (DEFAULT_TABLE table=default.Table_3da5a86d9f104d1ba39726b5a6dcec8a [id=da40d7d63ce348dd8a275cdd54f8f018]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.079396 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ac7111d13b6460a9f4c77e94e09d417" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:49.079522 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d94c695320f24a7f9776c493e462ed24. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.079614 9925 raft_consensus.cc:2468] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:49.081183 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ac7111d13b6460a9f4c77e94e09d417" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:49.081992 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 65fbcaa5447c48818fabfccac22c4048
I20251027 21:07:49.082403 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet 12aa6e3af0694429b45bef5af3592377 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.082778 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet 84305bcf21b64d2c8c1df015a2e9615c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.083575 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet 65fbcaa5447c48818fabfccac22c4048 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.085836 9740 tablet_service.cc:1505] Processing CreateTablet for tablet d94c695320f24a7f9776c493e462ed24 (DEFAULT_TABLE table=default.Table_3da5a86d9f104d1ba39726b5a6dcec8a [id=da40d7d63ce348dd8a275cdd54f8f018]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.087441 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 6db0f31c12d14e66a4ec7fc722d62c3d (DEFAULT_TABLE table=default.Table_42431224890f43859abe3dedfae4db51 [id=a6784a116865476b85fe6ff9c32bb9db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.087918 9745 tablet_service.cc:1505] Processing CreateTablet for tablet b87e809ff897417c88e5edc19e2d591e (DEFAULT_TABLE table=default.Table_d2bf88aca4cc44df99d13783ea2722f8 [id=7de1278fe67640ff8540392837b2eeae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.088395 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 583fba7c750e4ac7b768da447002c211 (DEFAULT_TABLE table=default.Table_2fd9ca76e8a0406cbfd32f91038ccc0e [id=14daa3e199fe4fa3ba2de06dd6be4330]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:49.089165 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 601c74459d3a4a41b812fdcdf8b0a62c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 601c74459d3a4a41b812fdcdf8b0a62c already in progress: creating tablet
W20251027 21:07:49.089387 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 1776590d8387494da5dfbd08db17f4b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1776590d8387494da5dfbd08db17f4b1 already in progress: creating tablet
W20251027 21:07:49.089594 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 019a892bfdd54e62bc3560a341d5c2da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 019a892bfdd54e62bc3560a341d5c2da already in progress: creating tablet
I20251027 21:07:49.090044 9743 tablet_service.cc:1552] Processing DeleteTablet for tablet 65fbcaa5447c48818fabfccac22c4048 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:49.091696 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 44783760defd433a972c9cd83933249c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44783760defd433a972c9cd83933249c already in progress: creating tablet
W20251027 21:07:49.091902 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 2d659e4d73ea4ff3975a4b58a794090a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d659e4d73ea4ff3975a4b58a794090a already in progress: creating tablet
W20251027 21:07:49.092077 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 65fbcaa5447c48818fabfccac22c4048 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65fbcaa5447c48818fabfccac22c4048 already in progress: creating tablet
I20251027 21:07:49.092388 10031 tablet_bootstrap.cc:654] T 0ac7111d13b6460a9f4c77e94e09d417 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.093063 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d94c695320f24a7f9776c493e462ed24. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.093497 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 6db0f31c12d14e66a4ec7fc722d62c3d (DEFAULT_TABLE table=default.Table_42431224890f43859abe3dedfae4db51 [id=a6784a116865476b85fe6ff9c32bb9db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.093861 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6db0f31c12d14e66a4ec7fc722d62c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.093935 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6db0f31c12d14e66a4ec7fc722d62c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.094429 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b87e809ff897417c88e5edc19e2d591e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.099709 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_44e3f093c8d7476a98074b07828da3ab" } modify_external_catalogs: true
I20251027 21:07:49.099917 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_44e3f093c8d7476a98074b07828da3ab" } modify_external_catalogs: true
I20251027 21:07:49.101468 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 583fba7c750e4ac7b768da447002c211. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.104243 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_fc9bf567c9ff4ddeaccaec2c6fddca99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.104846 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc9bf567c9ff4ddeaccaec2c6fddca99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.110321 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21b785277d604543bfa7816ed96d6181" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:49.110508 9925 raft_consensus.cc:2468] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:07:49.113878 10041 ts_tablet_manager.cc:1929] T 78a0a8b05f10477c923ab565839141b8 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:49.113987 10041 log.cc:1199] T 78a0a8b05f10477c923ab565839141b8 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/78a0a8b05f10477c923ab565839141b8
I20251027 21:07:49.115404 10041 ts_tablet_manager.cc:1950] T 78a0a8b05f10477c923ab565839141b8 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:49.116550 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 78a0a8b05f10477c923ab565839141b8 (table default.Table_108afd04fe5d45c195ba81996dee61d9 [id=02839636442f4955b245a06a594634c1]) successfully deleted
I20251027 21:07:49.116937 10041 tablet_replica.cc:333] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:49.117192 10041 raft_consensus.cc:2243] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:49.117367 10041 raft_consensus.cc:2272] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:49.118377 10041 ts_tablet_manager.cc:1916] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:49.122234 9711 leader_election.cc:304] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:07:49.122701 10049 raft_consensus.cc:2804] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:49.122838 10049 raft_consensus.cc:493] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:49.122926 10049 raft_consensus.cc:3060] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:49.125747 9309 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet aa32b163b43542acb445f98be9c736f7
I20251027 21:07:49.125680 10049 raft_consensus.cc:515] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:49.126281 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cbada3946dab4739a8369032ff3b8a48 consensus state has no leader
I20251027 21:07:49.126505 10049 leader_election.cc:290] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:49.127401 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21b785277d604543bfa7816ed96d6181" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:07:49.127667 9925 raft_consensus.cc:3060] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:49.129479 10041 ts_tablet_manager.cc:1929] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:49.129693 10041 log.cc:1199] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/0029b70ed3ff4db9a816a6190d36bf88
I20251027 21:07:49.130055 10041 ts_tablet_manager.cc:1950] T 0029b70ed3ff4db9a816a6190d36bf88 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:49.135246 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 0029b70ed3ff4db9a816a6190d36bf88 (table default.Table_450e3d88ee8f4a328418b279a354b9e2 [id=0f311c7d59a4471fbad8c2a0c59430f0]) successfully deleted
I20251027 21:07:49.136247 10049 raft_consensus.cc:493] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:49.136384 10049 raft_consensus.cc:515] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.137100 10049 leader_election.cc:290] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:07:49.138556 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1e5f5b57f5d948f3b2352451ccaacb09 consensus state has no leader
W20251027 21:07:49.139060 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 77797eea3e9c4f8099dc9d58f116200b consensus state has no leader
I20251027 21:07:49.139994 9311 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet ffe1182e8f4e4a44b9a12e0a5192989f
I20251027 21:07:49.146023 10031 tablet_bootstrap.cc:492] T 0ac7111d13b6460a9f4c77e94e09d417 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:49.146214 10031 ts_tablet_manager.cc:1403] T 0ac7111d13b6460a9f4c77e94e09d417 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.076s user 0.009s sys 0.000s
I20251027 21:07:49.147661 9308 master_service.cc:954] ReplaceTablet: received request to replace tablet 12aa6e3af0694429b45bef5af3592377 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:49.147657 10031 raft_consensus.cc:359] T 0ac7111d13b6460a9f4c77e94e09d417 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.147830 10031 raft_consensus.cc:385] T 0ac7111d13b6460a9f4c77e94e09d417 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.147894 10031 raft_consensus.cc:740] T 0ac7111d13b6460a9f4c77e94e09d417 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.148113 10031 consensus_queue.cc:260] T 0ac7111d13b6460a9f4c77e94e09d417 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.151623 10031 ts_tablet_manager.cc:1434] T 0ac7111d13b6460a9f4c77e94e09d417 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251027 21:07:49.154486 10031 tablet_bootstrap.cc:492] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:49.154973 9568 leader_election.cc:304] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:07:49.155203 10041 tablet_replica.cc:333] T f5da8d7ecccc4c3492f9d0d30ffea423 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:49.155376 10041 raft_consensus.cc:2243] T f5da8d7ecccc4c3492f9d0d30ffea423 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251027 21:07:49.155342 9568 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 75187.5us
I20251027 21:07:49.155661 10041 raft_consensus.cc:2272] T f5da8d7ecccc4c3492f9d0d30ffea423 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:49.155728 10039 raft_consensus.cc:2804] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:49.155853 10039 raft_consensus.cc:493] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:49.155964 10039 raft_consensus.cc:3060] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:07:49.156131 9566 leader_election.cc:343] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:49.156993 10032 tablet_bootstrap.cc:492] T 69b1fb42428748239d8939d98428d8fe P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:49.157799 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b0d840604cdc476b9a84b5c4cc69e369" } modify_external_catalogs: true
I20251027 21:07:49.157975 10041 ts_tablet_manager.cc:1916] T f5da8d7ecccc4c3492f9d0d30ffea423 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:49.158006 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b0d840604cdc476b9a84b5c4cc69e369" } modify_external_catalogs: true
I20251027 21:07:49.159173 10032 ts_tablet_manager.cc:1403] T 69b1fb42428748239d8939d98428d8fe P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.143s user 0.017s sys 0.012s
I20251027 21:07:49.159930 10032 raft_consensus.cc:359] T 69b1fb42428748239d8939d98428d8fe P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.160104 10032 raft_consensus.cc:385] T 69b1fb42428748239d8939d98428d8fe P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.160219 10032 raft_consensus.cc:740] T 69b1fb42428748239d8939d98428d8fe P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.160441 10032 consensus_queue.cc:260] T 69b1fb42428748239d8939d98428d8fe P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.163904 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 69b1fb42428748239d8939d98428d8fe with delete_type TABLET_DATA_DELETED (replaced by tablet 49b3d23798b740008c98cd0754ae33f4) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.164062 9925 raft_consensus.cc:2468] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:07:49.164327 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cec6294051b41868c9c7c7fadafa138" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:49.164557 9924 raft_consensus.cc:2468] T 9cec6294051b41868c9c7c7fadafa138 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:07:49.164534 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_790c0f90be104b8e8abf3491796d54bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.165159 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_790c0f90be104b8e8abf3491796d54bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.165191 9711 leader_election.cc:304] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:07:49.165505 9711 leader_election.cc:304] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:07:49.166608 10049 raft_consensus.cc:2804] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:49.166949 10032 ts_tablet_manager.cc:1434] T 69b1fb42428748239d8939d98428d8fe P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251027 21:07:49.167834 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 583fba7c750e4ac7b768da447002c211 (DEFAULT_TABLE table=default.Table_2fd9ca76e8a0406cbfd32f91038ccc0e [id=14daa3e199fe4fa3ba2de06dd6be4330]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.168239 9308 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 15e1ac58467a4571abda9010effc714f
I20251027 21:07:49.168747 10031 tablet_bootstrap.cc:654] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.169245 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet 65fbcaa5447c48818fabfccac22c4048 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:49.170107 10032 tablet_bootstrap.cc:492] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:49.172297 10089 raft_consensus.cc:493] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:49.173604 10032 tablet_bootstrap.cc:654] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.175339 10142 raft_consensus.cc:2804] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:49.175448 10142 raft_consensus.cc:493] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:49.175591 10142 raft_consensus.cc:3060] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:49.176033 10039 raft_consensus.cc:515] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.176805 10039 leader_election.cc:290] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:49.177371 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 6db0f31c12d14e66a4ec7fc722d62c3d (DEFAULT_TABLE table=default.Table_42431224890f43859abe3dedfae4db51 [id=a6784a116865476b85fe6ff9c32bb9db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.179514 10142 raft_consensus.cc:515] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.179903 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ac7111d13b6460a9f4c77e94e09d417" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:07:49.180068 9924 raft_consensus.cc:3060] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:49.180191 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_ba1d973c05e14c70944d76ff1ecbc1bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.180740 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba1d973c05e14c70944d76ff1ecbc1bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.184019 9924 raft_consensus.cc:2468] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:49.186120 10089 raft_consensus.cc:515] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.187376 10032 tablet_bootstrap.cc:492] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:49.187552 10032 ts_tablet_manager.cc:1403] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.006s
I20251027 21:07:49.188494 10032 raft_consensus.cc:359] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.188730 10032 raft_consensus.cc:385] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.188805 10032 raft_consensus.cc:740] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.188892 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 583fba7c750e4ac7b768da447002c211. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.189085 10032 consensus_queue.cc:260] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.189559 10032 ts_tablet_manager.cc:1434] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:07:49.189821 10032 tablet_bootstrap.cc:492] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:49.191850 10032 tablet_bootstrap.cc:654] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.192799 10142 leader_election.cc:290] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:49.192971 10142 raft_consensus.cc:493] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:49.193158 10142 raft_consensus.cc:515] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.194557 10142 leader_election.cc:290] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:49.195914 9568 leader_election.cc:304] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:07:49.196576 10031 tablet_bootstrap.cc:492] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:49.196696 10031 ts_tablet_manager.cc:1403] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.010s
I20251027 21:07:49.197948 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6db0f31c12d14e66a4ec7fc722d62c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.201169 10031 raft_consensus.cc:359] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.201341 10031 raft_consensus.cc:385] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.201450 10031 raft_consensus.cc:740] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.201581 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9c027ff744824da9b9cac5574017e7fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:49.201680 10031 consensus_queue.cc:260] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.202245 10031 ts_tablet_manager.cc:1434] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
W20251027 21:07:49.202279 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c027ff744824da9b9cac5574017e7fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.203434 10031 tablet_bootstrap.cc:492] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:49.206369 10031 tablet_bootstrap.cc:654] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:49.199541 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 105411us
I20251027 21:07:49.208276 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12aa6e3af0694429b45bef5af3592377" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:49.208405 10032 tablet_bootstrap.cc:492] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:49.208492 9924 raft_consensus.cc:2468] T 12aa6e3af0694429b45bef5af3592377 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:49.208565 10032 ts_tablet_manager.cc:1403] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20251027 21:07:49.208978 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_730efea4f3644d55a230c2d1b57ea69e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.209496 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_730efea4f3644d55a230c2d1b57ea69e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.209494 10032 raft_consensus.cc:359] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.209694 10032 raft_consensus.cc:385] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.209787 10032 raft_consensus.cc:740] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.210047 10032 consensus_queue.cc:260] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.214257 10032 ts_tablet_manager.cc:1434] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251027 21:07:49.214524 10032 tablet_bootstrap.cc:492] T 63662f175a1c450f8f67f37da971db37 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:49.216192 10032 tablet_bootstrap.cc:654] T 63662f175a1c450f8f67f37da971db37 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.217370 10031 tablet_bootstrap.cc:492] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:49.218233 10089 leader_election.cc:290] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:49.218618 10031 ts_tablet_manager.cc:1403] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.003s
I20251027 21:07:49.219305 10031 raft_consensus.cc:359] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.219482 10031 raft_consensus.cc:385] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.219542 10031 raft_consensus.cc:740] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.219784 10031 consensus_queue.cc:260] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.221005 10031 ts_tablet_manager.cc:1434] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:49.221931 10031 tablet_bootstrap.cc:492] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:49.222220 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12aa6e3af0694429b45bef5af3592377" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:49.222257 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ac7111d13b6460a9f4c77e94e09d417" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:49.222507 9774 raft_consensus.cc:2468] T 12aa6e3af0694429b45bef5af3592377 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:49.222529 9775 raft_consensus.cc:3060] T 0ac7111d13b6460a9f4c77e94e09d417 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:49.225580 10031 tablet_bootstrap.cc:654] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.227614 9775 raft_consensus.cc:2468] T 0ac7111d13b6460a9f4c77e94e09d417 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:49.227686 10032 tablet_bootstrap.cc:492] T 63662f175a1c450f8f67f37da971db37 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:49.227865 10032 ts_tablet_manager.cc:1403] T 63662f175a1c450f8f67f37da971db37 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.003s
I20251027 21:07:49.228633 10032 raft_consensus.cc:359] T 63662f175a1c450f8f67f37da971db37 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.228796 10032 raft_consensus.cc:385] T 63662f175a1c450f8f67f37da971db37 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.228854 10032 raft_consensus.cc:740] T 63662f175a1c450f8f67f37da971db37 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.229075 10032 consensus_queue.cc:260] T 63662f175a1c450f8f67f37da971db37 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.232352 10031 tablet_bootstrap.cc:492] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:49.232491 10031 ts_tablet_manager.cc:1403] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20251027 21:07:49.233172 10031 raft_consensus.cc:359] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.233422 10031 raft_consensus.cc:385] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.233521 10031 raft_consensus.cc:740] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.233635 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c1a6a7616f24f69b83ec8a282746e92 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d659e4d73ea4ff3975a4b58a794090a) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.233820 10031 consensus_queue.cc:260] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.234828 9568 leader_election.cc:304] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:07:49.235160 10032 ts_tablet_manager.cc:1434] T 63662f175a1c450f8f67f37da971db37 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251027 21:07:49.236017 10031 ts_tablet_manager.cc:1434] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:07:49.236863 10031 tablet_bootstrap.cc:492] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:49.236948 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_1ecaac632a024248b04e7c40e2c01d2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.237517 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ecaac632a024248b04e7c40e2c01d2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.237716 10032 tablet_bootstrap.cc:492] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:49.237926 10039 raft_consensus.cc:2804] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:49.238253 10089 raft_consensus.cc:2804] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:49.238343 10089 raft_consensus.cc:493] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:49.238454 10089 raft_consensus.cc:3060] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:49.239614 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cec6294051b41868c9c7c7fadafa138" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:49.239842 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 63662f175a1c450f8f67f37da971db37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:49.240540 9710 leader_election.cc:343] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:49.239902 10032 tablet_bootstrap.cc:654] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.240842 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cec6294051b41868c9c7c7fadafa138" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:07:49.241385 10089 raft_consensus.cc:515] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.241634 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21b785277d604543bfa7816ed96d6181" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
W20251027 21:07:49.242393 9710 leader_election.cc:343] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:49.243178 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12aa6e3af0694429b45bef5af3592377" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:49.243350 9775 raft_consensus.cc:3060] T 12aa6e3af0694429b45bef5af3592377 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:49.246390 9775 raft_consensus.cc:2468] T 12aa6e3af0694429b45bef5af3592377 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:49.247081 10043 ts_tablet_manager.cc:1929] T 5c87a5450d354506852c4b20bf0541d2 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:49.248077 10031 tablet_bootstrap.cc:654] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:49.248004 9710 leader_election.cc:343] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:49.248426 10032 tablet_bootstrap.cc:492] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:49.248548 10032 ts_tablet_manager.cc:1403] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20251027 21:07:49.249275 10032 raft_consensus.cc:359] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.249482 10032 raft_consensus.cc:385] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.249594 10032 raft_consensus.cc:740] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.249869 10032 consensus_queue.cc:260] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.253588 10032 ts_tablet_manager.cc:1434] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20251027 21:07:49.253681 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21b785277d604543bfa7816ed96d6181" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:49.256709 10032 tablet_bootstrap.cc:492] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:49.256871 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d5ad2155dc3543579943b617c2fdb9fe" } new_table_name: "default.Table_68f478946b1543a68842ffad9e14864f" modify_external_catalogs: true
I20251027 21:07:49.257607 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_97b01c1840c543838263ca4222695f54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.257887 9710 leader_election.cc:343] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:07:49.258124 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97b01c1840c543838263ca4222695f54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.258261 10032 tablet_bootstrap.cc:654] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.258509 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6abfd304277c48bfb3018f4e211e13b3" } new_table_name: "default.Table_5675f5f4f1424e2581d644b9be1878ce" modify_external_catalogs: true
I20251027 21:07:49.259351 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_03046ffdd6ca47e8aa4cde13574dc119" } modify_external_catalogs: true
I20251027 21:07:49.259511 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_03046ffdd6ca47e8aa4cde13574dc119" } modify_external_catalogs: true
I20251027 21:07:49.274334 10032 tablet_bootstrap.cc:492] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:49.274531 10032 ts_tablet_manager.cc:1403] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.002s
I20251027 21:07:49.275547 10032 raft_consensus.cc:359] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.275786 10032 raft_consensus.cc:385] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.275861 10032 raft_consensus.cc:740] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.276166 10032 consensus_queue.cc:260] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.280334 10032 ts_tablet_manager.cc:1434] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251027 21:07:49.281783 10031 tablet_bootstrap.cc:492] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:49.281947 10031 ts_tablet_manager.cc:1403] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.046s user 0.001s sys 0.007s
I20251027 21:07:49.282792 10031 raft_consensus.cc:359] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.282976 10031 raft_consensus.cc:385] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251027 21:07:49.282980 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0968c2f25b64f6685f38a999ee9d60e has no consensus state
I20251027 21:07:49.283043 10031 raft_consensus.cc:740] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.283289 10032 tablet_bootstrap.cc:492] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:49.284108 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b84197018ff44b40b7219e21846339f2" } modify_external_catalogs: true
I20251027 21:07:49.284278 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b84197018ff44b40b7219e21846339f2" } modify_external_catalogs: true
I20251027 21:07:49.285099 10031 consensus_queue.cc:260] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.285641 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12aa6e3af0694429b45bef5af3592377" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:07:49.285930 9924 raft_consensus.cc:3060] T 12aa6e3af0694429b45bef5af3592377 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:49.287225 10031 ts_tablet_manager.cc:1434] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251027 21:07:49.287253 10032 tablet_bootstrap.cc:654] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.288635 10031 tablet_bootstrap.cc:492] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:49.288941 10089 leader_election.cc:290] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:49.289453 10043 log.cc:1199] T 5c87a5450d354506852c4b20bf0541d2 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/5c87a5450d354506852c4b20bf0541d2
I20251027 21:07:49.289956 10043 ts_tablet_manager.cc:1950] T 5c87a5450d354506852c4b20bf0541d2 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:49.290225 9566 leader_election.cc:304] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:07:49.290460 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0029b70ed3ff4db9a816a6190d36bf88" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:49.291333 9710 leader_election.cc:343] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:49.292443 10039 raft_consensus.cc:697] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:07:49.292809 10039 consensus_queue.cc:237] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.296584 10031 tablet_bootstrap.cc:654] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.298840 10043 tablet_replica.cc:333] T 84305bcf21b64d2c8c1df015a2e9615c P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:49.299091 10043 raft_consensus.cc:2243] T 84305bcf21b64d2c8c1df015a2e9615c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:49.299304 10043 raft_consensus.cc:2272] T 84305bcf21b64d2c8c1df015a2e9615c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:49.299314 10032 tablet_bootstrap.cc:492] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:49.299458 10032 ts_tablet_manager.cc:1403] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.003s
I20251027 21:07:49.299964 10043 ts_tablet_manager.cc:1916] T 84305bcf21b64d2c8c1df015a2e9615c P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:49.300359 10032 raft_consensus.cc:359] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.300498 10032 raft_consensus.cc:385] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.300588 10032 raft_consensus.cc:740] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.300832 10050 raft_consensus.cc:493] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:49.300972 10050 raft_consensus.cc:515] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.301642 10050 leader_election.cc:290] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:49.302309 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 5c87a5450d354506852c4b20bf0541d2 (table default.Table_d1fb7f4ef09e4bfdac00a75f752f0873 [id=1cf515b159954979acd73a0b276e7503]) successfully deleted
I20251027 21:07:49.302376 10032 consensus_queue.cc:260] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.302681 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3dfef8b741c42f68b664d7e0eab49c6" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:49.303259 9860 leader_election.cc:343] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:49.304221 10032 ts_tablet_manager.cc:1434] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251027 21:07:49.304530 10032 tablet_bootstrap.cc:492] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:49.303752 10089 raft_consensus.cc:2804] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:49.305975 10089 raft_consensus.cc:697] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:07:49.306249 10089 consensus_queue.cc:237] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:49.312763 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 995284883d834bbb9ef583b736ca8402 consensus state has no leader
I20251027 21:07:49.316392 10032 tablet_bootstrap.cc:654] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.318548 10031 tablet_bootstrap.cc:492] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:49.318967 10031 ts_tablet_manager.cc:1403] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.007s
I20251027 21:07:49.319813 10031 raft_consensus.cc:359] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.319990 10031 raft_consensus.cc:385] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.320096 10031 raft_consensus.cc:740] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.320303 10031 consensus_queue.cc:260] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.321213 10041 ts_tablet_manager.cc:1929] T f5da8d7ecccc4c3492f9d0d30ffea423 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:49.321352 10041 log.cc:1199] T f5da8d7ecccc4c3492f9d0d30ffea423 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/f5da8d7ecccc4c3492f9d0d30ffea423
I20251027 21:07:49.321807 10031 ts_tablet_manager.cc:1434] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:07:49.321848 10041 ts_tablet_manager.cc:1950] T f5da8d7ecccc4c3492f9d0d30ffea423 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:49.321935 10032 tablet_bootstrap.cc:492] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:49.322079 10032 ts_tablet_manager.cc:1403] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20251027 21:07:49.322747 10031 tablet_bootstrap.cc:492] T cbada3946dab4739a8369032ff3b8a48 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:49.323935 10032 raft_consensus.cc:359] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.324136 10032 raft_consensus.cc:385] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.324218 10032 raft_consensus.cc:740] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.324450 10032 consensus_queue.cc:260] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.324878 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet b3dfef8b741c42f68b664d7e0eab49c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.324985 9924 raft_consensus.cc:2468] T 12aa6e3af0694429b45bef5af3592377 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:49.325060 10031 tablet_bootstrap.cc:654] T cbada3946dab4739a8369032ff3b8a48 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.328267 10032 ts_tablet_manager.cc:1434] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251027 21:07:49.328356 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet f5da8d7ecccc4c3492f9d0d30ffea423 (table default.Table_50cddb609f3043f29b42899ad521fbee [id=63988803b0384c0b9766dccecf60804a]) successfully deleted
I20251027 21:07:49.328405 10041 tablet_replica.cc:333] T d642da56a67449aa95932180bc5b2c5f P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:49.328634 10041 raft_consensus.cc:2243] T d642da56a67449aa95932180bc5b2c5f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:49.328557 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f16bc07fcb4c48b7969d13b8341aefdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:49.328768 10041 raft_consensus.cc:2272] T d642da56a67449aa95932180bc5b2c5f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:07:49.329062 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f16bc07fcb4c48b7969d13b8341aefdc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.330183 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_23cfb97e14e443a1bc6763bc466837a6" } new_table_name: "default.Table_4723d51b934b4f6a90c790ee4bc7ea86" modify_external_catalogs: true
I20251027 21:07:49.331801 10032 tablet_bootstrap.cc:492] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:49.332937 10031 tablet_bootstrap.cc:492] T cbada3946dab4739a8369032ff3b8a48 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:49.333065 10031 ts_tablet_manager.cc:1403] T cbada3946dab4739a8369032ff3b8a48 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20251027 21:07:49.333413 10032 tablet_bootstrap.cc:654] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.333837 10031 raft_consensus.cc:359] T cbada3946dab4739a8369032ff3b8a48 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.334012 10031 raft_consensus.cc:385] T cbada3946dab4739a8369032ff3b8a48 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.334617 10031 raft_consensus.cc:740] T cbada3946dab4739a8369032ff3b8a48 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.334911 10031 consensus_queue.cc:260] T cbada3946dab4739a8369032ff3b8a48 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.335017 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_42431224890f43859abe3dedfae4db51" } modify_external_catalogs: true
I20251027 21:07:49.335203 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_42431224890f43859abe3dedfae4db51" } modify_external_catalogs: true
I20251027 21:07:49.335791 10041 ts_tablet_manager.cc:1916] T d642da56a67449aa95932180bc5b2c5f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:49.336154 10031 ts_tablet_manager.cc:1434] T cbada3946dab4739a8369032ff3b8a48 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20251027 21:07:49.337208 10031 tablet_bootstrap.cc:492] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:49.338723 10031 tablet_bootstrap.cc:654] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.339814 10142 raft_consensus.cc:493] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:49.339946 10142 raft_consensus.cc:515] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.340468 10142 leader_election.cc:290] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:49.340520 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_fd19a2c8dc2f4164a0326865a21cd141" } new_table_name: "default.Table_65dba19877524bfe9b252239a7cdcd37" modify_external_catalogs: true
I20251027 21:07:49.340956 10029 tablet_bootstrap.cc:492] T ba8b125474bc42a5946e8f2dd718f7f4 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:49.341037 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3dfef8b741c42f68b664d7e0eab49c6" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:49.341123 10029 ts_tablet_manager.cc:1403] T ba8b125474bc42a5946e8f2dd718f7f4 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.335s user 0.039s sys 0.096s
W20251027 21:07:49.341549 9710 leader_election.cc:343] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:49.340520 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3922d39d16744b058b9a6116c7bc539f" } new_table_name: "default.Table_c24f2baac2d6497ead37ab560db1221b" modify_external_catalogs: true
I20251027 21:07:49.341876 10029 raft_consensus.cc:359] T ba8b125474bc42a5946e8f2dd718f7f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.342044 10029 raft_consensus.cc:385] T ba8b125474bc42a5946e8f2dd718f7f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.342116 10029 raft_consensus.cc:740] T ba8b125474bc42a5946e8f2dd718f7f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.342756 10031 tablet_bootstrap.cc:492] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:49.342913 10031 ts_tablet_manager.cc:1403] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20251027 21:07:49.343778 10031 raft_consensus.cc:359] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.343914 10031 raft_consensus.cc:385] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.343982 10031 raft_consensus.cc:740] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.344141 10031 consensus_queue.cc:260] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.345275 10031 ts_tablet_manager.cc:1434] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:49.345118 9305 catalog_manager.cc:5649] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20251027 21:07:49.345526 10031 tablet_bootstrap.cc:492] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:49.345562 9305 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 0ac7111d13b6460a9f4c77e94e09d417 (table default.Table_e65d5d66ce8747a9a1a2abfa04cd2855 [id=e2b68fb34c3b48e8952ef6d4201fd93f]). Expected version 1 got 0
I20251027 21:07:49.348371 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a7f999b292d04332b831f454d4aa6743" } modify_external_catalogs: true
I20251027 21:07:49.349756 10031 tablet_bootstrap.cc:654] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.350066 10032 tablet_bootstrap.cc:492] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:49.350221 10032 ts_tablet_manager.cc:1403] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.006s
I20251027 21:07:49.350894 10029 consensus_queue.cc:260] T ba8b125474bc42a5946e8f2dd718f7f4 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.351428 10029 ts_tablet_manager.cc:1434] T ba8b125474bc42a5946e8f2dd718f7f4 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20251027 21:07:49.354171 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a7f999b292d04332b831f454d4aa6743" } modify_external_catalogs: true
I20251027 21:07:49.355767 10032 raft_consensus.cc:359] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.356003 10032 raft_consensus.cc:385] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.356135 10032 raft_consensus.cc:740] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.356567 10032 consensus_queue.cc:260] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.357920 10029 tablet_bootstrap.cc:492] T 40bd202c1b834c949f9a5cee23b3486d P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:49.360683 10032 ts_tablet_manager.cc:1434] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251027 21:07:49.364190 9303 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 40bd202c1b834c949f9a5cee23b3486d (table default.Table_f9b456b781864738a6fa0d5c4bfa50fd [id=fd2beefa0629466daea4da402b033347]). Expected version 1 got 0
I20251027 21:07:49.364696 10032 tablet_bootstrap.cc:492] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:07:49.365796 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8fcfa6b85e7b46d78a84947e9f11ae46 consensus state has no leader
I20251027 21:07:49.367350 10032 tablet_bootstrap.cc:654] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.366195 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_90157e3841104d92aafc5d82ffea13a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.368203 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90157e3841104d92aafc5d82ffea13a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.369455 9303 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 9cec6294051b41868c9c7c7fadafa138 (table default.Table_5d8e4122af254070a2df7016e521a937 [id=12b1bbc770194455ab7292641980e8e1]). Expected version 2 got 0
I20251027 21:07:49.369649 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 9c239216cb8846f0b1aaa360ddb83321 (DEFAULT_TABLE table=default.Table_2d758ba690f8478ba7f983e8fd770219 [id=849a0760791548a38aceb18d0533f5f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.370131 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c239216cb8846f0b1aaa360ddb83321. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.371479 10029 tablet_bootstrap.cc:654] T 40bd202c1b834c949f9a5cee23b3486d P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:49.374397 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 533f231e3ff64531ae2331409a961425 consensus state has no leader
I20251027 21:07:49.374891 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_41078339d0834d069db2e8ae485bcc89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.375520 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e841c54a94d649c19b639f5021f150cd consensus state has no leader
W20251027 21:07:49.375761 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41078339d0834d069db2e8ae485bcc89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.376912 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d601b22c11c34ebd8559009cd144affe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.377426 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d601b22c11c34ebd8559009cd144affe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.377874 9304 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 40bd202c1b834c949f9a5cee23b3486d (table default.Table_f9b456b781864738a6fa0d5c4bfa50fd [id=fd2beefa0629466daea4da402b033347]). Expected version 1 got 0
I20251027 21:07:49.378072 9304 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet cbada3946dab4739a8369032ff3b8a48 (table default.Table_4f6664b7160e4549a50589454ff10313 [id=67b71acdadc64445bb2492e5d424cb2d]). Expected version 2 got 0
I20251027 21:07:49.380548 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet e3c9f64ee0544e59add4f73bca59df98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.380551 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 44783760defd433a972c9cd83933249c with delete_type TABLET_DATA_DELETED (replaced by tablet ffe1182e8f4e4a44b9a12e0a5192989f) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.384517 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 9c239216cb8846f0b1aaa360ddb83321 (DEFAULT_TABLE table=default.Table_2d758ba690f8478ba7f983e8fd770219 [id=849a0760791548a38aceb18d0533f5f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.384629 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b534fdb9e3084d6f9656dfdbb0e01750" } modify_external_catalogs: true
I20251027 21:07:49.384810 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b534fdb9e3084d6f9656dfdbb0e01750" } modify_external_catalogs: true
I20251027 21:07:49.384948 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c239216cb8846f0b1aaa360ddb83321. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.386057 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2ff5a5b701c540b8a9eb8b87aace0e77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.386504 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ff5a5b701c540b8a9eb8b87aace0e77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.389016 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 78884c2be75f45f881f5424d3b3f4e08 (DEFAULT_TABLE table=default.Table_12986d58c67145d69cc562041489fbc4 [id=0eb078640c174f8fadb959306a35cada]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.389422 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78884c2be75f45f881f5424d3b3f4e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.389664 10032 tablet_bootstrap.cc:492] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:49.389830 10032 ts_tablet_manager.cc:1403] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.000s
I20251027 21:07:49.390652 10032 raft_consensus.cc:359] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.390834 10032 raft_consensus.cc:385] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.390925 10032 raft_consensus.cc:740] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.391279 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 8fc2d7d6601d4cc2849064f4cdddc90d (DEFAULT_TABLE table=default.Table_741ffacfeaff46058d7e21e5bf7736fb [id=6f39910d23dd455f87a4ac847917b87f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.391332 10032 consensus_queue.cc:260] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.391978 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fc2d7d6601d4cc2849064f4cdddc90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.395851 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c305a5155cce4754bcbc792f3900b690"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.396318 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c305a5155cce4754bcbc792f3900b690 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.399253 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 78884c2be75f45f881f5424d3b3f4e08 (DEFAULT_TABLE table=default.Table_12986d58c67145d69cc562041489fbc4 [id=0eb078640c174f8fadb959306a35cada]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.399654 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78884c2be75f45f881f5424d3b3f4e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.399823 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 69da799c886841a19002d3c7c5084e73 (DEFAULT_TABLE table=default.Table_0a8b47a441114631917879e9fe99d612 [id=0cf4bae5f10d45c1a41880b45fa0b0f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.399989 9608 tablet_service.cc:1505] Processing CreateTablet for tablet ed5619d295a246379b625f35692203e4 (DEFAULT_TABLE table=default.Table_84fa0625821f4deeb3422622babb5429 [id=e3c04aba1dbf46928a0c36444dd69d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.400154 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69da799c886841a19002d3c7c5084e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.400386 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed5619d295a246379b625f35692203e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.402871 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 8fc2d7d6601d4cc2849064f4cdddc90d (DEFAULT_TABLE table=default.Table_741ffacfeaff46058d7e21e5bf7736fb [id=6f39910d23dd455f87a4ac847917b87f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.403162 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fc2d7d6601d4cc2849064f4cdddc90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.408627 9604 tablet_service.cc:1505] Processing CreateTablet for tablet 7aa8c538b6c849a28a28ffe706306891 (DEFAULT_TABLE table=default.Table_758d1e73474647eaa8a15b6cff350af6 [id=8ed572891f9b41cba8ed984911fa6b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.408958 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aa8c538b6c849a28a28ffe706306891. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.409673 10032 ts_tablet_manager.cc:1434] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20251027 21:07:49.414851 10032 tablet_bootstrap.cc:492] T ba8b125474bc42a5946e8f2dd718f7f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:49.414855 9606 tablet_service.cc:1505] Processing CreateTablet for tablet e0968c2f25b64f6685f38a999ee9d60e (DEFAULT_TABLE table=default.Table_68f478946b1543a68842ffad9e14864f [id=830444c79e80486aa3e7668a0ec91023]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.415271 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0968c2f25b64f6685f38a999ee9d60e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.419759 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet cad3a3e910ac42399cf371ad82c89ff7
I20251027 21:07:49.422277 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 6bf9c2a78aac42528d67846dc58adc40 (DEFAULT_TABLE table=default.Table_908c9569154d403eb8aeab2a65da7f67 [id=af6cb904f8cd4a929be7a978f9bc2001]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.422641 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet cad3a3e910ac42399cf371ad82c89ff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:49.423062 9604 tablet_service.cc:1505] Processing CreateTablet for tablet e841c54a94d649c19b639f5021f150cd (DEFAULT_TABLE table=default.Table_c24f2baac2d6497ead37ab560db1221b [id=d392262a11e24cf0991bc200475b0ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.430157 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6df2aec5939c4b3cbfb82b376fe5b53d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.430677 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6df2aec5939c4b3cbfb82b376fe5b53d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.432437 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet ba8b125474bc42a5946e8f2dd718f7f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 77797eea3e9c4f8099dc9d58f116200b) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:49.434021 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 81e914ecc3c74b869e9fc3ad102887ec
I20251027 21:07:49.434801 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_80ba3902da484ff8bc2100860960f098"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.435330 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80ba3902da484ff8bc2100860960f098 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.435388 10050 raft_consensus.cc:493] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:49.437539 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet 81e914ecc3c74b869e9fc3ad102887ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:49.438990 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bf9c2a78aac42528d67846dc58adc40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.441588 9610 tablet_service.cc:1505] Processing CreateTablet for tablet f1aa5f7f76cb4a1ba15f016942f17067 (DEFAULT_TABLE table=default.Table_3e0f707f8e2844bfaf937150b747a935 [id=d7dcec1e03494fd5a219a97ea8b1ab1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.442009 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1aa5f7f76cb4a1ba15f016942f17067. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.446345 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e841c54a94d649c19b639f5021f150cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.446259 10050 raft_consensus.cc:515] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.448099 10050 leader_election.cc:290] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:49.448146 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40bd202c1b834c949f9a5cee23b3486d" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:49.448987 9860 leader_election.cc:343] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:49.449707 9886 tablet_service.cc:1505] Processing CreateTablet for tablet e0968c2f25b64f6685f38a999ee9d60e (DEFAULT_TABLE table=default.Table_68f478946b1543a68842ffad9e14864f [id=830444c79e80486aa3e7668a0ec91023]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.450421 9892 tablet_service.cc:1505] Processing CreateTablet for tablet ed5619d295a246379b625f35692203e4 (DEFAULT_TABLE table=default.Table_84fa0625821f4deeb3422622babb5429 [id=e3c04aba1dbf46928a0c36444dd69d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.447121 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 69da799c886841a19002d3c7c5084e73 (DEFAULT_TABLE table=default.Table_0a8b47a441114631917879e9fe99d612 [id=0cf4bae5f10d45c1a41880b45fa0b0f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.452214 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 7aa8c538b6c849a28a28ffe706306891 (DEFAULT_TABLE table=default.Table_758d1e73474647eaa8a15b6cff350af6 [id=8ed572891f9b41cba8ed984911fa6b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.452292 10049 raft_consensus.cc:697] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:07:49.452762 10049 consensus_queue.cc:237] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.453986 9897 tablet_service.cc:1505] Processing CreateTablet for tablet e841c54a94d649c19b639f5021f150cd (DEFAULT_TABLE table=default.Table_c24f2baac2d6497ead37ab560db1221b [id=d392262a11e24cf0991bc200475b0ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.454646 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 6bf9c2a78aac42528d67846dc58adc40 (DEFAULT_TABLE table=default.Table_908c9569154d403eb8aeab2a65da7f67 [id=af6cb904f8cd4a929be7a978f9bc2001]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.455145 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet cad3a3e910ac42399cf371ad82c89ff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.455402 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69da799c886841a19002d3c7c5084e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.455713 9894 tablet_service.cc:1552] Processing DeleteTablet for tablet 81e914ecc3c74b869e9fc3ad102887ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.456386 9894 tablet_service.cc:1505] Processing CreateTablet for tablet ccaf5c2f8e544842880ba8ba27300ae6 (DEFAULT_TABLE table=default.Table_18e48b818fc040bfa4272f08e203dc3e [id=2b0d4e085a8e42879c67963231a24b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.456496 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 6db0f31c12d14e66a4ec7fc722d62c3d
I20251027 21:07:49.456812 9886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0968c2f25b64f6685f38a999ee9d60e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.457913 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccaf5c2f8e544842880ba8ba27300ae6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:49.459189 9711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 292391us
I20251027 21:07:49.460737 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet 6db0f31c12d14e66a4ec7fc722d62c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.461330 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_bf6a0088d06247ca9eb20f7fbfb0f0e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.461817 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf6a0088d06247ca9eb20f7fbfb0f0e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.465965 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aa8c538b6c849a28a28ffe706306891. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.466279 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bf9c2a78aac42528d67846dc58adc40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.466104 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c87a5450d354506852c4b20bf0541d2 with delete_type TABLET_DATA_DELETED (replaced by tablet 77b2d15e86b244708a2d07f88bf8278d) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.466578 9887 tablet_service.cc:1505] Processing CreateTablet for tablet f1aa5f7f76cb4a1ba15f016942f17067 (DEFAULT_TABLE table=default.Table_3e0f707f8e2844bfaf937150b747a935 [id=d7dcec1e03494fd5a219a97ea8b1ab1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.468494 10032 tablet_bootstrap.cc:654] T ba8b125474bc42a5946e8f2dd718f7f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.469110 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e841c54a94d649c19b639f5021f150cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.469647 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1aa5f7f76cb4a1ba15f016942f17067. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.470521 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed5619d295a246379b625f35692203e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.471156 9608 tablet_service.cc:1552] Processing DeleteTablet for tablet 6db0f31c12d14e66a4ec7fc722d62c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:49.472379 9611 tablet_service.cc:1505] Processing CreateTablet for tablet ccaf5c2f8e544842880ba8ba27300ae6 (DEFAULT_TABLE table=default.Table_18e48b818fc040bfa4272f08e203dc3e [id=2b0d4e085a8e42879c67963231a24b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.472756 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccaf5c2f8e544842880ba8ba27300ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.473024 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet f5da8d7ecccc4c3492f9d0d30ffea423 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.473157 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 9c239216cb8846f0b1aaa360ddb83321 (DEFAULT_TABLE table=default.Table_2d758ba690f8478ba7f983e8fd770219 [id=849a0760791548a38aceb18d0533f5f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.473315 10040 tablet_replica.cc:333] T f5da8d7ecccc4c3492f9d0d30ffea423 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:49.473518 10040 raft_consensus.cc:2243] T f5da8d7ecccc4c3492f9d0d30ffea423 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:49.473572 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c239216cb8846f0b1aaa360ddb83321. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.473688 10040 raft_consensus.cc:2272] T f5da8d7ecccc4c3492f9d0d30ffea423 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:49.473868 9610 tablet_service.cc:1505] Processing CreateTablet for tablet dc589852710342f7bd017193e4075303 (DEFAULT_TABLE table=default.Table_d2ff301b3e2b4602a2483859973a02d6 [id=ccc88d98bcd74bde873cca8c9ce3e134]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.474267 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc589852710342f7bd017193e4075303. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.474339 10040 ts_tablet_manager.cc:1916] T f5da8d7ecccc4c3492f9d0d30ffea423 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:49.483922 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9cec6294051b41868c9c7c7fadafa138" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:07:49.484141 9924 raft_consensus.cc:3060] T 9cec6294051b41868c9c7c7fadafa138 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:49.486140 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3dfef8b741c42f68b664d7e0eab49c6" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:49.488729 9925 raft_consensus.cc:2468] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:07:49.496824 10043 ts_tablet_manager.cc:1929] T 84305bcf21b64d2c8c1df015a2e9615c P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:49.497345 10043 log.cc:1199] T 84305bcf21b64d2c8c1df015a2e9615c P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/84305bcf21b64d2c8c1df015a2e9615c
I20251027 21:07:49.497713 10043 ts_tablet_manager.cc:1950] T 84305bcf21b64d2c8c1df015a2e9615c P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:49.498260 10041 ts_tablet_manager.cc:1929] T d642da56a67449aa95932180bc5b2c5f P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:49.517058 10041 log.cc:1199] T d642da56a67449aa95932180bc5b2c5f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/d642da56a67449aa95932180bc5b2c5f
I20251027 21:07:49.517391 10043 tablet_replica.cc:333] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:49.517369 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 84305bcf21b64d2c8c1df015a2e9615c (table default.Table_41a04598f12247fbac5f4103d2cbf22d [id=2e28a4d0f9204351b93335736a9abe18]) successfully deleted
I20251027 21:07:49.517652 10043 raft_consensus.cc:2243] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:07:49.517899 10043 pending_rounds.cc:70] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14: Trying to abort 1 pending ops.
I20251027 21:07:49.517966 10043 pending_rounds.cc:77] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510607083700224 op_type: NO_OP noop_request { }
I20251027 21:07:49.518088 10043 raft_consensus.cc:2889] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20251027 21:07:49.518067 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 779b7f63b0fa4f9ab47cf5799b3e085d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 779b7f63b0fa4f9ab47cf5799b3e085d already in progress: creating tablet
I20251027 21:07:49.518297 10043 raft_consensus.cc:2272] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251027 21:07:49.518601 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet b1bdb2be5032415187c2826c14e75f07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1bdb2be5032415187c2826c14e75f07 already in progress: creating tablet
I20251027 21:07:49.520021 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet a9cc64ad85334daa9a09538e24ebd8a5
I20251027 21:07:49.521529 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet ba8b125474bc42a5946e8f2dd718f7f4 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:49.521813 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c49775486ff24686b0f45c47e2757aba
I20251027 21:07:49.521898 9309 master_service.cc:954] ReplaceTablet: received request to replace tablet 0029b70ed3ff4db9a816a6190d36bf88 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:49.522100 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet a9cc64ad85334daa9a09538e24ebd8a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.522569 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet cbada3946dab4739a8369032ff3b8a48 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:49.523473 10043 ts_tablet_manager.cc:1916] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:49.523959 10050 raft_consensus.cc:493] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:49.524204 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet c49775486ff24686b0f45c47e2757aba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.524134 10050 raft_consensus.cc:515] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.524861 10050 leader_election.cc:290] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:49.525988 9306 catalog_manager.cc:5649] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77b0513f17c0472088901a1ce8b08218 (127.9.14.66). New cstate: current_term: 1 leader_uuid: "77b0513f17c0472088901a1ce8b08218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20251027 21:07:49.526434 9306 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 21b785277d604543bfa7816ed96d6181 (table default.Table_f1908c85b5144e90a366ca29ba5d1b9b [id=e3de2e5ea6484be7af68eb8601778b85]). Expected version 2 got 0
I20251027 21:07:49.526741 9306 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet cbada3946dab4739a8369032ff3b8a48 (table default.Table_4f6664b7160e4549a50589454ff10313 [id=67b71acdadc64445bb2492e5d424cb2d]). Expected version 2 got 0
I20251027 21:07:49.526868 9306 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 0ac7111d13b6460a9f4c77e94e09d417 (table default.Table_e65d5d66ce8747a9a1a2abfa04cd2855 [id=e2b68fb34c3b48e8952ef6d4201fd93f]). Expected version 1 got 0
I20251027 21:07:49.527678 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_053064f4a69d499ba2862255e92fbaab" } new_table_name: "default.Table_750d3fcb86e54ef3bdf3519d0eace745" modify_external_catalogs: true
I20251027 21:07:49.529536 9309 master_service.cc:954] ReplaceTablet: received request to replace tablet 019a892bfdd54e62bc3560a341d5c2da from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:49.530072 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_730efea4f3644d55a230c2d1b57ea69e" } new_table_name: "default.Table_a430283c1a4d4d8283ec004c3d705794" modify_external_catalogs: true
I20251027 21:07:49.531481 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_04ef6b3afcc34d84804cd6e9613d7bc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.531985 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04ef6b3afcc34d84804cd6e9613d7bc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.533556 9611 tablet_service.cc:1552] Processing DeleteTablet for tablet c49775486ff24686b0f45c47e2757aba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:49.533852 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet a9cc64ad85334daa9a09538e24ebd8a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:49.535245 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44783760defd433a972c9cd83933249c" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:49.535787 9860 leader_election.cc:343] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:49.537760 10041 ts_tablet_manager.cc:1950] T d642da56a67449aa95932180bc5b2c5f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:49.538693 10041 tablet_replica.cc:333] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:49.538862 10041 raft_consensus.cc:2243] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:49.539033 10041 raft_consensus.cc:2272] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:49.539525 10041 ts_tablet_manager.cc:1916] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:49.540702 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet d642da56a67449aa95932180bc5b2c5f (table default.Table_178e5d554faa4e119ac8f52da2d284d7 [id=05edeb649c544c2f899ed42a87cef2c2]) successfully deleted
I20251027 21:07:49.543174 10049 raft_consensus.cc:493] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:49.543329 10049 raft_consensus.cc:515] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.543941 10049 leader_election.cc:290] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
W20251027 21:07:49.544884 9710 leader_election.cc:343] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 63662f175a1c450f8f67f37da971db37
I20251027 21:07:49.545764 10031 tablet_bootstrap.cc:492] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:49.545923 10031 ts_tablet_manager.cc:1403] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.200s user 0.013s sys 0.021s
I20251027 21:07:49.546833 10031 raft_consensus.cc:359] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.547081 10031 raft_consensus.cc:385] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.547158 10031 raft_consensus.cc:740] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.547400 10031 consensus_queue.cc:260] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.547945 10031 ts_tablet_manager.cc:1434] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251027 21:07:49.548255 10031 tablet_bootstrap.cc:492] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:49.553906 10043 ts_tablet_manager.cc:1929] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:07:49.554149 10043 log.cc:1199] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/12aa6e3af0694429b45bef5af3592377
I20251027 21:07:49.554769 10043 ts_tablet_manager.cc:1950] T 12aa6e3af0694429b45bef5af3592377 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:49.556216 10031 tablet_bootstrap.cc:654] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.556242 10043 tablet_replica.cc:333] T f5da8d7ecccc4c3492f9d0d30ffea423 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:49.556555 10043 raft_consensus.cc:2243] T f5da8d7ecccc4c3492f9d0d30ffea423 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:49.556783 10043 raft_consensus.cc:2272] T f5da8d7ecccc4c3492f9d0d30ffea423 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:49.557471 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 12aa6e3af0694429b45bef5af3592377 (table default.Table_36bfef90af1d48f78b2cce6a78527a44 [id=0065037e95714bb6ad48cbe647173541]) successfully deleted
W20251027 21:07:49.557735 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 7d8825b80cec4ba68da077ca3b8d89db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d8825b80cec4ba68da077ca3b8d89db already in progress: creating tablet
I20251027 21:07:49.557787 10043 ts_tablet_manager.cc:1916] T f5da8d7ecccc4c3492f9d0d30ffea423 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:49.559463 9924 raft_consensus.cc:2468] T 9cec6294051b41868c9c7c7fadafa138 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
W20251027 21:07:49.566660 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 533f34d4872444278477abcce89da9ec has no consensus state
W20251027 21:07:49.570277 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ae124514964440c5b88b16ac9f8a2567 consensus state has no leader
W20251027 21:07:49.571604 9297 catalog_manager.cc:5101] could not convert tablet's 21b785277d604543bfa7816ed96d6181 schema from PB: OK
I20251027 21:07:49.571921 10043 ts_tablet_manager.cc:1929] T f5da8d7ecccc4c3492f9d0d30ffea423 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:49.572225 10043 log.cc:1199] T f5da8d7ecccc4c3492f9d0d30ffea423 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/f5da8d7ecccc4c3492f9d0d30ffea423
I20251027 21:07:49.572760 10043 ts_tablet_manager.cc:1950] T f5da8d7ecccc4c3492f9d0d30ffea423 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
W20251027 21:07:49.573510 9278 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 73956.3us
I20251027 21:07:49.574303 10040 ts_tablet_manager.cc:1929] T f5da8d7ecccc4c3492f9d0d30ffea423 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:49.574893 10040 log.cc:1199] T f5da8d7ecccc4c3492f9d0d30ffea423 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/f5da8d7ecccc4c3492f9d0d30ffea423
I20251027 21:07:49.577394 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet f5da8d7ecccc4c3492f9d0d30ffea423 (table default.Table_50cddb609f3043f29b42899ad521fbee [id=63988803b0384c0b9766dccecf60804a]) successfully deleted
W20251027 21:07:49.577694 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 019a892bfdd54e62bc3560a341d5c2da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 019a892bfdd54e62bc3560a341d5c2da already in progress: creating tablet
W20251027 21:07:49.577919 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 601c74459d3a4a41b812fdcdf8b0a62c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 601c74459d3a4a41b812fdcdf8b0a62c already in progress: creating tablet
I20251027 21:07:49.578075 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a061b11818fc4a2db61e51a5972d69db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.578543 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 2d659e4d73ea4ff3975a4b58a794090a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d659e4d73ea4ff3975a4b58a794090a already in progress: creating tablet
W20251027 21:07:49.578697 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a061b11818fc4a2db61e51a5972d69db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:49.578742 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 1776590d8387494da5dfbd08db17f4b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1776590d8387494da5dfbd08db17f4b1 already in progress: creating tablet
W20251027 21:07:49.578949 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 65fbcaa5447c48818fabfccac22c4048 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65fbcaa5447c48818fabfccac22c4048 already in progress: creating tablet
I20251027 21:07:49.579047 10043 tablet_replica.cc:333] T ba8b125474bc42a5946e8f2dd718f7f4 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
W20251027 21:07:49.579162 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet cad3a3e910ac42399cf371ad82c89ff7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cad3a3e910ac42399cf371ad82c89ff7 already in progress: creating tablet
I20251027 21:07:49.579360 10043 raft_consensus.cc:2243] T ba8b125474bc42a5946e8f2dd718f7f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:49.579658 10043 raft_consensus.cc:2272] T ba8b125474bc42a5946e8f2dd718f7f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:49.582387 10049 raft_consensus.cc:493] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:49.582558 10049 raft_consensus.cc:515] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.582659 10043 ts_tablet_manager.cc:1916] T ba8b125474bc42a5946e8f2dd718f7f4 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:49.582865 10040 ts_tablet_manager.cc:1950] T f5da8d7ecccc4c3492f9d0d30ffea423 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:49.583112 10049 leader_election.cc:290] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:49.583884 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_741ffacfeaff46058d7e21e5bf7736fb" } new_table_name: "default.Table_1e7bc61476164b5bbcad34206a7d7d8f" modify_external_catalogs: true
W20251027 21:07:49.584086 9710 leader_election.cc:343] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 84305bcf21b64d2c8c1df015a2e9615c
I20251027 21:07:49.585402 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet cbada3946dab4739a8369032ff3b8a48
I20251027 21:07:49.585819 9303 catalog_manager.cc:6719] ReplaceTablet: tablet cbada3946dab4739a8369032ff3b8a48 deleted and replaced by tablet 4e8c83b501d8426fa0cad6a89f9a0b9b
I20251027 21:07:49.587633 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet cbada3946dab4739a8369032ff3b8a48 with delete_type TABLET_DATA_DELETED (replaced by tablet 4e8c83b501d8426fa0cad6a89f9a0b9b) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.588277 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5675f5f4f1424e2581d644b9be1878ce" } new_table_name: "default.Table_f802ec6bccdf4ba897356686e0210911" modify_external_catalogs: true
I20251027 21:07:49.589458 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet cbada3946dab4739a8369032ff3b8a48 with delete_type TABLET_DATA_DELETED (replaced by tablet 4e8c83b501d8426fa0cad6a89f9a0b9b) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:49.590342 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_530197464d684950839a65bdf56549b8" } new_table_name: "default.Table_dddd93d5fb3c4727b64ded3ff6cc3382" modify_external_catalogs: true
I20251027 21:07:49.591586 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 0029b70ed3ff4db9a816a6190d36bf88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.592307 10040 tablet_replica.cc:333] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:49.592598 10040 raft_consensus.cc:2243] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:49.592869 10040 raft_consensus.cc:2272] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:49.593425 10040 ts_tablet_manager.cc:1916] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:49.593995 9281 master_runner.cc:186] Error getting master registration for 127.9.14.124:45071: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: connect: Connection refused (error 111),
I20251027 21:07:49.594179 9281 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251027 21:07:49.594487 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 12aa6e3af0694429b45bef5af3592377 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.597236 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e5d52b7ec114be5ae68e57280c2eea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.597352 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 78a0a8b05f10477c923ab565839141b8 with delete_type TABLET_DATA_DELETED (replaced by tablet 1698ae8a37ad4560836b3b1daca1cf5e) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.597393 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet d642da56a67449aa95932180bc5b2c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.597450 9755 tablet_service.cc:1505] Processing CreateTablet for tablet ccaf5c2f8e544842880ba8ba27300ae6 (DEFAULT_TABLE table=default.Table_18e48b818fc040bfa4272f08e203dc3e [id=2b0d4e085a8e42879c67963231a24b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.597568 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c87a5450d354506852c4b20bf0541d2 with delete_type TABLET_DATA_DELETED (replaced by tablet 77b2d15e86b244708a2d07f88bf8278d) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.597525 9749 tablet_service.cc:1505] Processing CreateTablet for tablet f1aa5f7f76cb4a1ba15f016942f17067 (DEFAULT_TABLE table=default.Table_3e0f707f8e2844bfaf937150b747a935 [id=d7dcec1e03494fd5a219a97ea8b1ab1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.597821 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet b3dfef8b741c42f68b664d7e0eab49c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.597879 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccaf5c2f8e544842880ba8ba27300ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.594880 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 63662f175a1c450f8f67f37da971db37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.595319 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 78884c2be75f45f881f5424d3b3f4e08 (DEFAULT_TABLE table=default.Table_12986d58c67145d69cc562041489fbc4 [id=0eb078640c174f8fadb959306a35cada]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.595638 9741 tablet_service.cc:1505] Processing CreateTablet for tablet e0968c2f25b64f6685f38a999ee9d60e (DEFAULT_TABLE table=default.Table_68f478946b1543a68842ffad9e14864f [id=830444c79e80486aa3e7668a0ec91023]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.595777 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 8fc2d7d6601d4cc2849064f4cdddc90d (DEFAULT_TABLE table=default.Table_741ffacfeaff46058d7e21e5bf7736fb [id=6f39910d23dd455f87a4ac847917b87f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.595932 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet cad3a3e910ac42399cf371ad82c89ff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.596010 9747 tablet_service.cc:1505] Processing CreateTablet for tablet 7aa8c538b6c849a28a28ffe706306891 (DEFAULT_TABLE table=default.Table_758d1e73474647eaa8a15b6cff350af6 [id=8ed572891f9b41cba8ed984911fa6b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.596146 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 69da799c886841a19002d3c7c5084e73 (DEFAULT_TABLE table=default.Table_0a8b47a441114631917879e9fe99d612 [id=0cf4bae5f10d45c1a41880b45fa0b0f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.596354 9751 tablet_service.cc:1505] Processing CreateTablet for tablet e841c54a94d649c19b639f5021f150cd (DEFAULT_TABLE table=default.Table_c24f2baac2d6497ead37ab560db1221b [id=d392262a11e24cf0991bc200475b0ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.596517 9743 tablet_service.cc:1505] Processing CreateTablet for tablet ed5619d295a246379b625f35692203e4 (DEFAULT_TABLE table=default.Table_84fa0625821f4deeb3422622babb5429 [id=e3c04aba1dbf46928a0c36444dd69d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.596554 9754 tablet_service.cc:1505] Processing CreateTablet for tablet dc589852710342f7bd017193e4075303 (DEFAULT_TABLE table=default.Table_d2ff301b3e2b4602a2483859973a02d6 [id=ccc88d98bcd74bde873cca8c9ce3e134]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.596762 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 6db0f31c12d14e66a4ec7fc722d62c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.596716 9744 tablet_service.cc:1505] Processing CreateTablet for tablet 6bf9c2a78aac42528d67846dc58adc40 (DEFAULT_TABLE table=default.Table_908c9569154d403eb8aeab2a65da7f67 [id=af6cb904f8cd4a929be7a978f9bc2001]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.596948 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 81e914ecc3c74b869e9fc3ad102887ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.596990 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet c49775486ff24686b0f45c47e2757aba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.603395 10043 ts_tablet_manager.cc:1929] T ba8b125474bc42a5946e8f2dd718f7f4 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:49.603516 10043 log.cc:1199] T ba8b125474bc42a5946e8f2dd718f7f4 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/ba8b125474bc42a5946e8f2dd718f7f4
I20251027 21:07:49.603514 9304 master_service.cc:954] ReplaceTablet: received request to replace tablet 583fba7c750e4ac7b768da447002c211 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:49.603832 10043 ts_tablet_manager.cc:1950] T ba8b125474bc42a5946e8f2dd718f7f4 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:49.604130 9891 tablet_service.cc:1505] Processing CreateTablet for tablet dc589852710342f7bd017193e4075303 (DEFAULT_TABLE table=default.Table_d2ff301b3e2b4602a2483859973a02d6 [id=ccc88d98bcd74bde873cca8c9ce3e134]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.604519 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc589852710342f7bd017193e4075303. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.606021 10029 tablet_bootstrap.cc:492] T 40bd202c1b834c949f9a5cee23b3486d P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:49.606179 10029 ts_tablet_manager.cc:1403] T 40bd202c1b834c949f9a5cee23b3486d P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.249s user 0.004s sys 0.004s
I20251027 21:07:49.606876 10029 raft_consensus.cc:359] T 40bd202c1b834c949f9a5cee23b3486d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.607052 10029 raft_consensus.cc:385] T 40bd202c1b834c949f9a5cee23b3486d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.607146 10029 raft_consensus.cc:740] T 40bd202c1b834c949f9a5cee23b3486d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.607427 10041 ts_tablet_manager.cc:1929] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:49.607551 10041 log.cc:1199] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/6e5d52b7ec114be5ae68e57280c2eea8
I20251027 21:07:49.608439 10041 ts_tablet_manager.cc:1950] T 6e5d52b7ec114be5ae68e57280c2eea8 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:49.609375 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 6e5d52b7ec114be5ae68e57280c2eea8 (table default.Table_d81917295a3548f787986bf566037719 [id=709b33acbd184a218ec6b700150b58b1]) successfully deleted
W20251027 21:07:49.611311 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8fc2d7d6601d4cc2849064f4cdddc90d consensus state has no leader
W20251027 21:07:49.616649 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 970fbf01a4c3406dafe1ea4cd005ea23 consensus state has no leader
I20251027 21:07:49.616806 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 583fba7c750e4ac7b768da447002c211
W20251027 21:07:49.617180 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 995284883d834bbb9ef583b736ca8402 consensus state has no leader
I20251027 21:07:49.617296 9304 catalog_manager.cc:6719] ReplaceTablet: tablet 583fba7c750e4ac7b768da447002c211 deleted and replaced by tablet f34512059f0c47fc9af7db2645f86b40
I20251027 21:07:49.618492 10050 raft_consensus.cc:493] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:49.618638 10050 raft_consensus.cc:515] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.619246 10050 leader_election.cc:290] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:49.619815 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3c9f64ee0544e59add4f73bca59df98" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:49.620306 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet 583fba7c750e4ac7b768da447002c211 with delete_type TABLET_DATA_DELETED (replaced by tablet f34512059f0c47fc9af7db2645f86b40) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:49.620553 9860 leader_election.cc:343] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:49.620489 10029 consensus_queue.cc:260] T 40bd202c1b834c949f9a5cee23b3486d P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.621536 10029 ts_tablet_manager.cc:1434] T 40bd202c1b834c949f9a5cee23b3486d P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251027 21:07:49.621819 10029 tablet_bootstrap.cc:492] T 0029b70ed3ff4db9a816a6190d36bf88 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:49.624148 10029 tablet_bootstrap.cc:654] T 0029b70ed3ff4db9a816a6190d36bf88 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.618350 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet 583fba7c750e4ac7b768da447002c211 with delete_type TABLET_DATA_DELETED (replaced by tablet f34512059f0c47fc9af7db2645f86b40) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.626430 10032 tablet_bootstrap.cc:492] T ba8b125474bc42a5946e8f2dd718f7f4 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:49.626899 10032 ts_tablet_manager.cc:1403] T ba8b125474bc42a5946e8f2dd718f7f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.217s user 0.005s sys 0.006s
I20251027 21:07:49.627334 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d2bf88aca4cc44df99d13783ea2722f8" } modify_external_catalogs: true
I20251027 21:07:49.627511 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d2bf88aca4cc44df99d13783ea2722f8" } modify_external_catalogs: true
I20251027 21:07:49.627887 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet cbada3946dab4739a8369032ff3b8a48 with delete_type TABLET_DATA_DELETED (replaced by tablet 4e8c83b501d8426fa0cad6a89f9a0b9b) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.628003 10032 raft_consensus.cc:359] T ba8b125474bc42a5946e8f2dd718f7f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.628201 10032 raft_consensus.cc:385] T ba8b125474bc42a5946e8f2dd718f7f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.628396 10032 raft_consensus.cc:740] T ba8b125474bc42a5946e8f2dd718f7f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.628767 10041 tablet_replica.cc:333] T 12aa6e3af0694429b45bef5af3592377 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:49.628710 10032 consensus_queue.cc:260] T ba8b125474bc42a5946e8f2dd718f7f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.629052 10041 raft_consensus.cc:2243] T 12aa6e3af0694429b45bef5af3592377 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:49.629256 10041 raft_consensus.cc:2272] T 12aa6e3af0694429b45bef5af3592377 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:49.629890 10041 ts_tablet_manager.cc:1916] T 12aa6e3af0694429b45bef5af3592377 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:49.635856 10029 tablet_bootstrap.cc:492] T 0029b70ed3ff4db9a816a6190d36bf88 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:49.636032 10029 ts_tablet_manager.cc:1403] T 0029b70ed3ff4db9a816a6190d36bf88 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20251027 21:07:49.636206 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet ba8b125474bc42a5946e8f2dd718f7f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 77797eea3e9c4f8099dc9d58f116200b) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.636814 10029 raft_consensus.cc:359] T 0029b70ed3ff4db9a816a6190d36bf88 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.637001 10029 raft_consensus.cc:385] T 0029b70ed3ff4db9a816a6190d36bf88 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:49.637072 10029 raft_consensus.cc:740] T 0029b70ed3ff4db9a816a6190d36bf88 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:49.625442 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet ba8b125474bc42a5946e8f2dd718f7f4 (table default.Table_16dd2e0479ea434dbbd0b900b340fc9f [id=e7b7bdb2ea7f468c98bfe88f900a5be8]) successfully deleted
W20251027 21:07:49.637454 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 81e914ecc3c74b869e9fc3ad102887ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81e914ecc3c74b869e9fc3ad102887ec already in progress: creating tablet
W20251027 21:07:49.637732 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 6db0f31c12d14e66a4ec7fc722d62c3d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6db0f31c12d14e66a4ec7fc722d62c3d already in progress: creating tablet
W20251027 21:07:49.637940 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet c49775486ff24686b0f45c47e2757aba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c49775486ff24686b0f45c47e2757aba already in progress: creating tablet
W20251027 21:07:49.638196 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet a9cc64ad85334daa9a09538e24ebd8a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9cc64ad85334daa9a09538e24ebd8a5 already in progress: creating tablet
W20251027 21:07:49.638407 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet cbada3946dab4739a8369032ff3b8a48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbada3946dab4739a8369032ff3b8a48 already in progress: creating tablet
W20251027 21:07:49.638662 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 583fba7c750e4ac7b768da447002c211 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 583fba7c750e4ac7b768da447002c211 already in progress: creating tablet
I20251027 21:07:49.638686 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b0f0589285ac4a56b7a2b23ace6248f6" } modify_external_catalogs: true
I20251027 21:07:49.638944 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b0f0589285ac4a56b7a2b23ace6248f6" } modify_external_catalogs: true
I20251027 21:07:49.639158 10032 ts_tablet_manager.cc:1434] T ba8b125474bc42a5946e8f2dd718f7f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251027 21:07:49.639324 10041 ts_tablet_manager.cc:1929] T 12aa6e3af0694429b45bef5af3592377 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:49.639461 10041 log.cc:1199] T 12aa6e3af0694429b45bef5af3592377 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/12aa6e3af0694429b45bef5af3592377
I20251027 21:07:49.639771 10041 ts_tablet_manager.cc:1950] T 12aa6e3af0694429b45bef5af3592377 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:49.640499 10050 raft_consensus.cc:493] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:49.640673 10050 raft_consensus.cc:515] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.640723 10029 consensus_queue.cc:260] T 0029b70ed3ff4db9a816a6190d36bf88 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.641234 10029 ts_tablet_manager.cc:1434] T 0029b70ed3ff4db9a816a6190d36bf88 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251027 21:07:49.641332 10050 leader_election.cc:290] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:49.641717 10029 tablet_bootstrap.cc:492] T 6e5d52b7ec114be5ae68e57280c2eea8 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:49.642334 10032 tablet_bootstrap.cc:492] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:49.642783 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbada3946dab4739a8369032ff3b8a48" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:49.643765 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 12aa6e3af0694429b45bef5af3592377 (table default.Table_36bfef90af1d48f78b2cce6a78527a44 [id=0065037e95714bb6ad48cbe647173541]) successfully deleted
I20251027 21:07:49.644965 10029 tablet_bootstrap.cc:654] T 6e5d52b7ec114be5ae68e57280c2eea8 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:49.643322 9860 leader_election.cc:343] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:49.646669 10041 tablet_replica.cc:333] T 84305bcf21b64d2c8c1df015a2e9615c P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:49.653820 10041 raft_consensus.cc:2243] T 84305bcf21b64d2c8c1df015a2e9615c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:49.654109 10041 raft_consensus.cc:2272] T 84305bcf21b64d2c8c1df015a2e9615c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:49.655037 9304 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 40bd202c1b834c949f9a5cee23b3486d (table default.Table_f9b456b781864738a6fa0d5c4bfa50fd [id=fd2beefa0629466daea4da402b033347]). Expected version 1 got 0
I20251027 21:07:49.655109 10041 ts_tablet_manager.cc:1916] T 84305bcf21b64d2c8c1df015a2e9615c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:49.658752 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4f6664b7160e4549a50589454ff10313" } modify_external_catalogs: true
I20251027 21:07:49.658963 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4f6664b7160e4549a50589454ff10313" } modify_external_catalogs: true
I20251027 21:07:49.660665 9304 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 40bd202c1b834c949f9a5cee23b3486d (table default.Table_f9b456b781864738a6fa0d5c4bfa50fd [id=fd2beefa0629466daea4da402b033347]). Expected version 1 got 0
I20251027 21:07:49.663547 9887 tablet_service.cc:1505] Processing CreateTablet for tablet d8e008c868d04281beba0e5b204eb69a (DEFAULT_TABLE table=default.Table_04ef6b3afcc34d84804cd6e9613d7bc2 [id=8a1cda11e1b3436cb8ffa2616d543a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.664022 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8e008c868d04281beba0e5b204eb69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.664314 9890 tablet_service.cc:1505] Processing CreateTablet for tablet a9339c6b1154400eace53d765528fdf9 (DEFAULT_TABLE table=default.Table_6df2aec5939c4b3cbfb82b376fe5b53d [id=91fb9f95fe9343fca1d8cb0d0f781afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.664773 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9339c6b1154400eace53d765528fdf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.668196 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 9b04349b15f3494bb0933db56b7b24bc (DEFAULT_TABLE table=default.Table_2ff5a5b701c540b8a9eb8b87aace0e77 [id=69f10c0f3f3a4227ab4c35e939085634]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.668574 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b04349b15f3494bb0933db56b7b24bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.671461 9606 tablet_service.cc:1505] Processing CreateTablet for tablet d8e008c868d04281beba0e5b204eb69a (DEFAULT_TABLE table=default.Table_04ef6b3afcc34d84804cd6e9613d7bc2 [id=8a1cda11e1b3436cb8ffa2616d543a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.671895 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8e008c868d04281beba0e5b204eb69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.672037 9611 tablet_service.cc:1552] Processing DeleteTablet for tablet 0029b70ed3ff4db9a816a6190d36bf88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:49.672230 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 21a8fc4a69f3486e92b1a68d10faed38 (DEFAULT_TABLE table=default.Table_ba1d973c05e14c70944d76ff1ecbc1bc [id=490e0e5ec83b4f76bf71eb41c900ac06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.672483 9608 tablet_service.cc:1505] Processing CreateTablet for tablet 6f1129eecfb54448965222269d9c32b7 (DEFAULT_TABLE table=default.Table_f16bc07fcb4c48b7969d13b8341aefdc [id=b5f4e2be1a5e42f58d98b64b8700ef1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.672628 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21a8fc4a69f3486e92b1a68d10faed38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.672857 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f1129eecfb54448965222269d9c32b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.674448 9604 tablet_service.cc:1505] Processing CreateTablet for tablet a9339c6b1154400eace53d765528fdf9 (DEFAULT_TABLE table=default.Table_6df2aec5939c4b3cbfb82b376fe5b53d [id=91fb9f95fe9343fca1d8cb0d0f781afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.674827 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9339c6b1154400eace53d765528fdf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.677932 9615 tablet_service.cc:1505] Processing CreateTablet for tablet c400101d2a39490e83722500839198cd (DEFAULT_TABLE table=default.Table_97b01c1840c543838263ca4222695f54 [id=bea51a58fda64d40916f306c4aa8ab4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.678395 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 9b04349b15f3494bb0933db56b7b24bc (DEFAULT_TABLE table=default.Table_2ff5a5b701c540b8a9eb8b87aace0e77 [id=69f10c0f3f3a4227ab4c35e939085634]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.678658 9612 tablet_service.cc:1505] Processing CreateTablet for tablet f22f013b25074d4db9c7f13caf5b9b42 (DEFAULT_TABLE table=default.Table_d601b22c11c34ebd8559009cd144affe [id=b96a43739ca54dac9e598e703186606b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.678776 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b04349b15f3494bb0933db56b7b24bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.678957 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f22f013b25074d4db9c7f13caf5b9b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.679124 10043 tablet_replica.cc:333] T 0029b70ed3ff4db9a816a6190d36bf88 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:49.679371 10043 raft_consensus.cc:2243] T 0029b70ed3ff4db9a816a6190d36bf88 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:49.679564 10043 raft_consensus.cc:2272] T 0029b70ed3ff4db9a816a6190d36bf88 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:49.680868 10032 tablet_bootstrap.cc:654] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:49.678395 9605 tablet_service.cc:1505] Processing CreateTablet for tablet a287439ea6c9469da5fb56d84ee68048 (DEFAULT_TABLE table=default.Table_1ecaac632a024248b04e7c40e2c01d2c [id=4e2210c84e4c464da017d1b36ddb0a37]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.682185 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a287439ea6c9469da5fb56d84ee68048. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.682752 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c400101d2a39490e83722500839198cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.683298 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 10a44aecfade42969454126b990eca24
I20251027 21:07:49.683732 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet b87e809ff897417c88e5edc19e2d591e
I20251027 21:07:49.684481 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 4e8c83b501d8426fa0cad6a89f9a0b9b
I20251027 21:07:49.686547 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet ba8b125474bc42a5946e8f2dd718f7f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 77797eea3e9c4f8099dc9d58f116200b) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.689442 9894 tablet_service.cc:1505] Processing CreateTablet for tablet 6f1129eecfb54448965222269d9c32b7 (DEFAULT_TABLE table=default.Table_f16bc07fcb4c48b7969d13b8341aefdc [id=b5f4e2be1a5e42f58d98b64b8700ef1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.690536 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f1129eecfb54448965222269d9c32b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.691375 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40bd202c1b834c949f9a5cee23b3486d" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:49.690685 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3dfef8b741c42f68b664d7e0eab49c6" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:49.691579 9886 tablet_service.cc:1505] Processing CreateTablet for tablet a287439ea6c9469da5fb56d84ee68048 (DEFAULT_TABLE table=default.Table_1ecaac632a024248b04e7c40e2c01d2c [id=4e2210c84e4c464da017d1b36ddb0a37]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.692312 9886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a287439ea6c9469da5fb56d84ee68048. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.693856 9896 tablet_service.cc:1505] Processing CreateTablet for tablet c400101d2a39490e83722500839198cd (DEFAULT_TABLE table=default.Table_97b01c1840c543838263ca4222695f54 [id=bea51a58fda64d40916f306c4aa8ab4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.694392 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 92a2393cff8044028c86c8cd2bccd930 (DEFAULT_TABLE table=default.Table_790c0f90be104b8e8abf3491796d54bc [id=e5fba763a46b488cb19c738cee468ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.695297 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 21a8fc4a69f3486e92b1a68d10faed38 (DEFAULT_TABLE table=default.Table_ba1d973c05e14c70944d76ff1ecbc1bc [id=490e0e5ec83b4f76bf71eb41c900ac06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.695900 9892 tablet_service.cc:1505] Processing CreateTablet for tablet f22f013b25074d4db9c7f13caf5b9b42 (DEFAULT_TABLE table=default.Table_d601b22c11c34ebd8559009cd144affe [id=b96a43739ca54dac9e598e703186606b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.698734 9888 tablet_service.cc:1505] Processing CreateTablet for tablet fe24af9838954a9bb680474800642e62 (DEFAULT_TABLE table=default.Table_80ba3902da484ff8bc2100860960f098 [id=dae1114f6b50443f8f6b79d18619d868]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.699707 9889 tablet_service.cc:1505] Processing CreateTablet for tablet abc7458fcfb94985acf76d77c0292b08 (DEFAULT_TABLE table=default.Table_9c027ff744824da9b9cac5574017e7fd [id=40a6848701284864b34f727da289347b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.700445 9905 tablet_service.cc:1505] Processing CreateTablet for tablet b719dc40d7e348608cdfc78401c19824 (DEFAULT_TABLE table=default.Table_41078339d0834d069db2e8ae485bcc89 [id=f641ffdf5fd7497096643ea3278c49d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.700855 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 132a01405a63447ab27b2137b803224f (DEFAULT_TABLE table=default.Table_673503358b54481c8877034c26349796 [id=c6aed7584a0b49d89bf9a8f8ef0cb429]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.702224 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet a9cc64ad85334daa9a09538e24ebd8a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.702426 9775 raft_consensus.cc:2468] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 0.
I20251027 21:07:49.702682 9774 raft_consensus.cc:2468] T 40bd202c1b834c949f9a5cee23b3486d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 0.
I20251027 21:07:49.702843 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet b87e809ff897417c88e5edc19e2d591e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.703405 9904 tablet_service.cc:1505] Processing CreateTablet for tablet c5f2e60caa824b5db2330b423221873c (DEFAULT_TABLE table=default.Table_fc9bf567c9ff4ddeaccaec2c6fddca99 [id=0e7e4ea125894e78a2162e9e8063ee33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.706442 9900 tablet_service.cc:1505] Processing CreateTablet for tablet d52d71f4976143e4a514350621ebe309 (DEFAULT_TABLE table=default.Table_c305a5155cce4754bcbc792f3900b690 [id=82b0b6e6d2e145a7870a13ed5b407b47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.706794 9903 tablet_service.cc:1552] Processing DeleteTablet for tablet 10a44aecfade42969454126b990eca24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.707032 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c400101d2a39490e83722500839198cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.708922 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e8c83b501d8426fa0cad6a89f9a0b9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:49.709086 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 4e8c83b501d8426fa0cad6a89f9a0b9b (DEFAULT_TABLE table=default.Table_4f6664b7160e4549a50589454ff10313 [id=67b71acdadc64445bb2492e5d424cb2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.712030 9901 tablet_service.cc:1505] Processing CreateTablet for tablet 533f34d4872444278477abcce89da9ec (DEFAULT_TABLE table=default.Table_a430283c1a4d4d8283ec004c3d705794 [id=439f48256cbd43f785f57857e83ae25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.722550 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5d5f327d728f4250841a9f4f10a2b533" } modify_external_catalogs: true
I20251027 21:07:49.722770 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5d5f327d728f4250841a9f4f10a2b533" } modify_external_catalogs: true
I20251027 21:07:49.725718 9903 tablet_service.cc:1505] Processing CreateTablet for tablet c1f99ca89f924b58b1416d2945a6ba40 (DEFAULT_TABLE table=default.Table_90157e3841104d92aafc5d82ffea13a9 [id=bc676ad23e1f4c4bae88752f31108ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.726075 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52d71f4976143e4a514350621ebe309. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.726275 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_546a5293975148e1841c8fff9eeac42a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:49.726544 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0c05919c1a104864a4e1cd182ae59e88" } modify_external_catalogs: true
I20251027 21:07:49.726684 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0c05919c1a104864a4e1cd182ae59e88" } modify_external_catalogs: true
W20251027 21:07:49.726744 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_546a5293975148e1841c8fff9eeac42a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.726852 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1f99ca89f924b58b1416d2945a6ba40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.726233 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_eb5c676ccdf040b2bf8822c15f54fb6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.728008 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb5c676ccdf040b2bf8822c15f54fb6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.728775 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e8c83b501d8426fa0cad6a89f9a0b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.729879 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_af921638ac16405c9835ce2b0446cec2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.730371 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af921638ac16405c9835ce2b0446cec2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.734385 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 9882b234e79940088be5f604c41576b7 (DEFAULT_TABLE table=default.Table_e0dee9209ca14dd98065b0384855d953 [id=92995507b7d2462f91b1a16434d3ac06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.734395 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe24af9838954a9bb680474800642e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.736481 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9882b234e79940088be5f604c41576b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.739776 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abc7458fcfb94985acf76d77c0292b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.741441 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f22f013b25074d4db9c7f13caf5b9b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.741946 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 132a01405a63447ab27b2137b803224f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.748776 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21a8fc4a69f3486e92b1a68d10faed38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.750074 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92a2393cff8044028c86c8cd2bccd930. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.750556 9901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533f34d4872444278477abcce89da9ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.751286 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b719dc40d7e348608cdfc78401c19824. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.754312 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5f2e60caa824b5db2330b423221873c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.766736 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 49b3d23798b740008c98cd0754ae33f4
I20251027 21:07:49.772677 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_750d3fcb86e54ef3bdf3519d0eace745" } new_table_name: "default.Table_11fbadd9eed14cdba7664715e389f7e5" modify_external_catalogs: true
I20251027 21:07:49.773283 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b8bc4fe8cb374bd990d2c777eae1d4f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:49.773469 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_fc9bf567c9ff4ddeaccaec2c6fddca99" } new_table_name: "default.Table_a90c80308cb345d7abc6c44d3d4c70ff" modify_external_catalogs: true
W20251027 21:07:49.773864 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8bc4fe8cb374bd990d2c777eae1d4f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.774197 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3f5e27698f4c4da88d9bacc9dd719304"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:49.774291 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7954e74d079b416cac73f9a77979c8b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.774755 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f5e27698f4c4da88d9bacc9dd719304 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.775017 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a46825cc888a4a0898d6db70b02e5ab6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.775478 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a46825cc888a4a0898d6db70b02e5ab6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:49.774777 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7954e74d079b416cac73f9a77979c8b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.790287 10041 ts_tablet_manager.cc:1929] T 84305bcf21b64d2c8c1df015a2e9615c P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:49.794620 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 87ec5a6b9b3c425da4e96cc6736d6310
I20251027 21:07:49.794896 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c04ff0f25ba04d25958346fc631b2802"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:49.795179 9897 tablet_service.cc:1505] Processing CreateTablet for tablet abd3b8e71d3041b4b9e5f6663841bfcf (DEFAULT_TABLE table=default.Table_bf6a0088d06247ca9eb20f7fbfb0f0e7 [id=f3e3147411e5488bae105ed755f5b50f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:49.795361 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c04ff0f25ba04d25958346fc631b2802 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.795288 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_1f82ff66f4a14bb9ac990665fd6ec38e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:49.795563 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abd3b8e71d3041b4b9e5f6663841bfcf. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:49.795773 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f82ff66f4a14bb9ac990665fd6ec38e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.796810 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78884c2be75f45f881f5424d3b3f4e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.802237 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44783760defd433a972c9cd83933249c" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:49.802870 10141 raft_consensus.cc:493] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:49.803054 10141 raft_consensus.cc:515] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.803829 10141 leader_election.cc:290] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:49.808645 9737 tablet_service.cc:1552] Processing DeleteTablet for tablet 84305bcf21b64d2c8c1df015a2e9615c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.815459 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_472e136ae79c4f73b52fd4b518db8663"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:49.816176 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_472e136ae79c4f73b52fd4b518db8663 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:49.820043 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet f5da8d7ecccc4c3492f9d0d30ffea423 (table default.Table_50cddb609f3043f29b42899ad521fbee [id=63988803b0384c0b9766dccecf60804a]) successfully deleted
W20251027 21:07:49.822304 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5f2e60caa824b5db2330b423221873c consensus state has no leader
I20251027 21:07:49.822242 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3c9f64ee0544e59add4f73bca59df98" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:49.823086 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ae124514964440c5b88b16ac9f8a2567 consensus state has no leader
I20251027 21:07:49.825315 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbada3946dab4739a8369032ff3b8a48" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:49.825438 9711 leader_election.cc:343] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 0029b70ed3ff4db9a816a6190d36bf88
I20251027 21:07:49.826315 9774 raft_consensus.cc:2468] T cbada3946dab4739a8369032ff3b8a48 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 0.
I20251027 21:07:49.827421 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1aa5f7f76cb4a1ba15f016942f17067. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.827435 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_790c0f90be104b8e8abf3491796d54bc" } new_table_name: "default.Table_40e7109b10b84bacaab337e0e52a63ef" modify_external_catalogs: true
I20251027 21:07:49.829460 9711 leader_election.cc:304] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:49.829623 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d9744e5483fd46a581cff3e8e787a6bd" } new_table_name: "default.Table_b0a525314a0d4e69aee8e2c4a291d999" modify_external_catalogs: true
W20251027 21:07:49.830506 9711 raft_consensus.cc:2686] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251027 21:07:49.830837 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_80ba3902da484ff8bc2100860960f098" } new_table_name: "default.Table_fc695aa989384c1ba355a09381428926" modify_external_catalogs: true
I20251027 21:07:49.830933 9711 leader_election.cc:304] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218; no voters: 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:49.831202 10141 raft_consensus.cc:2804] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:49.834245 9711 leader_election.cc:304] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218; no voters: 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:49.834288 10141 raft_consensus.cc:493] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:49.834427 10141 raft_consensus.cc:3060] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:49.836066 10049 raft_consensus.cc:2804] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:49.836229 10049 raft_consensus.cc:697] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:07:49.842461 9739 tablet_service.cc:1505] Processing CreateTablet for tablet a9339c6b1154400eace53d765528fdf9 (DEFAULT_TABLE table=default.Table_6df2aec5939c4b3cbfb82b376fe5b53d [id=91fb9f95fe9343fca1d8cb0d0f781afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.843251 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 6f1129eecfb54448965222269d9c32b7 (DEFAULT_TABLE table=default.Table_f16bc07fcb4c48b7969d13b8341aefdc [id=b5f4e2be1a5e42f58d98b64b8700ef1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.843550 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 9b04349b15f3494bb0933db56b7b24bc (DEFAULT_TABLE table=default.Table_2ff5a5b701c540b8a9eb8b87aace0e77 [id=69f10c0f3f3a4227ab4c35e939085634]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.844801 9752 tablet_service.cc:1505] Processing CreateTablet for tablet a287439ea6c9469da5fb56d84ee68048 (DEFAULT_TABLE table=default.Table_1ecaac632a024248b04e7c40e2c01d2c [id=4e2210c84e4c464da017d1b36ddb0a37]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.845383 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 21a8fc4a69f3486e92b1a68d10faed38 (DEFAULT_TABLE table=default.Table_ba1d973c05e14c70944d76ff1ecbc1bc [id=490e0e5ec83b4f76bf71eb41c900ac06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.843750 9737 tablet_service.cc:1552] Processing DeleteTablet for tablet 583fba7c750e4ac7b768da447002c211 with delete_type TABLET_DATA_DELETED (replaced by tablet f34512059f0c47fc9af7db2645f86b40) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:49.846395 10049 consensus_queue.cc:237] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.846678 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69da799c886841a19002d3c7c5084e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.847223 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 288b88e13fdf4427aae53f64b87733bf (DEFAULT_TABLE table=default.Table_a061b11818fc4a2db61e51a5972d69db [id=92ff9768853d403fb15db41e79fe77bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.847702 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 288b88e13fdf4427aae53f64b87733bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.849961 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a287439ea6c9469da5fb56d84ee68048. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.855043 10141 raft_consensus.cc:515] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.855888 10041 log.cc:1199] T 84305bcf21b64d2c8c1df015a2e9615c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/84305bcf21b64d2c8c1df015a2e9615c
I20251027 21:07:49.858789 10050 raft_consensus.cc:493] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:49.858970 10050 raft_consensus.cc:515] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.859791 10050 leader_election.cc:290] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:49.864892 9859 leader_election.cc:304] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218; no voters: 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:49.865033 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63662f175a1c450f8f67f37da971db37" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:49.868063 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet 49b3d23798b740008c98cd0754ae33f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:49.868374 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92a2393cff8044028c86c8cd2bccd930 consensus state has no leader
W20251027 21:07:49.868886 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9248f458baca409aa3505cc0db321b34 consensus state has no leader
I20251027 21:07:49.869316 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet 87ec5a6b9b3c425da4e96cc6736d6310 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:49.869947 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe24af9838954a9bb680474800642e62 consensus state has no leader
I20251027 21:07:49.855105 9750 tablet_service.cc:1505] Processing CreateTablet for tablet f22f013b25074d4db9c7f13caf5b9b42 (DEFAULT_TABLE table=default.Table_d601b22c11c34ebd8559009cd144affe [id=b96a43739ca54dac9e598e703186606b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.871498 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21a8fc4a69f3486e92b1a68d10faed38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.871526 9736 tablet_service.cc:1505] Processing CreateTablet for tablet c400101d2a39490e83722500839198cd (DEFAULT_TABLE table=default.Table_97b01c1840c543838263ca4222695f54 [id=bea51a58fda64d40916f306c4aa8ab4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.871942 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c400101d2a39490e83722500839198cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.875962 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f22f013b25074d4db9c7f13caf5b9b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.878266 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 92a2393cff8044028c86c8cd2bccd930 (DEFAULT_TABLE table=default.Table_790c0f90be104b8e8abf3491796d54bc [id=e5fba763a46b488cb19c738cee468ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.878671 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f1129eecfb54448965222269d9c32b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.878872 10050 raft_consensus.cc:2804] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:49.880776 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84305bcf21b64d2c8c1df015a2e9615c" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:49.881465 9924 raft_consensus.cc:2468] T 63662f175a1c450f8f67f37da971db37 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
W20251027 21:07:49.882687 9859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 46148.4us
I20251027 21:07:49.883054 9859 leader_election.cc:304] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218; no voters: 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:49.886927 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_dddd93d5fb3c4727b64ded3ff6cc3382" } modify_external_catalogs: true
I20251027 21:07:49.887151 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_dddd93d5fb3c4727b64ded3ff6cc3382" } modify_external_catalogs: true
I20251027 21:07:49.886562 10141 leader_election.cc:290] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:49.888887 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b0a525314a0d4e69aee8e2c4a291d999" } modify_external_catalogs: true
I20251027 21:07:49.888978 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b04349b15f3494bb0933db56b7b24bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.889075 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b0a525314a0d4e69aee8e2c4a291d999" } modify_external_catalogs: true
I20251027 21:07:49.890076 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9339c6b1154400eace53d765528fdf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.891076 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 343eaeca24dc4816ae032c7e674e9412 (DEFAULT_TABLE table=default.Table_546a5293975148e1841c8fff9eeac42a [id=b4569b8a023945989439915acb6097e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.891124 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_52c53bafe66b4c66b99e5b26e4df7ef5" } modify_external_catalogs: true
I20251027 21:07:49.891281 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_52c53bafe66b4c66b99e5b26e4df7ef5" } modify_external_catalogs: true
I20251027 21:07:49.892510 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 132a01405a63447ab27b2137b803224f (DEFAULT_TABLE table=default.Table_673503358b54481c8877034c26349796 [id=c6aed7584a0b49d89bf9a8f8ef0cb429]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.893448 9892 tablet_service.cc:1505] Processing CreateTablet for tablet f34512059f0c47fc9af7db2645f86b40 (DEFAULT_TABLE table=default.Table_2fd9ca76e8a0406cbfd32f91038ccc0e [id=14daa3e199fe4fa3ba2de06dd6be4330]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.893796 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f34512059f0c47fc9af7db2645f86b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.893342 10041 ts_tablet_manager.cc:1950] T 84305bcf21b64d2c8c1df015a2e9615c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:49.903066 10050 raft_consensus.cc:493] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:49.903236 10050 raft_consensus.cc:3060] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:49.912529 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 970fbf01a4c3406dafe1ea4cd005ea23
I20251027 21:07:49.914264 10041 tablet_replica.cc:333] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:49.892448 9923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8b125474bc42a5946e8f2dd718f7f4" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:49.918821 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92a2393cff8044028c86c8cd2bccd930. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.919276 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 9248f458baca409aa3505cc0db321b34
I20251027 21:07:49.923367 9890 tablet_service.cc:1505] Processing CreateTablet for tablet b8856c5dc8f144358445165324bfb477 (DEFAULT_TABLE table=default.Table_af921638ac16405c9835ce2b0446cec2 [id=5da6bd4463e24c03accd9985f72b6ebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.926097 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 343eaeca24dc4816ae032c7e674e9412. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.937938 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8856c5dc8f144358445165324bfb477. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:49.939707 9923 raft_consensus.cc:2468] T ba8b125474bc42a5946e8f2dd718f7f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:07:49.945649 10041 raft_consensus.cc:2243] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:49.956943 10041 raft_consensus.cc:2272] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:49.959009 10178 raft_consensus.cc:2804] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:49.959151 10178 raft_consensus.cc:493] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:49.962164 10041 ts_tablet_manager.cc:1916] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:49.960045 10050 raft_consensus.cc:515] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:49.964372 10178 raft_consensus.cc:3060] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:07:49.966095 9859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 74942.8us
W20251027 21:07:49.966459 9859 leader_election.cc:343] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:49.967073 9859 leader_election.cc:304] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:49.973153 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5d8e4122af254070a2df7016e521a937" } modify_external_catalogs: true
I20251027 21:07:49.973374 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5d8e4122af254070a2df7016e521a937" } modify_external_catalogs: true
I20251027 21:07:49.987949 10050 leader_election.cc:290] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:49.988266 10049 raft_consensus.cc:493] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:49.988921 10050 raft_consensus.cc:493] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:49.989105 10050 raft_consensus.cc:515] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:49.989912 10050 leader_election.cc:290] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:49.990087 10050 raft_consensus.cc:2749] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:49.996417 10040 ts_tablet_manager.cc:1929] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:49.996690 9740 tablet_service.cc:1505] Processing CreateTablet for tablet d8e008c868d04281beba0e5b204eb69a (DEFAULT_TABLE table=default.Table_04ef6b3afcc34d84804cd6e9613d7bc2 [id=8a1cda11e1b3436cb8ffa2616d543a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:49.997820 9711 leader_election.cc:304] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218; no voters: 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:49.998693 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 49d9e5c5960a4534a6603875006e6266
I20251027 21:07:50.009482 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 9cec6294051b41868c9c7c7fadafa138
W20251027 21:07:50.012773 9859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 46307.3us
I20251027 21:07:50.016655 9923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3dfef8b741c42f68b664d7e0eab49c6" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:07:50.017007 9923 raft_consensus.cc:2393] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 77b0513f17c0472088901a1ce8b08218 in current term 1: Already voted for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in this term.
W20251027 21:07:50.017999 9297 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 970fbf01a4c3406dafe1ea4cd005ea23 (table default.Table_dddd93d5fb3c4727b64ded3ff6cc3382 [id=741c9ce04c9d47d9a016bce1f4403ec7], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 970fbf01a4c3406dafe1ea4cd005ea23 consensus state has no leader
I20251027 21:07:50.018309 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 132a01405a63447ab27b2137b803224f (DEFAULT_TABLE table=default.Table_673503358b54481c8877034c26349796 [id=c6aed7584a0b49d89bf9a8f8ef0cb429]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.018776 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 132a01405a63447ab27b2137b803224f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.020049 10043 ts_tablet_manager.cc:1916] T 0029b70ed3ff4db9a816a6190d36bf88 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:50.021006 9603 tablet_service.cc:1505] Processing CreateTablet for tablet fe24af9838954a9bb680474800642e62 (DEFAULT_TABLE table=default.Table_80ba3902da484ff8bc2100860960f098 [id=dae1114f6b50443f8f6b79d18619d868]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.021662 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe24af9838954a9bb680474800642e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.024632 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_79211c9993c54e4ea0455d01fec68b8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.025261 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79211c9993c54e4ea0455d01fec68b8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.027908 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_79a3420c8b6542868231c740293ab1fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.028491 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79a3420c8b6542868231c740293ab1fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.031302 9609 tablet_service.cc:1505] Processing CreateTablet for tablet d52d71f4976143e4a514350621ebe309 (DEFAULT_TABLE table=default.Table_c305a5155cce4754bcbc792f3900b690 [id=82b0b6e6d2e145a7870a13ed5b407b47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.032004 9750 tablet_service.cc:1505] Processing CreateTablet for tablet d52d71f4976143e4a514350621ebe309 (DEFAULT_TABLE table=default.Table_c305a5155cce4754bcbc792f3900b690 [id=82b0b6e6d2e145a7870a13ed5b407b47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.032785 10049 raft_consensus.cc:515] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.034483 10040 log.cc:1199] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/0029b70ed3ff4db9a816a6190d36bf88
I20251027 21:07:50.034636 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fc2d7d6601d4cc2849064f4cdddc90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.039332 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_472e136ae79c4f73b52fd4b518db8663" } new_table_name: "default.Table_932e4079f74d40509aa6c2a1b8139d16" modify_external_catalogs: true
I20251027 21:07:50.041321 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2fd9ca76e8a0406cbfd32f91038ccc0e" } new_table_name: "default.Table_5dc2aa9f965e456dbf9b04d9032d4967" modify_external_catalogs: true
I20251027 21:07:50.042565 10178 raft_consensus.cc:515] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.042688 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2992b8e207024e06957ce7befb657acb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.043218 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2992b8e207024e06957ce7befb657acb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.044479 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9b06fbc2554c4780a754ed9cfb964fa9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.045007 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b06fbc2554c4780a754ed9cfb964fa9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.047489 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f16bc07fcb4c48b7969d13b8341aefdc" } modify_external_catalogs: true
I20251027 21:07:50.047711 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f16bc07fcb4c48b7969d13b8341aefdc" } modify_external_catalogs: true
I20251027 21:07:50.052243 10178 leader_election.cc:290] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:50.031157 9613 tablet_service.cc:1505] Processing CreateTablet for tablet 4e8c83b501d8426fa0cad6a89f9a0b9b (DEFAULT_TABLE table=default.Table_4f6664b7160e4549a50589454ff10313 [id=67b71acdadc64445bb2492e5d424cb2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.055894 10041 ts_tablet_manager.cc:1929] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:50.056035 10041 log.cc:1199] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/65fbcaa5447c48818fabfccac22c4048
I20251027 21:07:50.056123 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_17e67d6acbdd470d9b351522b7908579"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:50.056412 10041 ts_tablet_manager.cc:1950] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
W20251027 21:07:50.056701 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17e67d6acbdd470d9b351522b7908579 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.056994 10041 tablet_replica.cc:333] T 69b1fb42428748239d8939d98428d8fe P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:50.057206 10041 raft_consensus.cc:2243] T 69b1fb42428748239d8939d98428d8fe P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:50.057950 10041 raft_consensus.cc:2272] T 69b1fb42428748239d8939d98428d8fe P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.060026 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_ed32ba2bd26544e1a9165f6246ee5cb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.060572 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed32ba2bd26544e1a9165f6246ee5cb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.063408 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_8fa71eed37b040fcb5f63d829b4a6315"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.063934 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fa71eed37b040fcb5f63d829b4a6315 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.065232 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 84305bcf21b64d2c8c1df015a2e9615c (table default.Table_41a04598f12247fbac5f4103d2cbf22d [id=2e28a4d0f9204351b93335736a9abe18]) successfully deleted
I20251027 21:07:50.067340 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 65fbcaa5447c48818fabfccac22c4048 (table default.Table_c93601bc03f1438496c9610e8c23b5a6 [id=4369b0d36cc14f48a5f41f214aa4fe4c]) successfully deleted
I20251027 21:07:50.068126 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65fbcaa5447c48818fabfccac22c4048" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:50.066354 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 9248f458baca409aa3505cc0db321b34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:50.066570 9900 tablet_service.cc:1552] Processing DeleteTablet for tablet 49d9e5c5960a4534a6603875006e6266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:50.066777 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet 9cec6294051b41868c9c7c7fadafa138 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:50.067175 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 0a58cd07e1bb4ffdbf39e7e16a46022a (DEFAULT_TABLE table=default.Table_472e136ae79c4f73b52fd4b518db8663 [id=31172964ef544c5f99cade30a42b340d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.070020 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a58cd07e1bb4ffdbf39e7e16a46022a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.077350 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet b87e809ff897417c88e5edc19e2d591e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:50.077601 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 10a44aecfade42969454126b990eca24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:50.064860 10032 tablet_bootstrap.cc:492] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:50.077764 10049 leader_election.cc:290] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:50.080446 10032 ts_tablet_manager.cc:1403] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.438s user 0.006s sys 0.001s
I20251027 21:07:50.086284 10143 raft_consensus.cc:493] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:50.089746 10041 ts_tablet_manager.cc:1916] T 69b1fb42428748239d8939d98428d8fe P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:50.091478 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a58cd07e1bb4ffdbf39e7e16a46022a consensus state has no leader
I20251027 21:07:50.092151 10031 tablet_bootstrap.cc:492] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:50.092235 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8e008c868d04281beba0e5b204eb69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.092303 10031 ts_tablet_manager.cc:1403] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.544s user 0.007s sys 0.008s
I20251027 21:07:50.093102 10031 raft_consensus.cc:359] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.093281 10031 raft_consensus.cc:385] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.093387 10031 raft_consensus.cc:740] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.093621 10031 consensus_queue.cc:260] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.094029 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aa8c538b6c849a28a28ffe706306891. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.095700 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0968c2f25b64f6685f38a999ee9d60e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.096190 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bf9c2a78aac42528d67846dc58adc40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.096637 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed5619d295a246379b625f35692203e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.097098 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc589852710342f7bd017193e4075303. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.092113 10141 raft_consensus.cc:2804] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:50.098122 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e841c54a94d649c19b639f5021f150cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.098255 10141 raft_consensus.cc:493] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251027 21:07:50.099040 9859 leader_election.cc:343] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:50.099454 9859 leader_election.cc:304] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:50.099993 10179 raft_consensus.cc:2749] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:50.100888 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 970fbf01a4c3406dafe1ea4cd005ea23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:50.104693 9711 leader_election.cc:343] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Illegal state: Tablet not RUNNING: STOPPED
I20251027 21:07:50.105217 10040 ts_tablet_manager.cc:1950] T 0029b70ed3ff4db9a816a6190d36bf88 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:50.105216 10032 raft_consensus.cc:359] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.105415 10032 raft_consensus.cc:385] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.105484 10032 raft_consensus.cc:740] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.105686 10032 consensus_queue.cc:260] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.106267 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 132a01405a63447ab27b2137b803224f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.077730 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3dfef8b741c42f68b664d7e0eab49c6" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:07:50.107371 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c1a6a7616f24f69b83ec8a282746e92" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:50.108233 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e8c83b501d8426fa0cad6a89f9a0b9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:50.109318 9860 leader_election.cc:343] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:07:50.109551 9860 leader_election.cc:343] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:07:50.109755 9860 leader_election.cc:343] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:50.110867 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 533f34d4872444278477abcce89da9ec (DEFAULT_TABLE table=default.Table_a430283c1a4d4d8283ec004c3d705794 [id=439f48256cbd43f785f57857e83ae25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.104833 10143 raft_consensus.cc:515] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.112071 10032 ts_tablet_manager.cc:1434] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.031s user 0.003s sys 0.000s
I20251027 21:07:50.112576 9711 leader_election.cc:304] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:50.112900 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 4e8c83b501d8426fa0cad6a89f9a0b9b (DEFAULT_TABLE table=default.Table_4f6664b7160e4549a50589454ff10313 [id=67b71acdadc64445bb2492e5d424cb2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.112986 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 92a2393cff8044028c86c8cd2bccd930 (DEFAULT_TABLE table=default.Table_790c0f90be104b8e8abf3491796d54bc [id=e5fba763a46b488cb19c738cee468ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.113356 9610 tablet_service.cc:1505] Processing CreateTablet for tablet b719dc40d7e348608cdfc78401c19824 (DEFAULT_TABLE table=default.Table_41078339d0834d069db2e8ae485bcc89 [id=f641ffdf5fd7497096643ea3278c49d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.113796 9740 tablet_service.cc:1505] Processing CreateTablet for tablet fe24af9838954a9bb680474800642e62 (DEFAULT_TABLE table=default.Table_80ba3902da484ff8bc2100860960f098 [id=dae1114f6b50443f8f6b79d18619d868]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.114102 10141 raft_consensus.cc:3060] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:50.114465 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52d71f4976143e4a514350621ebe309. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.115242 10032 tablet_bootstrap.cc:492] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:50.115386 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e8c83b501d8426fa0cad6a89f9a0b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.117063 10049 raft_consensus.cc:493] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:50.117249 10049 raft_consensus.cc:515] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.117866 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 10a44aecfade42969454126b990eca24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:50.117825 9859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 86845us
I20251027 21:07:50.118347 10040 tablet_replica.cc:333] T 12aa6e3af0694429b45bef5af3592377 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:50.119403 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e8c83b501d8426fa0cad6a89f9a0b9b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:50.119690 9710 leader_election.cc:343] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: ba8b125474bc42a5946e8f2dd718f7f4
W20251027 21:07:50.120141 9710 leader_election.cc:343] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 5c87a5450d354506852c4b20bf0541d2
I20251027 21:07:50.120657 10040 raft_consensus.cc:2243] T 12aa6e3af0694429b45bef5af3592377 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:50.125434 10032 tablet_bootstrap.cc:654] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:50.118103 9859 leader_election.cc:304] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218; no voters: 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:50.125926 10179 raft_consensus.cc:2804] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:50.126092 10179 raft_consensus.cc:493] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:50.126323 10178 raft_consensus.cc:493] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:50.126462 10178 raft_consensus.cc:515] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.127135 10178 leader_election.cc:290] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:50.127336 10179 raft_consensus.cc:3060] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:50.121272 10143 leader_election.cc:290] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:50.128731 10143 raft_consensus.cc:2749] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:50.130645 10179 raft_consensus.cc:515] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.131348 10179 leader_election.cc:290] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:50.132838 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40bd202c1b834c949f9a5cee23b3486d" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:07:50.133035 9634 raft_consensus.cc:3060] T 40bd202c1b834c949f9a5cee23b3486d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:50.134629 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533f34d4872444278477abcce89da9ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.135463 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78a0a8b05f10477c923ab565839141b8" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:50.135794 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3dfef8b741c42f68b664d7e0eab49c6" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:07:50.136778 10041 ts_tablet_manager.cc:1929] T 69b1fb42428748239d8939d98428d8fe P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:50.136878 10041 log.cc:1199] T 69b1fb42428748239d8939d98428d8fe P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/69b1fb42428748239d8939d98428d8fe
I20251027 21:07:50.137290 10041 ts_tablet_manager.cc:1950] T 69b1fb42428748239d8939d98428d8fe P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:50.137315 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbada3946dab4739a8369032ff3b8a48" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:07:50.138031 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52d71f4976143e4a514350621ebe309. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:50.138319 9860 leader_election.cc:343] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:07:50.139137 9860 leader_election.cc:343] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 5c87a5450d354506852c4b20bf0541d2
I20251027 21:07:50.140883 10040 raft_consensus.cc:2272] T 12aa6e3af0694429b45bef5af3592377 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.140817 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet b87e809ff897417c88e5edc19e2d591e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:50.135735 9602 tablet_service.cc:1505] Processing CreateTablet for tablet c5f2e60caa824b5db2330b423221873c (DEFAULT_TABLE table=default.Table_fc9bf567c9ff4ddeaccaec2c6fddca99 [id=0e7e4ea125894e78a2162e9e8063ee33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.142748 10032 tablet_bootstrap.cc:492] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:50.142867 10032 ts_tablet_manager.cc:1403] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.002s
I20251027 21:07:50.135365 9598 tablet_service.cc:1505] Processing CreateTablet for tablet abc7458fcfb94985acf76d77c0292b08 (DEFAULT_TABLE table=default.Table_9c027ff744824da9b9cac5574017e7fd [id=40a6848701284864b34f727da289347b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.143435 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 288b88e13fdf4427aae53f64b87733bf (DEFAULT_TABLE table=default.Table_a061b11818fc4a2db61e51a5972d69db [id=92ff9768853d403fb15db41e79fe77bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.143501 10032 raft_consensus.cc:359] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.143648 10032 raft_consensus.cc:385] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.143719 10032 raft_consensus.cc:740] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
W20251027 21:07:50.145334 10108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 21:07:50.145701 10108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 21:07:50.145970 10108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251027 21:07:50.148324 10032 consensus_queue.cc:260] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.150821 9595 tablet_service.cc:1552] Processing DeleteTablet for tablet 49b3d23798b740008c98cd0754ae33f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:50.151154 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b719dc40d7e348608cdfc78401c19824. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.151249 10032 ts_tablet_manager.cc:1434] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20251027 21:07:50.151763 10041 tablet_replica.cc:333] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:50.151947 10041 raft_consensus.cc:2243] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:50.152079 10041 raft_consensus.cc:2272] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.152812 9774 raft_consensus.cc:1275] T 0ac7111d13b6460a9f4c77e94e09d417 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Refusing update from remote peer 51902b6b61ff4d5584ff21896e847e14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 21:07:50.153561 9608 tablet_service.cc:1505] Processing CreateTablet for tablet abd3b8e71d3041b4b9e5f6663841bfcf (DEFAULT_TABLE table=default.Table_bf6a0088d06247ca9eb20f7fbfb0f0e7 [id=f3e3147411e5488bae105ed755f5b50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.155108 10049 leader_election.cc:290] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:50.155975 10040 ts_tablet_manager.cc:1916] T 12aa6e3af0694429b45bef5af3592377 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:50.156635 9710 leader_election.cc:343] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:07:50.156844 9710 leader_election.cc:343] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:07:50.156999 9710 leader_election.cc:343] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: d642da56a67449aa95932180bc5b2c5f
I20251027 21:07:50.138447 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 9882b234e79940088be5f604c41576b7 (DEFAULT_TABLE table=default.Table_e0dee9209ca14dd98065b0384855d953 [id=92995507b7d2462f91b1a16434d3ac06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.157683 10032 tablet_bootstrap.cc:492] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:50.159686 10041 ts_tablet_manager.cc:1916] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:50.160019 10179 raft_consensus.cc:493] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:50.160346 10179 raft_consensus.cc:515] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.161237 10179 leader_election.cc:290] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:50.164780 9923 raft_consensus.cc:1275] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Refusing update from remote peer 51902b6b61ff4d5584ff21896e847e14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 21:07:50.167585 10032 tablet_bootstrap.cc:654] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:50.168010 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e8c83b501d8426fa0cad6a89f9a0b9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:50.169687 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 94f2dbf47a5c4458bee54b26f272156e (DEFAULT_TABLE table=default.Table_7954e74d079b416cac73f9a77979c8b3 [id=0103a75b0a714d3c9a533d94b6bda41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.170444 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94f2dbf47a5c4458bee54b26f272156e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.171602 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 3103c8e5a7c744be90be097179904ba3 (DEFAULT_TABLE table=default.Table_c04ff0f25ba04d25958346fc631b2802 [id=a6a32b47c2354a2d9535ff2b48d506de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.148777 9600 tablet_service.cc:1505] Processing CreateTablet for tablet c1f99ca89f924b58b1416d2945a6ba40 (DEFAULT_TABLE table=default.Table_90157e3841104d92aafc5d82ffea13a9 [id=bc676ad23e1f4c4bae88752f31108ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.171957 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3103c8e5a7c744be90be097179904ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.154011 9595 tablet_service.cc:1505] Processing CreateTablet for tablet f34512059f0c47fc9af7db2645f86b40 (DEFAULT_TABLE table=default.Table_2fd9ca76e8a0406cbfd32f91038ccc0e [id=14daa3e199fe4fa3ba2de06dd6be4330]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:50.175701 9711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 155572us
I20251027 21:07:50.176345 10029 tablet_bootstrap.cc:492] T 6e5d52b7ec114be5ae68e57280c2eea8 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:50.176390 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 288b88e13fdf4427aae53f64b87733bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.177271 10029 ts_tablet_manager.cc:1403] T 6e5d52b7ec114be5ae68e57280c2eea8 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.535s user 0.092s sys 0.176s
I20251027 21:07:50.177302 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet 87ec5a6b9b3c425da4e96cc6736d6310 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:50.177531 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet 9248f458baca409aa3505cc0db321b34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:50.178643 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet 49d9e5c5960a4534a6603875006e6266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:50.178843 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet 9cec6294051b41868c9c7c7fadafa138 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:50.179710 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 0a58cd07e1bb4ffdbf39e7e16a46022a (DEFAULT_TABLE table=default.Table_472e136ae79c4f73b52fd4b518db8663 [id=31172964ef544c5f99cade30a42b340d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.180949 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abd3b8e71d3041b4b9e5f6663841bfcf. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:50.182459 10108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 21:07:50.182557 10108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 21:07:50.182637 10108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 21:07:50.182696 10108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 21:07:50.187173 10089 consensus_queue.cc:1048] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20251027 21:07:50.189040 9615 tablet_service.cc:1505] Processing CreateTablet for tablet b8856c5dc8f144358445165324bfb477 (DEFAULT_TABLE table=default.Table_af921638ac16405c9835ce2b0446cec2 [id=5da6bd4463e24c03accd9985f72b6ebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.190399 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet 970fbf01a4c3406dafe1ea4cd005ea23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:50.190955 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9882b234e79940088be5f604c41576b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.186040 9612 tablet_service.cc:1505] Processing CreateTablet for tablet 343eaeca24dc4816ae032c7e674e9412 (DEFAULT_TABLE table=default.Table_546a5293975148e1841c8fff9eeac42a [id=b4569b8a023945989439915acb6097e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.198890 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 94f2dbf47a5c4458bee54b26f272156e (DEFAULT_TABLE table=default.Table_7954e74d079b416cac73f9a77979c8b3 [id=0103a75b0a714d3c9a533d94b6bda41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.200196 9595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f34512059f0c47fc9af7db2645f86b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.200480 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a58cd07e1bb4ffdbf39e7e16a46022a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.204675 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 343eaeca24dc4816ae032c7e674e9412. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.204883 10032 tablet_bootstrap.cc:492] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:50.204959 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1f99ca89f924b58b1416d2945a6ba40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.205050 10032 ts_tablet_manager.cc:1403] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20251027 21:07:50.205751 10032 raft_consensus.cc:359] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.205952 10032 raft_consensus.cc:385] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.206040 10032 raft_consensus.cc:740] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.206876 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe24af9838954a9bb680474800642e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.207638 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 533f34d4872444278477abcce89da9ec (DEFAULT_TABLE table=default.Table_a430283c1a4d4d8283ec004c3d705794 [id=439f48256cbd43f785f57857e83ae25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.208009 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533f34d4872444278477abcce89da9ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.208235 10032 consensus_queue.cc:260] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.212142 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6f1e2491cbc546a7b54ab25da6ebf334"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.212718 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f1e2491cbc546a7b54ab25da6ebf334 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.214236 9749 tablet_service.cc:1505] Processing CreateTablet for tablet c1f99ca89f924b58b1416d2945a6ba40 (DEFAULT_TABLE table=default.Table_90157e3841104d92aafc5d82ffea13a9 [id=bc676ad23e1f4c4bae88752f31108ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.214330 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_65dba19877524bfe9b252239a7cdcd37" } new_table_name: "default.Table_913666abca6044f9aacb48e76118aac5" modify_external_catalogs: true
I20251027 21:07:50.214450 9755 tablet_service.cc:1505] Processing CreateTablet for tablet c5f2e60caa824b5db2330b423221873c (DEFAULT_TABLE table=default.Table_fc9bf567c9ff4ddeaccaec2c6fddca99 [id=0e7e4ea125894e78a2162e9e8063ee33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.214792 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5f2e60caa824b5db2330b423221873c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.215118 10032 ts_tablet_manager.cc:1434] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251027 21:07:50.215458 9747 tablet_service.cc:1505] Processing CreateTablet for tablet b719dc40d7e348608cdfc78401c19824 (DEFAULT_TABLE table=default.Table_41078339d0834d069db2e8ae485bcc89 [id=f641ffdf5fd7497096643ea3278c49d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.215637 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_12da35feb05c42a79377049e12fe73d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.216535 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12da35feb05c42a79377049e12fe73d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.218936 9711 leader_election.cc:304] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218; no voters: 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:50.220934 10032 tablet_bootstrap.cc:492] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:50.225768 10032 tablet_bootstrap.cc:654] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:50.225953 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f34512059f0c47fc9af7db2645f86b40 consensus state has no leader
I20251027 21:07:50.228153 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet b1bdb2be5032415187c2826c14e75f07 with delete_type TABLET_DATA_DELETED (replaced by tablet f1aa5f7f76cb4a1ba15f016942f17067) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:50.228657 9296 consensus_peers.cc:597] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 -> Peer 6e86ca5e83dd4df5b72f045e7b787e81 (127.9.14.124:45071): Couldn't send request to peer 6e86ca5e83dd4df5b72f045e7b787e81. Status: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251027 21:07:50.232354 10041 ts_tablet_manager.cc:1929] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:50.232463 10041 log.cc:1199] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/0c1a6a7616f24f69b83ec8a282746e92
I20251027 21:07:50.233116 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 with delete_type TABLET_DATA_DELETED (replaced by tablet e07b8aa0a41241d79254c4dae8738e79) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:50.234499 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a430283c1a4d4d8283ec004c3d705794" } new_table_name: "default.Table_61bbf6b5c1e740128b80465ef3b9a50e" modify_external_catalogs: true
I20251027 21:07:50.235060 10041 ts_tablet_manager.cc:1950] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:50.235605 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 150) took 1002 ms. Trace:
I20251027 21:07:50.235751 10041 rpcz_store.cc:276] 1027 21:07:49.233497 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.233552 (+ 55us) service_pool.cc:225] Handling call
1027 21:07:50.151709 (+918157us) ts_tablet_manager.cc:1151] Deleting tablet 0c1a6a7616f24f69b83ec8a282746e92
1027 21:07:50.151718 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.194867 (+ 43149us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.232254 (+ 37387us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.235545 (+ 3291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":917950}
I20251027 21:07:50.237138 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 0c1a6a7616f24f69b83ec8a282746e92 (table default.Table_908c9569154d403eb8aeab2a65da7f67 [id=af6cb904f8cd4a929be7a978f9bc2001]) successfully deleted
I20251027 21:07:50.237409 10032 tablet_bootstrap.cc:492] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:50.236088 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1e7bc61476164b5bbcad34206a7d7d8f" } new_table_name: "default.Table_c19fea3933b84e87ad91298bdb880696" modify_external_catalogs: true
I20251027 21:07:50.237568 10032 ts_tablet_manager.cc:1403] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20251027 21:07:50.238371 10032 raft_consensus.cc:359] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.238531 10032 raft_consensus.cc:385] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.238620 10032 raft_consensus.cc:740] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.238696 10108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.14.84:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.14.126:34057,127.9.14.125:46313,127.9.14.124:45071
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.14.124:45071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.14.124
--webserver_port=41907
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 27 Oct 2025 20:43:25 UTC on 24a791456cd2
build id 8748
ASAN enabled
W20251027 21:07:50.239240 9604 consensus_meta.cc:287] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14: Time spent flushing consensus metadata: real 0.516s user 0.007s sys 0.001s
I20251027 21:07:50.241223 10089 consensus_queue.cc:1048] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251027 21:07:50.241287 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1bdb2be5032415187c2826c14e75f07" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:50.242012 9860 leader_election.cc:343] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:50.242228 10108 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 21:07:50.242785 10041 tablet_replica.cc:333] T 63662f175a1c450f8f67f37da971db37 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:50.243181 10041 raft_consensus.cc:2243] T 63662f175a1c450f8f67f37da971db37 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:50.243587 10041 raft_consensus.cc:2272] T 63662f175a1c450f8f67f37da971db37 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.242743 10029 raft_consensus.cc:359] T 6e5d52b7ec114be5ae68e57280c2eea8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.244303 10041 ts_tablet_manager.cc:1916] T 63662f175a1c450f8f67f37da971db37 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:50.244724 10108 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251027 21:07:50.246495 10032 consensus_queue.cc:260] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.248862 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94f2dbf47a5c4458bee54b26f272156e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.249563 10029 raft_consensus.cc:385] T 6e5d52b7ec114be5ae68e57280c2eea8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.250353 10179 raft_consensus.cc:493] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:50.250495 10179 raft_consensus.cc:515] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.250947 10089 consensus_queue.cc:1048] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251027 21:07:50.251058 10179 leader_election.cc:290] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:50.253566 9634 raft_consensus.cc:2468] T 40bd202c1b834c949f9a5cee23b3486d P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 1.
I20251027 21:07:50.258637 9860 leader_election.cc:304] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:07:50.260771 10179 raft_consensus.cc:2804] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:50.260937 10179 raft_consensus.cc:697] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Becoming Leader. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Running, Role: LEADER
I20251027 21:07:50.261332 10179 consensus_queue.cc:237] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.273196 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1f99ca89f924b58b1416d2945a6ba40. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:50.273154 10197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 21:07:50.274492 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2f29d30ca47244898036f6030063a5c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.275029 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f29d30ca47244898036f6030063a5c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.275909 9753 tablet_service.cc:1505] Processing CreateTablet for tablet abc7458fcfb94985acf76d77c0292b08 (DEFAULT_TABLE table=default.Table_9c027ff744824da9b9cac5574017e7fd [id=40a6848701284864b34f727da289347b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.276515 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5d8837ff3c44a66a1f933d80540ba0c" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:50.278297 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 6f1129eecfb54448965222269d9c32b7
I20251027 21:07:50.279027 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2d758ba690f8478ba7f983e8fd770219" } modify_external_catalogs: true
I20251027 21:07:50.279202 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2d758ba690f8478ba7f983e8fd770219" } modify_external_catalogs: true
I20251027 21:07:50.280014 10032 ts_tablet_manager.cc:1434] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20251027 21:07:50.280347 10032 tablet_bootstrap.cc:492] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:07:50.280238 10193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 21:07:50.283362 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c4369a10bf846fb94c9b71a70f21600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:50.283552 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f1129eecfb54448965222269d9c32b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:50.283751 9860 leader_election.cc:343] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:50.285181 10029 raft_consensus.cc:740] T 6e5d52b7ec114be5ae68e57280c2eea8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.286751 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abc7458fcfb94985acf76d77c0292b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.288275 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92a2393cff8044028c86c8cd2bccd930. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.289165 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8856c5dc8f144358445165324bfb477. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.289640 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abc7458fcfb94985acf76d77c0292b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.295774 10029 consensus_queue.cc:260] T 6e5d52b7ec114be5ae68e57280c2eea8 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.285029 9604 tablet_service.cc:1505] Processing CreateTablet for tablet 3103c8e5a7c744be90be097179904ba3 (DEFAULT_TABLE table=default.Table_c04ff0f25ba04d25958346fc631b2802 [id=a6a32b47c2354a2d9535ff2b48d506de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.297919 10049 raft_consensus.cc:2804] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:50.308626 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5f2e60caa824b5db2330b423221873c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:50.311252 9711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 110622us
W20251027 21:07:50.311329 10201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 21:07:50.313624 10049 raft_consensus.cc:493] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:50.313763 10049 raft_consensus.cc:3060] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:50.315981 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f1129eecfb54448965222269d9c32b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:50.318308 9611 tablet_service.cc:1505] Processing CreateTablet for tablet adaa4f66bbab491f8030d439d85cc7d1 (DEFAULT_TABLE table=default.Table_3f5e27698f4c4da88d9bacc9dd719304 [id=d174e821ea8641fc8b6658d5c61c7e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.320423 10032 tablet_bootstrap.cc:654] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:50.322086 9893 tablet_service.cc:1505] Processing CreateTablet for tablet adaa4f66bbab491f8030d439d85cc7d1 (DEFAULT_TABLE table=default.Table_3f5e27698f4c4da88d9bacc9dd719304 [id=d174e821ea8641fc8b6658d5c61c7e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.322567 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adaa4f66bbab491f8030d439d85cc7d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.327746 9711 leader_election.cc:304] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:50.328575 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3103c8e5a7c744be90be097179904ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.329463 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adaa4f66bbab491f8030d439d85cc7d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.332379 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 0029b70ed3ff4db9a816a6190d36bf88 (table default.Table_450e3d88ee8f4a328418b279a354b9e2 [id=0f311c7d59a4471fbad8c2a0c59430f0]) successfully deleted
I20251027 21:07:50.333472 9923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c87a5450d354506852c4b20bf0541d2" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:50.333711 9923 raft_consensus.cc:2468] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:07:50.335291 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b719dc40d7e348608cdfc78401c19824. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:50.336185 9710 consensus_peers.cc:597] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 -> Peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Couldn't send request to peer 51902b6b61ff4d5584ff21896e847e14. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:07:50.337255 10143 raft_consensus.cc:2749] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251027 21:07:50.337453 10143 raft_consensus.cc:493] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:50.337597 10143 raft_consensus.cc:515] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.337627 9306 catalog_manager.cc:5649] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67). New cstate: current_term: 1 leader_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: HEALTHY } } }
I20251027 21:07:50.337988 9306 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 40bd202c1b834c949f9a5cee23b3486d (table default.Table_f9b456b781864738a6fa0d5c4bfa50fd [id=fd2beefa0629466daea4da402b033347]). Expected version 1 got 0
I20251027 21:07:50.338196 9306 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 0ac7111d13b6460a9f4c77e94e09d417 (table default.Table_e65d5d66ce8747a9a1a2abfa04cd2855 [id=e2b68fb34c3b48e8952ef6d4201fd93f]). Expected version 1 got 0
I20251027 21:07:50.338312 10143 leader_election.cc:290] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:50.338658 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65fbcaa5447c48818fabfccac22c4048" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:50.338869 9773 raft_consensus.cc:2468] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 0.
I20251027 21:07:50.339071 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3dfef8b741c42f68b664d7e0eab49c6" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:50.339308 9775 raft_consensus.cc:2393] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fbd567f929a4f6d8e4d78feef7ed678 in current term 1: Already voted for candidate 77b0513f17c0472088901a1ce8b08218 in this term.
I20251027 21:07:50.340685 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_68f478946b1543a68842ffad9e14864f" } modify_external_catalogs: true
I20251027 21:07:50.340852 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_68f478946b1543a68842ffad9e14864f" } modify_external_catalogs: true
I20251027 21:07:50.343674 10141 raft_consensus.cc:515] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.344458 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e5d52b7ec114be5ae68e57280c2eea8" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:50.345504 9893 tablet_service.cc:1505] Processing CreateTablet for tablet c4996d8fae094db8a96599b9057f1649 (DEFAULT_TABLE table=default.Table_b8bc4fe8cb374bd990d2c777eae1d4f0 [id=68da6285e1d247f1950a0b4929c885a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.346002 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4996d8fae094db8a96599b9057f1649. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.345366 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 9882b234e79940088be5f604c41576b7 (DEFAULT_TABLE table=default.Table_e0dee9209ca14dd98065b0384855d953 [id=92995507b7d2462f91b1a16434d3ac06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.348691 9305 master_service.cc:954] ReplaceTablet: received request to replace tablet 94e94cd50c91429d8da9804149501c86 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:50.349097 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbada3946dab4739a8369032ff3b8a48" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:50.349268 9775 raft_consensus.cc:3060] T cbada3946dab4739a8369032ff3b8a48 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:50.349749 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1bdb2be5032415187c2826c14e75f07" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:50.355250 10108 server_base.cc:1047] running on GCE node
I20251027 21:07:50.350019 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c87a5450d354506852c4b20bf0541d2" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:50.356120 9770 raft_consensus.cc:2468] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 0.
I20251027 21:07:50.357343 10041 ts_tablet_manager.cc:1929] T 63662f175a1c450f8f67f37da971db37 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:50.357468 10041 log.cc:1199] T 63662f175a1c450f8f67f37da971db37 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/63662f175a1c450f8f67f37da971db37
I20251027 21:07:50.357574 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9882b234e79940088be5f604c41576b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.357811 10041 ts_tablet_manager.cc:1950] T 63662f175a1c450f8f67f37da971db37 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:50.357843 10108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 21:07:50.353705 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c1a6a7616f24f69b83ec8a282746e92" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:50.361064 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet 5c87a5450d354506852c4b20bf0541d2 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:50.361598 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet cad3a3e910ac42399cf371ad82c89ff7 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:50.362059 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e0dee9209ca14dd98065b0384855d953" } modify_external_catalogs: true
I20251027 21:07:50.362246 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e0dee9209ca14dd98065b0384855d953" } modify_external_catalogs: true
I20251027 21:07:50.340845 10032 tablet_bootstrap.cc:492] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:50.363301 10032 ts_tablet_manager.cc:1403] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.083s user 0.004s sys 0.002s
I20251027 21:07:50.367990 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5d8837ff3c44a66a1f933d80540ba0c" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:50.368692 10141 leader_election.cc:290] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:50.370311 10108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20251027 21:07:50.371105 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 533f231e3ff64531ae2331409a961425 consensus state has no leader
I20251027 21:07:50.372318 10032 raft_consensus.cc:359] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:50.372555 10032 raft_consensus.cc:385] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.372664 10032 raft_consensus.cc:740] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.372896 10032 consensus_queue.cc:260] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:50.373601 10108 hybrid_clock.cc:648] HybridClock initialized: now 1761599270373563 us; error 32 us; skew 500 ppm
I20251027 21:07:50.373646 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b23313ec4d7b4e32a4283dd404578616" } modify_external_catalogs: true
I20251027 21:07:50.373795 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b23313ec4d7b4e32a4283dd404578616" } modify_external_catalogs: true
I20251027 21:07:50.374096 10108 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 21:07:50.374329 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8c58943248a74b5ab81a5b742eafc5b4" } new_table_name: "default.Table_8d288192611b43af971f1e081bb362ad" modify_external_catalogs: true
I20251027 21:07:50.374405 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 151) took 1134 ms. Trace:
I20251027 21:07:50.375598 10041 rpcz_store.cc:276] 1027 21:07:49.239700 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.239754 (+ 54us) service_pool.cc:225] Handling call
1027 21:07:50.242711 (+1002957us) ts_tablet_manager.cc:1151] Deleting tablet 63662f175a1c450f8f67f37da971db37
1027 21:07:50.242722 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.316073 (+ 73351us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.357225 (+ 41152us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.374356 (+ 17131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1002758}
I20251027 21:07:50.375949 10041 tablet_replica.cc:333] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:50.376003 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 63662f175a1c450f8f67f37da971db37 (table default.Table_d7caf208b9864991a0e50ced803eb60d [id=4142b5bb00004c83afe20e6982409849]) successfully deleted
I20251027 21:07:50.376165 10041 raft_consensus.cc:2243] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:50.378595 10032 ts_tablet_manager.cc:1434] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
W20251027 21:07:50.379704 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 533f34d4872444278477abcce89da9ec consensus state has no leader
I20251027 21:07:50.379738 10041 raft_consensus.cc:2272] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.380354 10041 ts_tablet_manager.cc:1916] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:50.381532 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8fc2d7d6601d4cc2849064f4cdddc90d consensus state has no leader
I20251027 21:07:50.383781 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_94d4f5b9325a4c27a6305962fe774808"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.384277 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94d4f5b9325a4c27a6305962fe774808 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.385541 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c73837b0d9514a8980a8a70c61ae8019"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.386102 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c73837b0d9514a8980a8a70c61ae8019 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.386750 10032 tablet_bootstrap.cc:492] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:50.362262 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40bd202c1b834c949f9a5cee23b3486d" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:50.387727 9771 raft_consensus.cc:3060] T 40bd202c1b834c949f9a5cee23b3486d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:50.388799 10032 tablet_bootstrap.cc:654] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:50.402374 10032 tablet_bootstrap.cc:492] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:50.402530 10032 ts_tablet_manager.cc:1403] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20251027 21:07:50.403359 10032 raft_consensus.cc:359] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.403564 10032 raft_consensus.cc:385] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.403663 10032 raft_consensus.cc:740] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.403878 10032 consensus_queue.cc:260] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.408514 9738 tablet_service.cc:1505] Processing CreateTablet for tablet abd3b8e71d3041b4b9e5f6663841bfcf (DEFAULT_TABLE table=default.Table_bf6a0088d06247ca9eb20f7fbfb0f0e7 [id=f3e3147411e5488bae105ed755f5b50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.411800 10108 webserver.cc:492] Webserver started at http://127.9.14.124:41907/ using document root <none> and password file <none>
I20251027 21:07:50.412803 10108 fs_manager.cc:362] Metadata directory not provided
I20251027 21:07:50.412930 10108 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 21:07:50.417876 10032 ts_tablet_manager.cc:1434] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251027 21:07:50.419371 10032 tablet_bootstrap.cc:492] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:50.420951 10032 tablet_bootstrap.cc:654] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:50.423478 9739 tablet_service.cc:1505] Processing CreateTablet for tablet 288b88e13fdf4427aae53f64b87733bf (DEFAULT_TABLE table=default.Table_a061b11818fc4a2db61e51a5972d69db [id=92ff9768853d403fb15db41e79fe77bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.423846 10041 ts_tablet_manager.cc:1929] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:50.423972 10041 log.cc:1199] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/b3dfef8b741c42f68b664d7e0eab49c6
I20251027 21:07:50.424312 10041 ts_tablet_manager.cc:1950] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:50.424793 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 152) took 1100 ms. Trace:
I20251027 21:07:50.424896 10041 rpcz_store.cc:276] 1027 21:07:49.324736 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.324794 (+ 58us) service_pool.cc:225] Handling call
1027 21:07:50.375884 (+1051090us) ts_tablet_manager.cc:1151] Deleting tablet b3dfef8b741c42f68b664d7e0eab49c6
1027 21:07:50.375893 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.383968 (+ 8075us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.423747 (+ 39779us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.424734 (+ 987us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3383,"tablet-delete.queue_time_us":1050885}
I20251027 21:07:50.425413 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abd3b8e71d3041b4b9e5f6663841bfcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.425380 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet b3dfef8b741c42f68b664d7e0eab49c6 (table default.Table_be23b67279484b3db60ef67b1b157253 [id=9e10359cf7a640e9b16f01096987ca59]) successfully deleted
I20251027 21:07:50.425473 10041 tablet_replica.cc:333] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:50.425707 10041 raft_consensus.cc:2243] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:50.425916 10041 raft_consensus.cc:2272] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.426523 10041 ts_tablet_manager.cc:1916] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:50.428844 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f9b456b781864738a6fa0d5c4bfa50fd" } new_table_name: "default.Table_7572ec58e5f347bfaf585d1fd0266335" modify_external_catalogs: true
I20251027 21:07:50.432771 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 9c239216cb8846f0b1aaa360ddb83321
I20251027 21:07:50.433926 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c239216cb8846f0b1aaa360ddb83321 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:50.434731 10049 raft_consensus.cc:515] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.435876 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_60b6845211be4b7798d0690248d2b718"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.436403 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60b6845211be4b7798d0690248d2b718 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.439258 10032 tablet_bootstrap.cc:492] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:50.439455 10032 ts_tablet_manager.cc:1403] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.001s
I20251027 21:07:50.440340 10032 raft_consensus.cc:359] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.440585 10032 raft_consensus.cc:385] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.440671 10032 raft_consensus.cc:740] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.441089 10049 leader_election.cc:290] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:50.444455 10108 fs_manager.cc:714] Time spent opening directory manager: real 0.028s user 0.001s sys 0.008s
I20251027 21:07:50.446203 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet 49b3d23798b740008c98cd0754ae33f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:50.446722 9751 tablet_service.cc:1505] Processing CreateTablet for tablet f34512059f0c47fc9af7db2645f86b40 (DEFAULT_TABLE table=default.Table_2fd9ca76e8a0406cbfd32f91038ccc0e [id=14daa3e199fe4fa3ba2de06dd6be4330]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.447180 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f34512059f0c47fc9af7db2645f86b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.448561 9746 tablet_service.cc:1552] Processing DeleteTablet for tablet 87ec5a6b9b3c425da4e96cc6736d6310 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:50.451253 10032 consensus_queue.cc:260] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.451876 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 343eaeca24dc4816ae032c7e674e9412 (DEFAULT_TABLE table=default.Table_546a5293975148e1841c8fff9eeac42a [id=b4569b8a023945989439915acb6097e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.452385 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 343eaeca24dc4816ae032c7e674e9412. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.453289 10032 ts_tablet_manager.cc:1434] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20251027 21:07:50.454725 10032 tablet_bootstrap.cc:492] T cad3a3e910ac42399cf371ad82c89ff7 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:50.457821 9743 tablet_service.cc:1505] Processing CreateTablet for tablet b8856c5dc8f144358445165324bfb477 (DEFAULT_TABLE table=default.Table_af921638ac16405c9835ce2b0446cec2 [id=5da6bd4463e24c03accd9985f72b6ebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.461486 10032 tablet_bootstrap.cc:654] T cad3a3e910ac42399cf371ad82c89ff7 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:50.462393 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet 970fbf01a4c3406dafe1ea4cd005ea23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:50.462814 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet 9248f458baca409aa3505cc0db321b34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:50.464183 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet 49d9e5c5960a4534a6603875006e6266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:50.464421 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet 9cec6294051b41868c9c7c7fadafa138 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:50.464849 9744 tablet_service.cc:1505] Processing CreateTablet for tablet 3103c8e5a7c744be90be097179904ba3 (DEFAULT_TABLE table=default.Table_c04ff0f25ba04d25958346fc631b2802 [id=a6a32b47c2354a2d9535ff2b48d506de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.465797 10214 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 21:07:50.468976 10108 fs_manager.cc:730] Time spent opening block manager: real 0.022s user 0.003s sys 0.002s
I20251027 21:07:50.471288 10041 ts_tablet_manager.cc:1929] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:50.471524 10041 log.cc:1199] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/e3c9f64ee0544e59add4f73bca59df98
I20251027 21:07:50.472247 10041 ts_tablet_manager.cc:1950] T e3c9f64ee0544e59add4f73bca59df98 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:50.472268 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c239216cb8846f0b1aaa360ddb83321 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:50.469554 10108 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data,/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/wal
uuid: "6e86ca5e83dd4df5b72f045e7b787e81"
format_stamp: "Formatted at 2025-10-27 21:07:44 on dist-test-slave-r942"
I20251027 21:07:50.473771 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8856c5dc8f144358445165324bfb477. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.473846 9631 raft_consensus.cc:2468] T 6e5d52b7ec114be5ae68e57280c2eea8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:07:50.473702 10108 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 21:07:50.482838 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 0a58cd07e1bb4ffdbf39e7e16a46022a (DEFAULT_TABLE table=default.Table_472e136ae79c4f73b52fd4b518db8663 [id=31172964ef544c5f99cade30a42b340d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.486318 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 288b88e13fdf4427aae53f64b87733bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.487987 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 45568792d7954b498581ce69fb611948 (DEFAULT_TABLE table=default.Table_1f82ff66f4a14bb9ac990665fd6ec38e [id=2eb093cd58e643a58b14e564d5914fbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.488372 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45568792d7954b498581ce69fb611948. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.489174 9612 tablet_service.cc:1505] Processing CreateTablet for tablet c4996d8fae094db8a96599b9057f1649 (DEFAULT_TABLE table=default.Table_b8bc4fe8cb374bd990d2c777eae1d4f0 [id=68da6285e1d247f1950a0b4929c885a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.489508 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4996d8fae094db8a96599b9057f1649. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.490271 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a58cd07e1bb4ffdbf39e7e16a46022a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:50.490792 9710 leader_election.cc:343] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 63662f175a1c450f8f67f37da971db37
I20251027 21:07:50.491020 9710 leader_election.cc:304] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
W20251027 21:07:50.492344 9710 leader_election.cc:343] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: ba8b125474bc42a5946e8f2dd718f7f4
I20251027 21:07:50.494295 10182 raft_consensus.cc:2804] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:50.494390 10182 raft_consensus.cc:493] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:50.494568 10182 raft_consensus.cc:3060] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:50.497030 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3103c8e5a7c744be90be097179904ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.510829 9771 raft_consensus.cc:2468] T 40bd202c1b834c949f9a5cee23b3486d P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 1.
I20251027 21:07:50.518733 10202 tablet.cc:1722] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:07:50.530066 9754 tablet_service.cc:1505] Processing CreateTablet for tablet adaa4f66bbab491f8030d439d85cc7d1 (DEFAULT_TABLE table=default.Table_3f5e27698f4c4da88d9bacc9dd719304 [id=d174e821ea8641fc8b6658d5c61c7e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.499938 10198 consensus_queue.cc:1048] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.059s
I20251027 21:07:50.472426 9925 raft_consensus.cc:1275] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Refusing update from remote peer 77b0513f17c0472088901a1ce8b08218: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 21:07:50.471359 9859 leader_election.cc:304] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218; no voters: 51902b6b61ff4d5584ff21896e847e14
W20251027 21:07:50.543659 9859 raft_consensus.cc:2686] T 65fbcaa5447c48818fabfccac22c4048 P 0fbd567f929a4f6d8e4d78feef7ed678: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251027 21:07:50.544096 9859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 72747.9us
I20251027 21:07:50.544348 9859 leader_election.cc:304] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:07:50.544559 9859 raft_consensus.cc:2686] T b3dfef8b741c42f68b664d7e0eab49c6 P 0fbd567f929a4f6d8e4d78feef7ed678: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251027 21:07:50.545644 9859 leader_election.cc:343] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:50.545780 9859 leader_election.cc:304] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:50.546042 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 154) took 1163 ms. Trace:
I20251027 21:07:50.473042 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8b125474bc42a5946e8f2dd718f7f4" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:07:50.546247 9924 raft_consensus.cc:3060] T ba8b125474bc42a5946e8f2dd718f7f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:50.546331 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e0968c2f25b64f6685f38a999ee9d60e
I20251027 21:07:50.547031 10032 tablet_bootstrap.cc:492] T cad3a3e910ac42399cf371ad82c89ff7 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:50.547156 10032 ts_tablet_manager.cc:1403] T cad3a3e910ac42399cf371ad82c89ff7 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.093s user 0.005s sys 0.000s
I20251027 21:07:50.547299 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 94e94cd50c91429d8da9804149501c86
I20251027 21:07:50.547708 9305 catalog_manager.cc:6719] ReplaceTablet: tablet 94e94cd50c91429d8da9804149501c86 deleted and replaced by tablet 26872be132fb46ee9e79a9c2e62374ef
I20251027 21:07:50.547936 10032 raft_consensus.cc:359] T cad3a3e910ac42399cf371ad82c89ff7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.548105 10032 raft_consensus.cc:385] T cad3a3e910ac42399cf371ad82c89ff7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.548188 10032 raft_consensus.cc:740] T cad3a3e910ac42399cf371ad82c89ff7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.548264 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 9882b234e79940088be5f604c41576b7
I20251027 21:07:50.548373 10032 consensus_queue.cc:260] T cad3a3e910ac42399cf371ad82c89ff7 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.548774 10032 ts_tablet_manager.cc:1434] T cad3a3e910ac42399cf371ad82c89ff7 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:07:50.549041 10032 tablet_bootstrap.cc:492] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:50.549105 9924 raft_consensus.cc:2468] T ba8b125474bc42a5946e8f2dd718f7f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:07:50.549535 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 648f506ec87c4fb8a3f6d4591873dbb8 (DEFAULT_TABLE table=default.Table_a46825cc888a4a0898d6db70b02e5ab6 [id=8579d3ff46d64c75b1da213fdee9cc1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.549767 10178 raft_consensus.cc:2749] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:50.549959 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648f506ec87c4fb8a3f6d4591873dbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.550750 10032 tablet_bootstrap.cc:654] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:50.546159 10041 rpcz_store.cc:276] 1027 21:07:49.380406 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.380454 (+ 48us) service_pool.cc:225] Handling call
1027 21:07:50.425364 (+1044910us) ts_tablet_manager.cc:1151] Deleting tablet e3c9f64ee0544e59add4f73bca59df98
1027 21:07:50.425376 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.430071 (+ 4695us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.471162 (+ 41091us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.544018 (+ 72856us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1044683}
I20251027 21:07:50.551821 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet e3c9f64ee0544e59add4f73bca59df98 (table default.Table_79febe3afdb94e8781e2633f126402dc [id=0e1ad615b4494a6dbb8c1390c1b8416f]) successfully deleted
I20251027 21:07:50.552229 10041 tablet_replica.cc:333] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:50.553220 9904 tablet_service.cc:1505] Processing CreateTablet for tablet a9bf3a093be8498c9dd5511b53436b16 (DEFAULT_TABLE table=default.Table_eb5c676ccdf040b2bf8822c15f54fb6f [id=5c9fd870185a4211a5a956171f491cc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.553516 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9bf3a093be8498c9dd5511b53436b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.499224 10194 mvcc.cc:204] Tried to move back new op lower bound from 7215510610206552064 to 7215510607026130944. Current Snapshot: MvccSnapshot[applied={T|T < 7215510610206552064}]
I20251027 21:07:50.557207 9900 tablet_service.cc:1552] Processing DeleteTablet for tablet e0968c2f25b64f6685f38a999ee9d60e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:50.557448 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 94e94cd50c91429d8da9804149501c86 with delete_type TABLET_DATA_DELETED (replaced by tablet 26872be132fb46ee9e79a9c2e62374ef) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:50.557672 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet 9882b234e79940088be5f604c41576b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:50.518563 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 45568792d7954b498581ce69fb611948 (DEFAULT_TABLE table=default.Table_1f82ff66f4a14bb9ac990665fd6ec38e [id=2eb093cd58e643a58b14e564d5914fbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.558099 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45568792d7954b498581ce69fb611948. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.558933 10041 raft_consensus.cc:2243] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:50.559113 10041 raft_consensus.cc:2272] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.559672 10041 ts_tablet_manager.cc:1916] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:50.565485 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet efc0019840244d14bb95ae581793e74b consensus state has no leader
I20251027 21:07:50.565730 10217 tablet.cc:1722] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251027 21:07:50.566217 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 205427us
I20251027 21:07:50.569007 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 648f506ec87c4fb8a3f6d4591873dbb8 (DEFAULT_TABLE table=default.Table_a46825cc888a4a0898d6db70b02e5ab6 [id=8579d3ff46d64c75b1da213fdee9cc1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.571187 9711 leader_election.cc:304] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218; no voters: 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:50.575830 9859 leader_election.cc:304] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218; no voters: 51902b6b61ff4d5584ff21896e847e14
W20251027 21:07:50.576377 9859 leader_election.cc:343] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:50.576505 9859 leader_election.cc:304] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:07:50.576723 9859 raft_consensus.cc:2686] T 0c1a6a7616f24f69b83ec8a282746e92 P 0fbd567f929a4f6d8e4d78feef7ed678: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251027 21:07:50.576750 9775 raft_consensus.cc:2468] T cbada3946dab4739a8369032ff3b8a48 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 1.
I20251027 21:07:50.577171 10178 raft_consensus.cc:2804] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:50.577296 10178 raft_consensus.cc:493] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:50.577420 10178 raft_consensus.cc:3060] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:50.578073 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 94f2dbf47a5c4458bee54b26f272156e (DEFAULT_TABLE table=default.Table_7954e74d079b416cac73f9a77979c8b3 [id=0103a75b0a714d3c9a533d94b6bda41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.578303 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_8a1e00e5debb481cbc1fdef16d74527b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:50.578619 10029 ts_tablet_manager.cc:1434] T 6e5d52b7ec114be5ae68e57280c2eea8 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.363s user 0.062s sys 0.059s
W20251027 21:07:50.578886 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a1e00e5debb481cbc1fdef16d74527b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.580241 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5dc88891b75b4f1ca7b3ffba0d2b1fda"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.580740 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5dc88891b75b4f1ca7b3ffba0d2b1fda in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.582739 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9e88d5f78e27447c8072ce0563686ac8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.583220 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e88d5f78e27447c8072ce0563686ac8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.584309 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_757ff3ea762e447ba76041b257234644"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.584733 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_757ff3ea762e447ba76041b257234644 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.585927 9615 tablet_service.cc:1505] Processing CreateTablet for tablet a9bf3a093be8498c9dd5511b53436b16 (DEFAULT_TABLE table=default.Table_eb5c676ccdf040b2bf8822c15f54fb6f [id=5c9fd870185a4211a5a956171f491cc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.585983 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0ce16ec9dc8b44eab80cedc074577cba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.586488 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ce16ec9dc8b44eab80cedc074577cba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.589362 10041 ts_tablet_manager.cc:1929] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:50.589507 10041 log.cc:1199] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/44783760defd433a972c9cd83933249c
I20251027 21:07:50.589833 10041 ts_tablet_manager.cc:1950] T 44783760defd433a972c9cd83933249c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:50.589861 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2d874641d88e434c8022d7026802972f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:50.590932 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_40e7109b10b84bacaab337e0e52a63ef" } new_table_name: "default.Table_0d0de750eda84895820447f59e8ab3d3" modify_external_catalogs: true
W20251027 21:07:50.591322 9859 leader_election.cc:343] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:50.591401 10178 raft_consensus.cc:515] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.593780 9859 leader_election.cc:304] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:50.595155 10179 raft_consensus.cc:2749] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:50.595670 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 153) took 1215 ms. Trace:
I20251027 21:07:50.595750 10041 rpcz_store.cc:276] 1027 21:07:49.380220 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.380458 (+ 238us) service_pool.cc:225] Handling call
1027 21:07:50.552136 (+1171678us) ts_tablet_manager.cc:1151] Deleting tablet 44783760defd433a972c9cd83933249c
1027 21:07:50.552146 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.570013 (+ 17867us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.589281 (+ 19268us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.595604 (+ 6323us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6838,"tablet-delete.queue_time_us":1171364}
I20251027 21:07:50.596192 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 44783760defd433a972c9cd83933249c (table default.Table_44e3f093c8d7476a98074b07828da3ab [id=30fbbddfaed646e3b2d80f788f774741]) successfully deleted
I20251027 21:07:50.597077 10041 tablet_replica.cc:333] T cad3a3e910ac42399cf371ad82c89ff7 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:50.597258 10041 raft_consensus.cc:2243] T cad3a3e910ac42399cf371ad82c89ff7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:50.597445 10041 raft_consensus.cc:2272] T cad3a3e910ac42399cf371ad82c89ff7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:07:50.598402 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d874641d88e434c8022d7026802972f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.599538 10041 ts_tablet_manager.cc:1916] T cad3a3e910ac42399cf371ad82c89ff7 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:50.599640 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a90c80308cb345d7abc6c44d3d4c70ff" } modify_external_catalogs: true
I20251027 21:07:50.599815 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a90c80308cb345d7abc6c44d3d4c70ff" } modify_external_catalogs: true
I20251027 21:07:50.604336 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c19fea3933b84e87ad91298bdb880696" } new_table_name: "default.Table_caf93d0298314165abb645155ffd1a41" modify_external_catalogs: true
I20251027 21:07:50.604512 10178 leader_election.cc:290] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:50.610546 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet 9882b234e79940088be5f604c41576b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:50.610622 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648f506ec87c4fb8a3f6d4591873dbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.610790 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e5d52b7ec114be5ae68e57280c2eea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:50.613420 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adaa4f66bbab491f8030d439d85cc7d1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:50.613642 9711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 99181.1us
I20251027 21:07:50.614243 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9bf3a093be8498c9dd5511b53436b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.618935 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 65fbcaa5447c48818fabfccac22c4048 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:50.619714 9598 tablet_service.cc:1552] Processing DeleteTablet for tablet e0968c2f25b64f6685f38a999ee9d60e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:50.620800 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 94e94cd50c91429d8da9804149501c86 with delete_type TABLET_DATA_DELETED (replaced by tablet 26872be132fb46ee9e79a9c2e62374ef) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:50.624539 9860 leader_election.cc:343] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 5c87a5450d354506852c4b20bf0541d2
W20251027 21:07:50.625087 9711 leader_election.cc:343] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 78a0a8b05f10477c923ab565839141b8
I20251027 21:07:50.625249 9711 leader_election.cc:304] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:50.626631 10041 ts_tablet_manager.cc:1929] T cad3a3e910ac42399cf371ad82c89ff7 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:50.626871 10041 log.cc:1199] T cad3a3e910ac42399cf371ad82c89ff7 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/cad3a3e910ac42399cf371ad82c89ff7
I20251027 21:07:50.627246 10041 ts_tablet_manager.cc:1950] T cad3a3e910ac42399cf371ad82c89ff7 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:50.627570 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 163) took 1178 ms. Trace:
I20251027 21:07:50.628665 10041 rpcz_store.cc:276] 1027 21:07:49.449541 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.451558 (+ 2017us) service_pool.cc:225] Handling call
1027 21:07:50.596989 (+1145431us) ts_tablet_manager.cc:1151] Deleting tablet cad3a3e910ac42399cf371ad82c89ff7
1027 21:07:50.596999 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.611316 (+ 14317us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.626476 (+ 15160us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.627543 (+ 1067us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1141728}
I20251027 21:07:50.629781 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet cad3a3e910ac42399cf371ad82c89ff7 (table default.Table_b84197018ff44b40b7219e21846339f2 [id=0286a1b3137649eeaa5339cbdf97aa03]) successfully deleted
I20251027 21:07:50.630391 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 164) took 1180 ms. Trace:
I20251027 21:07:50.630932 10041 rpcz_store.cc:276] 1027 21:07:49.449636 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.449989 (+ 353us) service_pool.cc:225] Handling call
1027 21:07:50.629880 (+1179891us) ts_tablet_manager.cc:1151] Deleting tablet 81e914ecc3c74b869e9fc3ad102887ec
1027 21:07:50.629889 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.630355 (+ 466us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1174055}
W20251027 21:07:50.631706 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 81e914ecc3c74b869e9fc3ad102887ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81e914ecc3c74b869e9fc3ad102887ec already in progress: creating tablet
I20251027 21:07:50.631983 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 168) took 1172 ms. Trace:
I20251027 21:07:50.632743 10041 rpcz_store.cc:276] 1027 21:07:49.459736 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.460133 (+ 397us) service_pool.cc:225] Handling call
1027 21:07:50.631828 (+1171695us) ts_tablet_manager.cc:1151] Deleting tablet 6db0f31c12d14e66a4ec7fc722d62c3d
1027 21:07:50.631836 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.631935 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1170953}
W20251027 21:07:50.633575 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 6db0f31c12d14e66a4ec7fc722d62c3d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6db0f31c12d14e66a4ec7fc722d62c3d already in progress: creating tablet
I20251027 21:07:50.633684 10041 tablet_replica.cc:333] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:50.633910 10041 raft_consensus.cc:2243] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251027 21:07:50.634305 9711 leader_election.cc:343] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: d642da56a67449aa95932180bc5b2c5f
I20251027 21:07:50.634438 9711 leader_election.cc:304] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:50.635154 10041 raft_consensus.cc:2272] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.636266 10041 ts_tablet_manager.cc:1916] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:50.637902 10049 raft_consensus.cc:2804] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:50.638386 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94f2dbf47a5c4458bee54b26f272156e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.640007 10029 tablet_bootstrap.cc:492] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:50.640704 10182 raft_consensus.cc:515] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.642189 10184 raft_consensus.cc:2749] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:50.642371 10141 raft_consensus.cc:2749] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:50.646829 10032 tablet_bootstrap.cc:492] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:50.647154 10032 ts_tablet_manager.cc:1403] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.098s user 0.012s sys 0.012s
I20251027 21:07:50.646922 9631 raft_consensus.cc:1275] T 40bd202c1b834c949f9a5cee23b3486d P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Refusing update from remote peer 0fbd567f929a4f6d8e4d78feef7ed678: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 21:07:50.648388 10032 raft_consensus.cc:359] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.649776 10032 raft_consensus.cc:385] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.649895 10032 raft_consensus.cc:740] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.651243 10108 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 21:07:50.654320 9748 tablet_service.cc:1505] Processing CreateTablet for tablet c4996d8fae094db8a96599b9057f1649 (DEFAULT_TABLE table=default.Table_b8bc4fe8cb374bd990d2c777eae1d4f0 [id=68da6285e1d247f1950a0b4929c885a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.654626 10032 consensus_queue.cc:260] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.655360 10032 ts_tablet_manager.cc:1434] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251027 21:07:50.655788 10032 tablet_bootstrap.cc:492] T 36e168cc44274dae9682b0c1329d4558 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:50.657011 10108 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 21:07:50.660756 10032 tablet_bootstrap.cc:654] T 36e168cc44274dae9682b0c1329d4558 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:50.660898 10043 ts_tablet_manager.cc:1929] T 0029b70ed3ff4db9a816a6190d36bf88 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:50.661676 10182 leader_election.cc:290] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:50.658058 10049 raft_consensus.cc:493] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:50.662629 10043 log.cc:1199] T 0029b70ed3ff4db9a816a6190d36bf88 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/0029b70ed3ff4db9a816a6190d36bf88
I20251027 21:07:50.662827 10108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 21:07:50.662824 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e5d52b7ec114be5ae68e57280c2eea8" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:07:50.662423 10178 consensus_queue.cc:1048] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:07:50.663560 10049 raft_consensus.cc:3060] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:50.663528 10179 raft_consensus.cc:493] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:50.664268 10179 raft_consensus.cc:515] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.664644 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f1129eecfb54448965222269d9c32b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:50.664917 10179 leader_election.cc:290] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:50.665324 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c239216cb8846f0b1aaa360ddb83321 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:50.659205 10041 ts_tablet_manager.cc:1929] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:50.665525 10041 log.cc:1199] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/5c87a5450d354506852c4b20bf0541d2
I20251027 21:07:50.665652 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4996d8fae094db8a96599b9057f1649. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.665867 10041 ts_tablet_manager.cc:1950] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:50.666301 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 169) took 1201 ms. Trace:
I20251027 21:07:50.666687 10043 ts_tablet_manager.cc:1950] T 0029b70ed3ff4db9a816a6190d36bf88 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:50.666841 9634 raft_consensus.cc:3060] T 6e5d52b7ec114be5ae68e57280c2eea8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:50.667316 10041 rpcz_store.cc:276] 1027 21:07:49.465149 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.465223 (+ 74us) service_pool.cc:225] Handling call
1027 21:07:50.633629 (+1168406us) ts_tablet_manager.cc:1151] Deleting tablet 5c87a5450d354506852c4b20bf0541d2
1027 21:07:50.633638 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.649092 (+ 15454us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.659110 (+ 10018us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.666254 (+ 7144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1167114}
I20251027 21:07:50.668171 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 5c87a5450d354506852c4b20bf0541d2 (table default.Table_d1fb7f4ef09e4bfdac00a75f752f0873 [id=1cf515b159954979acd73a0b276e7503]) successfully deleted
I20251027 21:07:50.668359 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 170) took 1146 ms. Trace:
I20251027 21:07:50.668452 10041 rpcz_store.cc:276] 1027 21:07:49.521968 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.522020 (+ 52us) service_pool.cc:225] Handling call
1027 21:07:50.668215 (+1146195us) ts_tablet_manager.cc:1151] Deleting tablet a9cc64ad85334daa9a09538e24ebd8a5
1027 21:07:50.668223 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.668313 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1145971}
I20251027 21:07:50.666880 9741 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 153) took 1067 ms. Trace:
I20251027 21:07:50.669865 10040 ts_tablet_manager.cc:1929] T 12aa6e3af0694429b45bef5af3592377 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:50.669992 10040 log.cc:1199] T 12aa6e3af0694429b45bef5af3592377 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/12aa6e3af0694429b45bef5af3592377
I20251027 21:07:50.670464 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 171) took 1147 ms. Trace:
W20251027 21:07:50.671326 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet a9cc64ad85334daa9a09538e24ebd8a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9cc64ad85334daa9a09538e24ebd8a5 already in progress: creating tablet
I20251027 21:07:50.672730 10041 rpcz_store.cc:276] 1027 21:07:49.523096 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.524098 (+ 1002us) service_pool.cc:225] Handling call
1027 21:07:50.670298 (+1146200us) ts_tablet_manager.cc:1151] Deleting tablet c49775486ff24686b0f45c47e2757aba
1027 21:07:50.670307 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.670414 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1145963}
I20251027 21:07:50.673656 10041 tablet_replica.cc:333] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:50.674232 10041 raft_consensus.cc:2243] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251027 21:07:50.674346 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet c49775486ff24686b0f45c47e2757aba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c49775486ff24686b0f45c47e2757aba already in progress: creating tablet
I20251027 21:07:50.674928 10041 raft_consensus.cc:2272] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.676005 10041 ts_tablet_manager.cc:1916] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:50.676379 9312 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 26872be132fb46ee9e79a9c2e62374ef
W20251027 21:07:50.681360 9711 leader_election.cc:343] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 6e5d52b7ec114be5ae68e57280c2eea8
I20251027 21:07:50.687093 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61cd8ee8e1f8459eb7af5dd48c018e02" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:50.688838 9860 leader_election.cc:343] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:50.691406 9634 raft_consensus.cc:2468] T 6e5d52b7ec114be5ae68e57280c2eea8 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
W20251027 21:07:50.691275 9711 leader_election.cc:343] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 63662f175a1c450f8f67f37da971db37
I20251027 21:07:50.695336 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_eb5c676ccdf040b2bf8822c15f54fb6f" } new_table_name: "default.Table_627b1e3af2d34bc9b4508237abf020f5" modify_external_catalogs: true
I20251027 21:07:50.697069 10029 tablet_bootstrap.cc:654] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:50.697981 10041 ts_tablet_manager.cc:1929] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:50.698241 9305 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 0ac7111d13b6460a9f4c77e94e09d417 (table default.Table_e65d5d66ce8747a9a1a2abfa04cd2855 [id=e2b68fb34c3b48e8952ef6d4201fd93f]). Expected version 1 got 0
I20251027 21:07:50.699220 9711 leader_election.cc:304] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:50.700232 10141 raft_consensus.cc:2749] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251027 21:07:50.700438 9710 leader_election.cc:304] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:07:50.700807 10141 raft_consensus.cc:2804] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:50.701880 10141 raft_consensus.cc:697] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:07:50.703560 9711 leader_election.cc:304] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218; no voters: 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:50.703629 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 163) took 1012 ms. Trace:
I20251027 21:07:50.707027 10041 log.cc:1199] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/cbada3946dab4739a8369032ff3b8a48
I20251027 21:07:50.707639 10041 ts_tablet_manager.cc:1950] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:50.709067 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 172) took 1121 ms. Trace:
I20251027 21:07:50.709686 10041 rpcz_store.cc:276] 1027 21:07:49.587502 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.587545 (+ 43us) service_pool.cc:225] Handling call
1027 21:07:50.673587 (+1086042us) ts_tablet_manager.cc:1151] Deleting tablet cbada3946dab4739a8369032ff3b8a48
1027 21:07:50.673596 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.689606 (+ 16010us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.697862 (+ 8256us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.709038 (+ 11176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1085827}
I20251027 21:07:50.710325 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 174) took 1092 ms. Trace:
I20251027 21:07:50.710448 10032 tablet_bootstrap.cc:492] T 36e168cc44274dae9682b0c1329d4558 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:50.710589 10032 ts_tablet_manager.cc:1403] T 36e168cc44274dae9682b0c1329d4558 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.006s
I20251027 21:07:50.710489 10041 rpcz_store.cc:276] 1027 21:07:49.618237 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.618287 (+ 50us) service_pool.cc:225] Handling call
1027 21:07:50.710180 (+1091893us) ts_tablet_manager.cc:1151] Deleting tablet 583fba7c750e4ac7b768da447002c211
1027 21:07:50.710189 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.710287 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1085236}
I20251027 21:07:50.711063 10041 tablet_replica.cc:333] T ba8b125474bc42a5946e8f2dd718f7f4 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
W20251027 21:07:50.711052 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 583fba7c750e4ac7b768da447002c211 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 583fba7c750e4ac7b768da447002c211 already in progress: creating tablet
I20251027 21:07:50.711490 10041 raft_consensus.cc:2243] T ba8b125474bc42a5946e8f2dd718f7f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:50.711709 10041 raft_consensus.cc:2272] T ba8b125474bc42a5946e8f2dd718f7f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.711912 9859 leader_election.cc:304] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218; no voters: 51902b6b61ff4d5584ff21896e847e14
W20251027 21:07:50.712133 9859 raft_consensus.cc:2686] T cbada3946dab4739a8369032ff3b8a48 P 0fbd567f929a4f6d8e4d78feef7ed678: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251027 21:07:50.712730 10041 ts_tablet_manager.cc:1916] T ba8b125474bc42a5946e8f2dd718f7f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:50.713114 10032 raft_consensus.cc:359] T 36e168cc44274dae9682b0c1329d4558 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.713307 10032 raft_consensus.cc:385] T 36e168cc44274dae9682b0c1329d4558 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.713394 10032 raft_consensus.cc:740] T 36e168cc44274dae9682b0c1329d4558 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.713531 10184 raft_consensus.cc:2804] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:50.704550 10182 consensus_queue.cc:1048] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20251027 21:07:50.713985 10032 consensus_queue.cc:260] T 36e168cc44274dae9682b0c1329d4558 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.714797 10184 raft_consensus.cc:697] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:07:50.715447 10032 ts_tablet_manager.cc:1434] T 36e168cc44274dae9682b0c1329d4558 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20251027 21:07:50.715085 10184 consensus_queue.cc:237] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.717427 10032 tablet_bootstrap.cc:492] T 2d659e4d73ea4ff3975a4b58a794090a P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:50.718300 9771 raft_consensus.cc:1275] T 40bd202c1b834c949f9a5cee23b3486d P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Refusing update from remote peer 0fbd567f929a4f6d8e4d78feef7ed678: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 21:07:50.718994 10179 raft_consensus.cc:493] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:50.719168 10179 raft_consensus.cc:515] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.718690 10141 consensus_queue.cc:237] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.720296 10179 leader_election.cc:290] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:50.720897 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35e5b7bbab7f4ee9bbf23d913c0aee76" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:50.721426 9860 leader_election.cc:343] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:50.724485 10041 ts_tablet_manager.cc:1929] T ba8b125474bc42a5946e8f2dd718f7f4 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:50.731683 10041 log.cc:1199] T ba8b125474bc42a5946e8f2dd718f7f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/ba8b125474bc42a5946e8f2dd718f7f4
I20251027 21:07:50.732074 10041 ts_tablet_manager.cc:1950] T ba8b125474bc42a5946e8f2dd718f7f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:50.732808 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 175) took 1096 ms. Trace:
I20251027 21:07:50.732982 10041 rpcz_store.cc:276] 1027 21:07:49.636029 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.636089 (+ 60us) service_pool.cc:225] Handling call
1027 21:07:50.711004 (+1074915us) ts_tablet_manager.cc:1151] Deleting tablet ba8b125474bc42a5946e8f2dd718f7f4
1027 21:07:50.711012 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.720780 (+ 9768us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.724407 (+ 3627us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.732763 (+ 8356us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1074647}
I20251027 21:07:50.733574 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 189) took 1036 ms. Trace:
I20251027 21:07:50.733563 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet ba8b125474bc42a5946e8f2dd718f7f4 (table default.Table_16dd2e0479ea434dbbd0b900b340fc9f [id=e7b7bdb2ea7f468c98bfe88f900a5be8]) successfully deleted
I20251027 21:07:50.733738 10041 rpcz_store.cc:276] 1027 21:07:49.697036 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.697483 (+ 447us) service_pool.cc:225] Handling call
1027 21:07:50.733416 (+1035933us) ts_tablet_manager.cc:1151] Deleting tablet b87e809ff897417c88e5edc19e2d591e
1027 21:07:50.733425 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.733530 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1030427}
I20251027 21:07:50.734297 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 188) took 1037 ms. Trace:
I20251027 21:07:50.734426 10041 rpcz_store.cc:276] 1027 21:07:49.696938 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.697453 (+ 515us) service_pool.cc:225] Handling call
1027 21:07:50.734120 (+1036667us) ts_tablet_manager.cc:1151] Deleting tablet 10a44aecfade42969454126b990eca24
1027 21:07:50.734128 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.734257 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1027192}
I20251027 21:07:50.734958 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 190) took 1036 ms. Trace:
W20251027 21:07:50.734978 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet b87e809ff897417c88e5edc19e2d591e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b87e809ff897417c88e5edc19e2d591e already in progress: creating tablet
W20251027 21:07:50.735286 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 10a44aecfade42969454126b990eca24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10a44aecfade42969454126b990eca24 already in progress: creating tablet
I20251027 21:07:50.735293 10041 rpcz_store.cc:276] 1027 21:07:49.698222 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.699107 (+ 885us) service_pool.cc:225] Handling call
1027 21:07:50.734813 (+1035706us) ts_tablet_manager.cc:1151] Deleting tablet 4e8c83b501d8426fa0cad6a89f9a0b9b
1027 21:07:50.734823 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.734912 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1025755}
W20251027 21:07:50.736150 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 4e8c83b501d8426fa0cad6a89f9a0b9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e8c83b501d8426fa0cad6a89f9a0b9b already in progress: creating tablet
W20251027 21:07:50.736406 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 49b3d23798b740008c98cd0754ae33f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49b3d23798b740008c98cd0754ae33f4 already in progress: creating tablet
I20251027 21:07:50.711438 10043 rpcz_store.cc:276] 1027 21:07:49.670909 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.671959 (+ 1050us) service_pool.cc:225] Handling call
1027 21:07:49.679060 (+ 7101us) ts_tablet_manager.cc:1151] Deleting tablet 0029b70ed3ff4db9a816a6190d36bf88
1027 21:07:49.679069 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.183880 (+504811us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.660810 (+476930us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.683308 (+ 22498us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6873}
I20251027 21:07:50.736733 10041 tablet_replica.cc:333] T 87ec5a6b9b3c425da4e96cc6736d6310 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:50.737229 10041 raft_consensus.cc:2243] T 87ec5a6b9b3c425da4e96cc6736d6310 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:50.737690 10041 raft_consensus.cc:2272] T 87ec5a6b9b3c425da4e96cc6736d6310 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.737684 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 0029b70ed3ff4db9a816a6190d36bf88 (table default.Table_450e3d88ee8f4a328418b279a354b9e2 [id=0f311c7d59a4471fbad8c2a0c59430f0]) successfully deleted
W20251027 21:07:50.737991 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet b87e809ff897417c88e5edc19e2d591e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b87e809ff897417c88e5edc19e2d591e already in progress: creating tablet
W20251027 21:07:50.738276 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 10a44aecfade42969454126b990eca24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10a44aecfade42969454126b990eca24 already in progress: creating tablet
I20251027 21:07:50.739063 10041 ts_tablet_manager.cc:1916] T 87ec5a6b9b3c425da4e96cc6736d6310 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:50.739248 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 4e8c83b501d8426fa0cad6a89f9a0b9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e8c83b501d8426fa0cad6a89f9a0b9b already in progress: creating tablet
W20251027 21:07:50.739553 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 49b3d23798b740008c98cd0754ae33f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49b3d23798b740008c98cd0754ae33f4 already in progress: creating tablet
I20251027 21:07:50.740609 10043 tablet_replica.cc:333] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:50.740870 10043 raft_consensus.cc:2243] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:07:50.741765 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c87a5450d354506852c4b20bf0541d2" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:50.742379 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61cd8ee8e1f8459eb7af5dd48c018e02" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:50.742904 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35e5b7bbab7f4ee9bbf23d913c0aee76" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:50.744870 9859 leader_election.cc:343] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:50.744858 10179 consensus_queue.cc:1048] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251027 21:07:50.744987 9711 leader_election.cc:343] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 6e5d52b7ec114be5ae68e57280c2eea8
I20251027 21:07:50.745043 9859 leader_election.cc:304] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:50.745354 10179 raft_consensus.cc:2749] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:50.745316 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 94e94cd50c91429d8da9804149501c86 with delete_type TABLET_DATA_DELETED (replaced by tablet 26872be132fb46ee9e79a9c2e62374ef) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:50.745509 9859 leader_election.cc:343] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:50.745627 9859 leader_election.cc:304] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:50.745786 9755 tablet_service.cc:1505] Processing CreateTablet for tablet a9bf3a093be8498c9dd5511b53436b16 (DEFAULT_TABLE table=default.Table_eb5c676ccdf040b2bf8822c15f54fb6f [id=5c9fd870185a4211a5a956171f491cc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.746062 10179 raft_consensus.cc:2749] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:50.746188 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9bf3a093be8498c9dd5511b53436b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.747881 10043 raft_consensus.cc:2272] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.748297 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet e0968c2f25b64f6685f38a999ee9d60e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:50.748723 9747 tablet_service.cc:1505] Processing CreateTablet for tablet 648f506ec87c4fb8a3f6d4591873dbb8 (DEFAULT_TABLE table=default.Table_a46825cc888a4a0898d6db70b02e5ab6 [id=8579d3ff46d64c75b1da213fdee9cc1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.748919 10043 ts_tablet_manager.cc:1916] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:50.749114 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648f506ec87c4fb8a3f6d4591873dbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.750443 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 9882b234e79940088be5f604c41576b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:50.750459 10179 raft_consensus.cc:493] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:50.750622 10179 raft_consensus.cc:515] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.751190 10179 leader_election.cc:290] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:50.752050 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 45568792d7954b498581ce69fb611948 (DEFAULT_TABLE table=default.Table_1f82ff66f4a14bb9ac990665fd6ec38e [id=2eb093cd58e643a58b14e564d5914fbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:50.752452 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45568792d7954b498581ce69fb611948. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:50.752678 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c4369a10bf846fb94c9b71a70f21600" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:50.753358 9860 leader_election.cc:343] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:50.752382 10049 raft_consensus.cc:515] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.758633 10049 leader_election.cc:290] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:50.758538 10202 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:59712 (request call id 162) took 1108 ms. Trace:
W20251027 21:07:50.759356 9710 leader_election.cc:343] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 5c87a5450d354506852c4b20bf0541d2
I20251027 21:07:50.759894 9771 raft_consensus.cc:2393] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fbd567f929a4f6d8e4d78feef7ed678 in current term 1: Already voted for candidate 77b0513f17c0472088901a1ce8b08218 in this term.
I20251027 21:07:50.761508 9859 leader_election.cc:304] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:07:50.761708 9859 raft_consensus.cc:2686] T 5c87a5450d354506852c4b20bf0541d2 P 0fbd567f929a4f6d8e4d78feef7ed678: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251027 21:07:50.764767 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c4369a10bf846fb94c9b71a70f21600" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:50.765355 9741 rpcz_store.cc:276] 1027 21:07:49.595310 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.595364 (+ 54us) service_pool.cc:225] Handling call
1027 21:07:49.692143 (+ 96779us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:49.692172 (+ 29us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:50.396893 (+704721us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.662600 (+265707us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:50.662752 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:50.765781 10040 ts_tablet_manager.cc:1950] T 12aa6e3af0694429b45bef5af3592377 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:50.766075 10217 tablet.cc:1722] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:07:50.766126 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 147) took 1174 ms. Trace:
I20251027 21:07:50.766259 10040 rpcz_store.cc:276] 1027 21:07:49.591562 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.594406 (+ 2844us) service_pool.cc:225] Handling call
1027 21:07:50.105636 (+511230us) ts_tablet_manager.cc:1151] Deleting tablet 12aa6e3af0694429b45bef5af3592377
1027 21:07:50.105644 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.406476 (+300832us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.658709 (+252233us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.766091 (+107382us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":508460}
I20251027 21:07:50.766600 10040 tablet_replica.cc:333] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:50.767047 10031 ts_tablet_manager.cc:1434] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.576s user 0.013s sys 0.026s
W20251027 21:07:50.767264 10031 ts_tablet_manager.cc:1467] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218: Tablet startup took 674ms
W20251027 21:07:50.767413 9859 leader_election.cc:343] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:50.767563 9859 leader_election.cc:304] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:07:50.767761 10031 ts_tablet_manager.cc:1469] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218: Trace:
1027 21:07:47.802109 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:47.802186 (+ 77us) service_pool.cc:225] Handling call
1027 21:07:47.802649 (+ 463us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:47.802683 (+ 34us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:47.807139 (+ 4456us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:47.809871 (+ 2732us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:47.810006 (+ 135us) inbound_call.cc:173] Queueing success response
1027 21:07:49.548124 (+1738118us) ts_tablet_manager.cc:1379] Bootstrapping tablet
1027 21:07:50.092357 (+544233us) ts_tablet_manager.cc:1435] Starting tablet replica
1027 21:07:50.092561 (+ 204us) tablet_replica.cc:261] Starting instrumentation
1027 21:07:50.092696 (+ 135us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":34295,"raft.run_cpu_time_us":66,"raft.run_wall_time_us":65,"tablet-open.queue_time_us":1738153,"thread_start_us":256917,"threads_started":1}
I20251027 21:07:50.768939 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 12aa6e3af0694429b45bef5af3592377 (table default.Table_36bfef90af1d48f78b2cce6a78527a44 [id=0065037e95714bb6ad48cbe647173541]) successfully deleted
I20251027 21:07:50.771633 10179 raft_consensus.cc:2749] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:50.772230 10031 tablet_bootstrap.cc:492] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:07:50.771871 9711 leader_election.cc:343] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 5c87a5450d354506852c4b20bf0541d2
I20251027 21:07:50.758800 10202 rpcz_store.cc:276] 1027 21:07:49.649973 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.650284 (+ 311us) service_pool.cc:225] Handling call
1027 21:07:50.758474 (+1108190us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:50.069646 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1027 21:07:50.069850 (+ 204us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1027 21:07:50.069881 (+ 31us) alter_schema_op.cc:66] Acquired schema lock
1027 21:07:50.069928 (+ 47us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1027 21:07:50.070038 (+ 110us) alter_schema_op.cc:128] START. Timestamp: 1761599270069959
1027 21:07:50.070045 (+ 7us) op_driver.cc:348] REPLICATION: starting
1027 21:07:50.070404 (+ 359us) log.cc:844] Serialized 173 byte log entry
1027 21:07:50.285610 (+215206us) op_driver.cc:464] REPLICATION: finished
1027 21:07:50.518588 (+232978us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1027 21:07:50.520405 (+ 1817us) log.cc:844] Serialized 5 byte log entry
1027 21:07:50.728670 (+208265us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.752855 (+ 24185us) log.cc:844] Serialized 14 byte log entry
1027 21:07:50.753104 (+ 249us) alter_schema_op.cc:73] Released schema lock
1027 21:07:50.753299 (+ 195us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":232849,"mutex_wait_us":440,"prepare.queue_time_us":417125,"prepare.run_cpu_time_us":967,"prepare.run_wall_time_us":1399,"raft.queue_time_us":549,"raft.run_cpu_time_us":535,"raft.run_wall_time_us":533,"replication_time_us":215492,"thread_start_us":673002,"threads_started":2}]]}
I20251027 21:07:50.777601 10041 ts_tablet_manager.cc:1929] T 87ec5a6b9b3c425da4e96cc6736d6310 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:07:50.777721 10041 log.cc:1199] T 87ec5a6b9b3c425da4e96cc6736d6310 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/87ec5a6b9b3c425da4e96cc6736d6310
I20251027 21:07:50.778191 10041 ts_tablet_manager.cc:1950] T 87ec5a6b9b3c425da4e96cc6736d6310 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:50.774130 10031 tablet_bootstrap.cc:654] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:50.779201 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 87ec5a6b9b3c425da4e96cc6736d6310 (table default.Table_0c05919c1a104864a4e1cd182ae59e88 [id=16df16d9dc6144cabc7f453c319255d9]) successfully deleted
I20251027 21:07:50.774296 9711 leader_election.cc:304] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:50.779572 10182 raft_consensus.cc:2749] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20251027 21:07:50.790248 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 9248f458baca409aa3505cc0db321b34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9248f458baca409aa3505cc0db321b34 already in progress: creating tablet
I20251027 21:07:50.790913 10041 tablet_replica.cc:333] T 9cec6294051b41868c9c7c7fadafa138 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:50.791105 10041 raft_consensus.cc:2243] T 9cec6294051b41868c9c7c7fadafa138 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:50.791249 10041 raft_consensus.cc:2272] T 9cec6294051b41868c9c7c7fadafa138 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.791730 10041 ts_tablet_manager.cc:1916] T 9cec6294051b41868c9c7c7fadafa138 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:50.792368 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 49d9e5c5960a4534a6603875006e6266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49d9e5c5960a4534a6603875006e6266 already in progress: creating tablet
I20251027 21:07:50.793164 10029 tablet_bootstrap.cc:492] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:50.793310 10029 ts_tablet_manager.cc:1403] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.211s user 0.004s sys 0.003s
I20251027 21:07:50.794070 10029 raft_consensus.cc:359] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.794297 10029 raft_consensus.cc:385] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.794364 10029 raft_consensus.cc:740] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.794585 10029 consensus_queue.cc:260] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.797901 10032 tablet_bootstrap.cc:654] T 2d659e4d73ea4ff3975a4b58a794090a P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:50.799360 10217 tablet.cc:1722] T 40bd202c1b834c949f9a5cee23b3486d P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:07:50.799432 10029 ts_tablet_manager.cc:1434] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251027 21:07:50.799413 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_31d10565065d494194f0dda082708ac0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:50.799679 10029 tablet_bootstrap.cc:492] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
W20251027 21:07:50.799989 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31d10565065d494194f0dda082708ac0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.800763 10174 mvcc.cc:204] Tried to move back new op lower bound from 7215510612270841856 to 7215510610994663424. Current Snapshot: MvccSnapshot[applied={T|T < 7215510612270841856}]
I20251027 21:07:50.801689 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e68b56d3ddd94f308243ec6a03ea9e6e" } modify_external_catalogs: true
I20251027 21:07:50.801848 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e68b56d3ddd94f308243ec6a03ea9e6e" } modify_external_catalogs: true
W20251027 21:07:50.803206 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92a2393cff8044028c86c8cd2bccd930 consensus state has no leader
I20251027 21:07:50.803438 9306 master_service.cc:954] ReplaceTablet: received request to replace tablet b3dfef8b741c42f68b664d7e0eab49c6 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:50.803891 9306 master_service.cc:954] ReplaceTablet: received request to replace tablet 87ec5a6b9b3c425da4e96cc6736d6310 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:50.804385 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ba1d973c05e14c70944d76ff1ecbc1bc" } modify_external_catalogs: true
I20251027 21:07:50.804549 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ba1d973c05e14c70944d76ff1ecbc1bc" } modify_external_catalogs: true
I20251027 21:07:50.805359 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_546a5293975148e1841c8fff9eeac42a" } modify_external_catalogs: true
I20251027 21:07:50.805543 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_546a5293975148e1841c8fff9eeac42a" } modify_external_catalogs: true
I20251027 21:07:50.807334 10029 tablet_bootstrap.cc:654] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:50.810570 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c5f2e60caa824b5db2330b423221873c
I20251027 21:07:50.811111 10032 tablet_bootstrap.cc:492] T 2d659e4d73ea4ff3975a4b58a794090a P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:50.811291 10032 ts_tablet_manager.cc:1403] T 2d659e4d73ea4ff3975a4b58a794090a P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.096s user 0.001s sys 0.002s
I20251027 21:07:50.811288 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_908c9569154d403eb8aeab2a65da7f67" } new_table_name: "default.Table_e9fd9bee6a2a40d7ad92d2d31b03afce" modify_external_catalogs: true
I20251027 21:07:50.812103 10032 raft_consensus.cc:359] T 2d659e4d73ea4ff3975a4b58a794090a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.812296 10032 raft_consensus.cc:385] T 2d659e4d73ea4ff3975a4b58a794090a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.812675 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet c5f2e60caa824b5db2330b423221873c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:50.812919 10202 tablet.cc:1722] T 40bd202c1b834c949f9a5cee23b3486d P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:07:50.812376 10032 raft_consensus.cc:740] T 2d659e4d73ea4ff3975a4b58a794090a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.819928 10032 consensus_queue.cc:260] T 2d659e4d73ea4ff3975a4b58a794090a P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.821866 10041 ts_tablet_manager.cc:1929] T 9cec6294051b41868c9c7c7fadafa138 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:50.821997 10041 log.cc:1199] T 9cec6294051b41868c9c7c7fadafa138 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/9cec6294051b41868c9c7c7fadafa138
I20251027 21:07:50.822368 10041 ts_tablet_manager.cc:1950] T 9cec6294051b41868c9c7c7fadafa138 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:50.823215 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 9cec6294051b41868c9c7c7fadafa138 (table default.Table_5d8e4122af254070a2df7016e521a937 [id=12b1bbc770194455ab7292641980e8e1]) successfully deleted
I20251027 21:07:50.826161 10032 ts_tablet_manager.cc:1434] T 2d659e4d73ea4ff3975a4b58a794090a P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.015s user 0.002s sys 0.001s
I20251027 21:07:50.826514 10032 tablet_bootstrap.cc:492] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:50.828553 10032 tablet_bootstrap.cc:654] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:50.833503 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 970fbf01a4c3406dafe1ea4cd005ea23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 970fbf01a4c3406dafe1ea4cd005ea23 already in progress: creating tablet
I20251027 21:07:50.833971 10041 tablet_replica.cc:333] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:50.834159 10041 raft_consensus.cc:2243] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:50.834323 10041 raft_consensus.cc:2272] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.834437 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet c5f2e60caa824b5db2330b423221873c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:50.834856 10041 ts_tablet_manager.cc:1916] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:50.838488 9754 tablet_service.cc:1552] Processing DeleteTablet for tablet c5f2e60caa824b5db2330b423221873c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:50.839982 10040 raft_consensus.cc:2243] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:07:50.839756 10043 ts_tablet_manager.cc:1929] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:07:50.840655 10040 pending_rounds.cc:70] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
I20251027 21:07:50.840816 10040 pending_rounds.cc:77] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510612873150464 op_type: NO_OP noop_request { }
I20251027 21:07:50.840945 10040 raft_consensus.cc:2889] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:07:50.841060 10040 raft_consensus.cc:2272] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251027 21:07:50.843257 10234 consensus_queue.cc:1175] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "77b0513f17c0472088901a1ce8b08218" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20251027 21:07:50.844002 10040 ts_tablet_manager.cc:1916] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:50.843591 10043 log.cc:1199] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/87ec5a6b9b3c425da4e96cc6736d6310
I20251027 21:07:50.845515 10241 tablet.cc:1722] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:07:50.850298 10043 ts_tablet_manager.cc:1950] T 87ec5a6b9b3c425da4e96cc6736d6310 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:50.851055 10029 tablet_bootstrap.cc:492] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:50.851210 10029 ts_tablet_manager.cc:1403] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.009s
W20251027 21:07:50.854969 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8fc2d7d6601d4cc2849064f4cdddc90d consensus state has no leader
I20251027 21:07:50.855412 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_69f618a7f3c74a1d87cb5e0c17f38f94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.856012 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69f618a7f3c74a1d87cb5e0c17f38f94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.857041 9307 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_e65d5d66ce8747a9a1a2abfa04cd2855 [id=e2b68fb34c3b48e8952ef6d4201fd93f] alter complete (version 1)
I20251027 21:07:50.857367 10029 raft_consensus.cc:359] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.858651 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_bd23960abfd741b49e67220c481d5d17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.859086 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd23960abfd741b49e67220c481d5d17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.858809 10029 raft_consensus.cc:385] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.859617 10029 raft_consensus.cc:740] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.860666 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_758d1e73474647eaa8a15b6cff350af6" } new_table_name: "default.Table_b779f2756a924ab6ba1bc1702a412e06" modify_external_catalogs: true
I20251027 21:07:50.861259 10029 consensus_queue.cc:260] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.862326 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet cad3a3e910ac42399cf371ad82c89ff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:50.862428 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 61cd8ee8e1f8459eb7af5dd48c018e02 with delete_type TABLET_DATA_DELETED (replaced by tablet 49d9e5c5960a4534a6603875006e6266) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:50.862540 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 87ec5a6b9b3c425da4e96cc6736d6310 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:50.862612 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 94a00ff844f0415fa419613d70a318e6 with delete_type TABLET_DATA_DELETED (replaced by tablet efc0019840244d14bb95ae581793e74b) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:50.862787 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet d8ef795c88da49b58082ba1264d83378 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:50.863389 10241 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:40732 (request call id 171) took 1265 ms. Trace:
I20251027 21:07:50.863674 10029 ts_tablet_manager.cc:1434] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20251027 21:07:50.863512 10241 rpcz_store.cc:276] 1027 21:07:49.597620 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.597659 (+ 39us) service_pool.cc:225] Handling call
1027 21:07:50.863340 (+1265681us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:50.153602 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1027 21:07:50.163058 (+ 9456us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1027 21:07:50.163089 (+ 31us) alter_schema_op.cc:66] Acquired schema lock
1027 21:07:50.163129 (+ 40us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1027 21:07:50.163229 (+ 100us) alter_schema_op.cc:128] START. Timestamp: 1761599270163165
1027 21:07:50.163235 (+ 6us) op_driver.cc:348] REPLICATION: starting
1027 21:07:50.286503 (+123268us) log.cc:844] Serialized 173 byte log entry
1027 21:07:50.314723 (+ 28220us) spinlock_profiling.cc:243] Waited 9.99 ms on lock 0x60c0001a3cc0. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d97c04d65 00007f8d8408dc0e 00007f8d8409104b 00007f8d84090ec8 0000000000a8867c 00007f8d8406cee8 00007f8d876776da 00007f8d801a971e
1027 21:07:50.762393 (+447670us) op_driver.cc:464] REPLICATION: finished
1027 21:07:50.845430 (+ 83037us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1027 21:07:50.848916 (+ 3486us) log.cc:844] Serialized 5 byte log entry
1027 21:07:50.862484 (+ 13568us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.862926 (+ 442us) log.cc:844] Serialized 14 byte log entry
1027 21:07:50.863221 (+ 295us) alter_schema_op.cc:73] Released schema lock
1027 21:07:50.863234 (+ 13us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":82929,"prepare.queue_time_us":450142,"prepare.run_cpu_time_us":1968,"prepare.run_wall_time_us":134987,"raft.queue_time_us":150,"raft.run_cpu_time_us":1040,"raft.run_wall_time_us":61232,"replication_time_us":599093,"spinlock_wait_cycles":22983936,"thread_start_us":419135,"threads_started":2}]]}
I20251027 21:07:50.864858 10241 tablet.cc:1722] T 0ac7111d13b6460a9f4c77e94e09d417 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:07:50.866459 10029 tablet_bootstrap.cc:492] T 9cec6294051b41868c9c7c7fadafa138 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:50.867590 10043 tablet_replica.cc:333] T 6e5d52b7ec114be5ae68e57280c2eea8 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:50.867790 10043 raft_consensus.cc:2243] T 6e5d52b7ec114be5ae68e57280c2eea8 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251027 21:07:50.867906 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9bf3a093be8498c9dd5511b53436b16 consensus state has no leader
I20251027 21:07:50.867959 10043 raft_consensus.cc:2272] T 6e5d52b7ec114be5ae68e57280c2eea8 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.868254 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4bd57498972a413ca2bb7de2cd588e8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:50.868479 10043 ts_tablet_manager.cc:1916] T 6e5d52b7ec114be5ae68e57280c2eea8 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:50.868690 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bd57498972a413ca2bb7de2cd588e8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.869750 10198 raft_consensus.cc:493] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:50.869959 10040 ts_tablet_manager.cc:1929] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:07:50.869912 10198 raft_consensus.cc:515] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.870054 10040 log.cc:1199] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/6e5d52b7ec114be5ae68e57280c2eea8
I20251027 21:07:50.870513 10040 ts_tablet_manager.cc:1950] T 6e5d52b7ec114be5ae68e57280c2eea8 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:50.870651 10198 leader_election.cc:290] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:50.870833 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 166) took 1273 ms. Trace:
I20251027 21:07:50.870932 10040 rpcz_store.cc:276] 1027 21:07:49.596982 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.597102 (+ 120us) service_pool.cc:225] Handling call
1027 21:07:50.766531 (+1169429us) ts_tablet_manager.cc:1151] Deleting tablet 6e5d52b7ec114be5ae68e57280c2eea8
1027 21:07:50.766538 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.840333 (+ 73795us) spinlock_profiling.cc:243] Waited 73.2 ms on lock 0x616000140e54. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d8ff12100 00007f8d9125fae3 00007f8d93e20287 00007f8d93e532d4 00007f8d93e33d70 00007f8d93e33afd 0000000000a8867c 00007f8d8408dd7f 00007f8d8409104b 00007f8d84090ec8
1027 21:07:50.847863 (+ 7530us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.869846 (+ 21983us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.870794 (+ 948us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":168314240,"tablet-delete.queue_time_us":1169206}
I20251027 21:07:50.871341 9923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21b785277d604543bfa7816ed96d6181" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:50.873013 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 6e5d52b7ec114be5ae68e57280c2eea8 (table default.Table_d81917295a3548f787986bf566037719 [id=709b33acbd184a218ec6b700150b58b1]) successfully deleted
I20251027 21:07:50.874308 10041 ts_tablet_manager.cc:1929] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:50.874435 10041 log.cc:1199] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/b1bdb2be5032415187c2826c14e75f07
I20251027 21:07:50.876062 9308 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 21b785277d604543bfa7816ed96d6181 (table default.Table_f1908c85b5144e90a366ca29ba5d1b9b [id=e3de2e5ea6484be7af68eb8601778b85]). Expected version 2 got 0
I20251027 21:07:50.876713 10029 tablet_bootstrap.cc:654] T 9cec6294051b41868c9c7c7fadafa138 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:50.878494 10041 ts_tablet_manager.cc:1950] T b1bdb2be5032415187c2826c14e75f07 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:50.881354 10041 tablet_replica.cc:333] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:50.881525 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet b1bdb2be5032415187c2826c14e75f07 (table default.Table_3e0f707f8e2844bfaf937150b747a935 [id=d7dcec1e03494fd5a219a97ea8b1ab1d]) successfully deleted
I20251027 21:07:50.881598 10041 raft_consensus.cc:2243] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:50.881753 10041 raft_consensus.cc:2272] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.883410 10041 ts_tablet_manager.cc:1916] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:50.886847 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 87ec5a6b9b3c425da4e96cc6736d6310 (table default.Table_0c05919c1a104864a4e1cd182ae59e88 [id=16df16d9dc6144cabc7f453c319255d9]) successfully deleted
W20251027 21:07:50.887112 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 9248f458baca409aa3505cc0db321b34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9248f458baca409aa3505cc0db321b34 already in progress: creating tablet
I20251027 21:07:50.888099 10176 mvcc.cc:204] Tried to move back new op lower bound from 7215510610588323840 to 7215510607683457024. Current Snapshot: MvccSnapshot[applied={T|T < 7215510610588323840}]
W20251027 21:07:50.890340 9711 consensus_peers.cc:597] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9cec6294051b41868c9c7c7fadafa138. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:07:50.890322 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 49d9e5c5960a4534a6603875006e6266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49d9e5c5960a4534a6603875006e6266 already in progress: creating tablet
W20251027 21:07:50.890538 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 9cec6294051b41868c9c7c7fadafa138 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9cec6294051b41868c9c7c7fadafa138 already in progress: creating tablet
W20251027 21:07:50.890810 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 970fbf01a4c3406dafe1ea4cd005ea23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 970fbf01a4c3406dafe1ea4cd005ea23 already in progress: creating tablet
W20251027 21:07:50.891033 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 6f1129eecfb54448965222269d9c32b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f1129eecfb54448965222269d9c32b7 already in progress: creating tablet
I20251027 21:07:50.891803 10217 tablet.cc:1717] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678: Skipping requested alter to schema version 2, tablet already version 2
W20251027 21:07:50.892962 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 9c239216cb8846f0b1aaa360ddb83321 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c239216cb8846f0b1aaa360ddb83321 already in progress: creating tablet
W20251027 21:07:50.893515 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 9882b234e79940088be5f604c41576b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9882b234e79940088be5f604c41576b7 already in progress: creating tablet
I20251027 21:07:50.893915 10241 tablet.cc:1717] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218: Skipping requested alter to schema version 2, tablet already version 2
I20251027 21:07:50.894456 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21b785277d604543bfa7816ed96d6181" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:50.895044 9566 leader_election.cc:304] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:50.895764 10240 tablet.cc:1722] T 40bd202c1b834c949f9a5cee23b3486d P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:07:50.895895 10198 raft_consensus.cc:2749] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:50.896081 10029 tablet_bootstrap.cc:492] T 9cec6294051b41868c9c7c7fadafa138 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:50.896281 10029 ts_tablet_manager.cc:1403] T 9cec6294051b41868c9c7c7fadafa138 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.001s
I20251027 21:07:50.897236 10029 raft_consensus.cc:359] T 9cec6294051b41868c9c7c7fadafa138 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.897436 10029 raft_consensus.cc:385] T 9cec6294051b41868c9c7c7fadafa138 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.897506 10029 raft_consensus.cc:740] T 9cec6294051b41868c9c7c7fadafa138 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.897749 10029 consensus_queue.cc:260] T 9cec6294051b41868c9c7c7fadafa138 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.899443 10029 ts_tablet_manager.cc:1434] T 9cec6294051b41868c9c7c7fadafa138 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:07:50.899945 10040 tablet_replica.cc:333] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:50.899961 10041 ts_tablet_manager.cc:1929] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:50.900089 10041 log.cc:1199] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/35e5b7bbab7f4ee9bbf23d913c0aee76
I20251027 21:07:50.900169 10029 tablet_bootstrap.cc:492] T 44783760defd433a972c9cd83933249c P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:50.900156 10040 raft_consensus.cc:2243] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:50.900315 10040 raft_consensus.cc:2272] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.900399 10041 ts_tablet_manager.cc:1950] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:50.900786 10040 ts_tablet_manager.cc:1916] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:50.901283 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 (table default.Table_e68b56d3ddd94f308243ec6a03ea9e6e [id=0a94e88b69094ffe8be3367470be86da]) successfully deleted
I20251027 21:07:50.908070 10043 ts_tablet_manager.cc:1929] T 6e5d52b7ec114be5ae68e57280c2eea8 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:50.908234 10043 log.cc:1199] T 6e5d52b7ec114be5ae68e57280c2eea8 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/6e5d52b7ec114be5ae68e57280c2eea8
I20251027 21:07:50.908660 10043 ts_tablet_manager.cc:1950] T 6e5d52b7ec114be5ae68e57280c2eea8 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:50.909595 10032 tablet_bootstrap.cc:492] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:50.909741 10032 ts_tablet_manager.cc:1403] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.083s user 0.000s sys 0.003s
I20251027 21:07:50.909890 10029 tablet_bootstrap.cc:654] T 44783760defd433a972c9cd83933249c P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:50.910597 10032 raft_consensus.cc:359] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.912031 10179 raft_consensus.cc:493] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:50.912196 10179 raft_consensus.cc:515] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.912838 10179 leader_election.cc:290] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:50.913287 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94a00ff844f0415fa419613d70a318e6" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:50.913278 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94a00ff844f0415fa419613d70a318e6" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:50.913760 9859 leader_election.cc:343] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:07:50.914268 9860 leader_election.cc:343] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:50.914414 9860 leader_election.cc:304] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:50.914906 10179 raft_consensus.cc:2749] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:50.918097 10040 ts_tablet_manager.cc:1929] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:50.918313 10040 log.cc:1199] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/78a0a8b05f10477c923ab565839141b8
I20251027 21:07:50.918694 10040 ts_tablet_manager.cc:1950] T 78a0a8b05f10477c923ab565839141b8 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:50.919028 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 167) took 1321 ms. Trace:
I20251027 21:07:50.919147 10040 rpcz_store.cc:276] 1027 21:07:49.597080 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.597294 (+ 214us) service_pool.cc:225] Handling call
1027 21:07:50.899856 (+1302562us) ts_tablet_manager.cc:1151] Deleting tablet 78a0a8b05f10477c923ab565839141b8
1027 21:07:50.899864 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.908652 (+ 8788us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.918006 (+ 9354us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.918971 (+ 965us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1302399}
I20251027 21:07:50.919615 10029 tablet_bootstrap.cc:492] T 44783760defd433a972c9cd83933249c P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:50.919783 10029 ts_tablet_manager.cc:1403] T 44783760defd433a972c9cd83933249c P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.001s
I20251027 21:07:50.920509 10029 raft_consensus.cc:359] T 44783760defd433a972c9cd83933249c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.920737 10029 raft_consensus.cc:385] T 44783760defd433a972c9cd83933249c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.920845 10029 raft_consensus.cc:740] T 44783760defd433a972c9cd83933249c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.921092 10029 consensus_queue.cc:260] T 44783760defd433a972c9cd83933249c P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.921579 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 78a0a8b05f10477c923ab565839141b8 (table default.Table_108afd04fe5d45c195ba81996dee61d9 [id=02839636442f4955b245a06a594634c1]) successfully deleted
I20251027 21:07:50.922474 10041 tablet_replica.cc:333] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:50.922492 10029 ts_tablet_manager.cc:1434] T 44783760defd433a972c9cd83933249c P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251027 21:07:50.922657 10041 raft_consensus.cc:2243] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:50.922822 10041 raft_consensus.cc:2272] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.923327 10041 ts_tablet_manager.cc:1916] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:50.923509 10029 tablet_bootstrap.cc:492] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:50.924268 10040 tablet_replica.cc:333] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:50.924434 10040 raft_consensus.cc:2243] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:50.924599 10040 raft_consensus.cc:2272] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.925082 10040 ts_tablet_manager.cc:1916] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:50.925163 10029 tablet_bootstrap.cc:654] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:50.936429 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 6e5d52b7ec114be5ae68e57280c2eea8 (table default.Table_d81917295a3548f787986bf566037719 [id=709b33acbd184a218ec6b700150b58b1]) successfully deleted
W20251027 21:07:50.936705 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet e0968c2f25b64f6685f38a999ee9d60e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0968c2f25b64f6685f38a999ee9d60e already in progress: creating tablet
W20251027 21:07:50.937189 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet c5f2e60caa824b5db2330b423221873c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5f2e60caa824b5db2330b423221873c already in progress: creating tablet
I20251027 21:07:50.912052 10032 raft_consensus.cc:385] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.937565 10032 raft_consensus.cc:740] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.937812 10032 consensus_queue.cc:260] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.938004 10029 tablet_bootstrap.cc:492] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:50.938596 10032 ts_tablet_manager.cc:1434] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20251027 21:07:50.940286 10029 ts_tablet_manager.cc:1403] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20251027 21:07:50.940979 10029 raft_consensus.cc:359] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.941128 10040 ts_tablet_manager.cc:1929] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:50.941159 10029 raft_consensus.cc:385] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.941219 10029 raft_consensus.cc:740] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.941250 10040 log.cc:1199] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/d642da56a67449aa95932180bc5b2c5f
I20251027 21:07:50.941426 10029 consensus_queue.cc:260] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.941567 10040 ts_tablet_manager.cc:1950] T d642da56a67449aa95932180bc5b2c5f P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:50.942267 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 168) took 1344 ms. Trace:
I20251027 21:07:50.942377 10040 rpcz_store.cc:276] 1027 21:07:49.597288 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.597341 (+ 53us) service_pool.cc:225] Handling call
1027 21:07:50.924188 (+1326847us) ts_tablet_manager.cc:1151] Deleting tablet d642da56a67449aa95932180bc5b2c5f
1027 21:07:50.924197 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.928498 (+ 4301us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.941017 (+ 12519us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.942236 (+ 1219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1326718}
I20251027 21:07:50.942708 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet d642da56a67449aa95932180bc5b2c5f (table default.Table_178e5d554faa4e119ac8f52da2d284d7 [id=05edeb649c544c2f899ed42a87cef2c2]) successfully deleted
I20251027 21:07:50.942790 10040 tablet_replica.cc:333] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:50.942970 10040 raft_consensus.cc:2243] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:50.943080 10032 tablet_bootstrap.cc:492] T 81e914ecc3c74b869e9fc3ad102887ec P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:50.943141 10040 raft_consensus.cc:2272] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.943630 10040 ts_tablet_manager.cc:1916] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:50.943694 10029 ts_tablet_manager.cc:1434] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251027 21:07:50.944756 10029 tablet_bootstrap.cc:492] T 0c1a6a7616f24f69b83ec8a282746e92 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:50.945096 10032 tablet_bootstrap.cc:654] T 81e914ecc3c74b869e9fc3ad102887ec P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:50.946709 10029 tablet_bootstrap.cc:654] T 0c1a6a7616f24f69b83ec8a282746e92 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:50.947214 10031 tablet_bootstrap.cc:492] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:50.947408 10031 ts_tablet_manager.cc:1403] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.175s user 0.002s sys 0.008s
I20251027 21:07:50.948218 10031 raft_consensus.cc:359] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.948508 10031 raft_consensus.cc:385] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.948727 10031 raft_consensus.cc:740] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.949033 10031 consensus_queue.cc:260] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.950304 10031 ts_tablet_manager.cc:1434] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251027 21:07:50.951152 10031 tablet_bootstrap.cc:492] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:50.952824 10031 tablet_bootstrap.cc:654] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:50.946084 10041 ts_tablet_manager.cc:1929] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:50.963285 10041 log.cc:1199] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/9c4369a10bf846fb94c9b71a70f21600
I20251027 21:07:50.963694 10041 ts_tablet_manager.cc:1950] T 9c4369a10bf846fb94c9b71a70f21600 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:50.964602 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e07b8aa0a41241d79254c4dae8738e79
I20251027 21:07:50.965446 10029 tablet_bootstrap.cc:492] T 0c1a6a7616f24f69b83ec8a282746e92 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:50.965575 10029 ts_tablet_manager.cc:1403] T 0c1a6a7616f24f69b83ec8a282746e92 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.001s
I20251027 21:07:50.966451 10029 raft_consensus.cc:359] T 0c1a6a7616f24f69b83ec8a282746e92 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.966621 10029 raft_consensus.cc:385] T 0c1a6a7616f24f69b83ec8a282746e92 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.966710 10029 raft_consensus.cc:740] T 0c1a6a7616f24f69b83ec8a282746e92 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.966938 10029 consensus_queue.cc:260] T 0c1a6a7616f24f69b83ec8a282746e92 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:50.967064 9309 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 40bd202c1b834c949f9a5cee23b3486d (table default.Table_7572ec58e5f347bfaf585d1fd0266335 [id=fd2beefa0629466daea4da402b033347]). Expected version 2 got 0
I20251027 21:07:50.969705 10029 ts_tablet_manager.cc:1434] T 0c1a6a7616f24f69b83ec8a282746e92 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251027 21:07:50.969741 10041 tablet_replica.cc:333] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:50.969911 10041 raft_consensus.cc:2243] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:50.970067 10041 raft_consensus.cc:2272] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:50.970643 10041 ts_tablet_manager.cc:1916] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:50.971004 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6bf9c2a78aac42528d67846dc58adc40 consensus state has no leader
I20251027 21:07:50.971349 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_04ef6b3afcc34d84804cd6e9613d7bc2" } modify_external_catalogs: true
I20251027 21:07:50.971494 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_04ef6b3afcc34d84804cd6e9613d7bc2" } modify_external_catalogs: true
I20251027 21:07:50.972070 10029 tablet_bootstrap.cc:492] T 78a0a8b05f10477c923ab565839141b8 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:50.972612 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 9c4369a10bf846fb94c9b71a70f21600 (table default.Table_ec86de5add82433b81e1060e76d559e3 [id=731981b3bb2c4078a597da73199e31d8]) successfully deleted
I20251027 21:07:50.972772 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet e07b8aa0a41241d79254c4dae8738e79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:50.972815 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 6f1129eecfb54448965222269d9c32b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f1129eecfb54448965222269d9c32b7 already in progress: creating tablet
W20251027 21:07:50.972993 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 9c239216cb8846f0b1aaa360ddb83321 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c239216cb8846f0b1aaa360ddb83321 already in progress: creating tablet
W20251027 21:07:50.973129 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet e0968c2f25b64f6685f38a999ee9d60e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0968c2f25b64f6685f38a999ee9d60e already in progress: creating tablet
W20251027 21:07:50.973352 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 9882b234e79940088be5f604c41576b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9882b234e79940088be5f604c41576b7 already in progress: creating tablet
W20251027 21:07:50.973526 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet c5f2e60caa824b5db2330b423221873c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5f2e60caa824b5db2330b423221873c already in progress: creating tablet
W20251027 21:07:50.973699 9299 catalog_manager.cc:4952] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet cad3a3e910ac42399cf371ad82c89ff7 because the tablet was not found. No further retry: Not found: Tablet not found: cad3a3e910ac42399cf371ad82c89ff7
W20251027 21:07:50.974181 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7aa8c538b6c849a28a28ffe706306891 consensus state has no leader
I20251027 21:07:50.974227 10029 tablet_bootstrap.cc:654] T 78a0a8b05f10477c923ab565839141b8 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:50.974538 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_ac66ea1435fe49518c073dfc9b2368c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.975061 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac66ea1435fe49518c073dfc9b2368c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.976702 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_11d4ee667e1c4d9badc00349e1c61463"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.977144 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11d4ee667e1c4d9badc00349e1c61463 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.980787 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_34189a0358a840a19f81cdf18c207177"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:50.981232 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34189a0358a840a19f81cdf18c207177 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.983696 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet c400101d2a39490e83722500839198cd from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:50.987807 10029 tablet_bootstrap.cc:492] T 78a0a8b05f10477c923ab565839141b8 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:50.987952 10029 ts_tablet_manager.cc:1403] T 78a0a8b05f10477c923ab565839141b8 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.005s
I20251027 21:07:50.988682 10029 raft_consensus.cc:359] T 78a0a8b05f10477c923ab565839141b8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.988844 10029 raft_consensus.cc:385] T 78a0a8b05f10477c923ab565839141b8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:50.988915 10029 raft_consensus.cc:740] T 78a0a8b05f10477c923ab565839141b8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:50.989182 10029 consensus_queue.cc:260] T 78a0a8b05f10477c923ab565839141b8 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:50.991533 9308 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_7572ec58e5f347bfaf585d1fd0266335 [id=fd2beefa0629466daea4da402b033347] alter complete (version 2)
I20251027 21:07:50.992069 9308 master_service.cc:954] ReplaceTablet: received request to replace tablet 87ec5a6b9b3c425da4e96cc6736d6310 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:50.992383 10029 ts_tablet_manager.cc:1434] T 78a0a8b05f10477c923ab565839141b8 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251027 21:07:50.992508 9308 master_service.cc:954] ReplaceTablet: received request to replace tablet d642da56a67449aa95932180bc5b2c5f from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:50.994112 9307 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_f1908c85b5144e90a366ca29ba5d1b9b [id=e3de2e5ea6484be7af68eb8601778b85] alter complete (version 2)
I20251027 21:07:50.994622 10029 tablet_bootstrap.cc:492] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:50.996397 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:50.996634 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 36e168cc44274dae9682b0c1329d4558 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:50.997247 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d659e4d73ea4ff3975a4b58a794090a with delete_type TABLET_DATA_DELETED (replaced by tablet 6bf9c2a78aac42528d67846dc58adc40) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:50.998090 10029 tablet_bootstrap.cc:654] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:50.998615 9631 raft_consensus.cc:3060] T 9cec6294051b41868c9c7c7fadafa138 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:51.006012 10029 tablet_bootstrap.cc:492] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:51.006227 10029 ts_tablet_manager.cc:1403] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.003s
W20251027 21:07:51.007225 9298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.125:46313, user_credentials={real_user=slave}} blocked reactor thread for 46743.2us
I20251027 21:07:51.007259 10029 raft_consensus.cc:359] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:51.007426 10029 raft_consensus.cc:385] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.007513 10029 raft_consensus.cc:740] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.007772 10029 consensus_queue.cc:260] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:51.008034 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet e07b8aa0a41241d79254c4dae8738e79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.008344 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 9cec6294051b41868c9c7c7fadafa138 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.009073 10043 tablet_replica.cc:333] T 9cec6294051b41868c9c7c7fadafa138 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:51.009255 10043 raft_consensus.cc:2243] T 9cec6294051b41868c9c7c7fadafa138 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.009456 10043 raft_consensus.cc:2272] T 9cec6294051b41868c9c7c7fadafa138 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.010046 10041 ts_tablet_manager.cc:1929] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.010273 10041 log.cc:1199] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/61cd8ee8e1f8459eb7af5dd48c018e02
I20251027 21:07:51.010660 10041 ts_tablet_manager.cc:1950] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:51.011372 10043 ts_tablet_manager.cc:1916] T 9cec6294051b41868c9c7c7fadafa138 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.011668 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 61cd8ee8e1f8459eb7af5dd48c018e02 (table default.Table_52c53bafe66b4c66b99e5b26e4df7ef5 [id=27dc835f316f41e091be9b2b1f560bbf]) successfully deleted
W20251027 21:07:51.015889 9299 catalog_manager.cc:4952] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 87ec5a6b9b3c425da4e96cc6736d6310 because the tablet was not found. No further retry: Not found: Tablet not found: 87ec5a6b9b3c425da4e96cc6736d6310
I20251027 21:07:51.016208 10041 tablet_replica.cc:333] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:51.016400 10041 raft_consensus.cc:2243] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.016567 10041 raft_consensus.cc:2272] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.017088 10041 ts_tablet_manager.cc:1916] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.018774 10043 ts_tablet_manager.cc:1929] T 9cec6294051b41868c9c7c7fadafa138 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.018891 10043 log.cc:1199] T 9cec6294051b41868c9c7c7fadafa138 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/9cec6294051b41868c9c7c7fadafa138
I20251027 21:07:51.019255 10043 ts_tablet_manager.cc:1950] T 9cec6294051b41868c9c7c7fadafa138 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:51.020119 10029 ts_tablet_manager.cc:1434] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251027 21:07:51.022433 10029 tablet_bootstrap.cc:492] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:51.025071 10029 tablet_bootstrap.cc:654] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:51.028175 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 78a0a8b05f10477c923ab565839141b8 with delete_type TABLET_DATA_DELETED (replaced by tablet 1698ae8a37ad4560836b3b1daca1cf5e) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.028348 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet 44783760defd433a972c9cd83933249c with delete_type TABLET_DATA_DELETED (replaced by tablet ffe1182e8f4e4a44b9a12e0a5192989f) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.028453 10043 tablet_replica.cc:333] T 78a0a8b05f10477c923ab565839141b8 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:51.028649 10043 raft_consensus.cc:2243] T 78a0a8b05f10477c923ab565839141b8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.028807 10043 raft_consensus.cc:2272] T 78a0a8b05f10477c923ab565839141b8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.029587 10043 ts_tablet_manager.cc:1916] T 78a0a8b05f10477c923ab565839141b8 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.029698 10297 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.14.124:45071 every 8 connection(s)
I20251027 21:07:51.029856 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet b3dfef8b741c42f68b664d7e0eab49c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.030395 10108 rpc_server.cc:307] RPC server started. Bound to: 127.9.14.124:45071
I20251027 21:07:51.030398 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet 69b1fb42428748239d8939d98428d8fe with delete_type TABLET_DATA_DELETED (replaced by tablet 49b3d23798b740008c98cd0754ae33f4) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:51.030704 9296 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 8fc2d7d6601d4cc2849064f4cdddc90d (table default.Table_caf93d0298314165abb645155ffd1a41 [id=6f39910d23dd455f87a4ac847917b87f], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8fc2d7d6601d4cc2849064f4cdddc90d consensus state has no leader
I20251027 21:07:51.031181 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c1a6a7616f24f69b83ec8a282746e92 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d659e4d73ea4ff3975a4b58a794090a) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.032397 10108 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data/info.pb
I20251027 21:07:51.035876 10043 ts_tablet_manager.cc:1929] T 78a0a8b05f10477c923ab565839141b8 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.036695 10043 log.cc:1199] T 78a0a8b05f10477c923ab565839141b8 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/78a0a8b05f10477c923ab565839141b8
I20251027 21:07:51.037062 10043 ts_tablet_manager.cc:1950] T 78a0a8b05f10477c923ab565839141b8 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:51.040591 9273 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu as pid 10108
I20251027 21:07:51.046056 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9904948d307e479ea184d6dc441b5e61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.046720 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9904948d307e479ea184d6dc441b5e61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.048100 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2f29d30ca47244898036f6030063a5c6" } new_table_name: "default.Table_5d7b7b6b754149609ebbb51ebf00b364" modify_external_catalogs: true
I20251027 21:07:51.037781 10029 tablet_bootstrap.cc:492] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:51.041532 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet cbada3946dab4739a8369032ff3b8a48 with delete_type TABLET_DATA_DELETED (replaced by tablet 4e8c83b501d8426fa0cad6a89f9a0b9b) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.048583 10029 ts_tablet_manager.cc:1403] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.005s
I20251027 21:07:51.049544 10029 raft_consensus.cc:359] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.049816 10029 raft_consensus.cc:385] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.049971 10029 raft_consensus.cc:740] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.050351 10029 consensus_queue.cc:260] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.050602 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_44053023614445a79986595acb8e2e60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.051852 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44053023614445a79986595acb8e2e60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.052688 10029 ts_tablet_manager.cc:1434] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251027 21:07:51.053040 10029 tablet_bootstrap.cc:492] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:51.057271 10029 tablet_bootstrap.cc:654] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:51.057499 9309 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 33e41145c79b4465b8d564741174b8e0 (table default.Table_e4d0e9336a834d6d8ab29ac0b1de7ace [id=af5b5b94fcef4677affe7e7afaa7f023]). Expected version 1 got 0
I20251027 21:07:51.058036 10043 tablet_replica.cc:333] T 44783760defd433a972c9cd83933249c P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:51.058534 10043 raft_consensus.cc:2243] T 44783760defd433a972c9cd83933249c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.058966 10043 raft_consensus.cc:2272] T 44783760defd433a972c9cd83933249c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.059723 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c6c89ff2d21c4e4db430817f6398887e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:51.061022 10043 ts_tablet_manager.cc:1916] T 44783760defd433a972c9cd83933249c P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.063369 10041 ts_tablet_manager.cc:1929] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.063505 10041 log.cc:1199] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/94a00ff844f0415fa419613d70a318e6
I20251027 21:07:51.063859 10041 ts_tablet_manager.cc:1950] T 94a00ff844f0415fa419613d70a318e6 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:51.065232 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 94a00ff844f0415fa419613d70a318e6 (table default.Table_8d288192611b43af971f1e081bb362ad [id=f773d65a07a4473ab8b87cd82ac50bcf]) successfully deleted
I20251027 21:07:51.071028 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 22d861810ed846c3b77a452d05e93d72 (DEFAULT_TABLE table=default.Table_94d4f5b9325a4c27a6305962fe774808 [id=c92fda1282d7419f94b0004ab8b8403d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:51.071112 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6c89ff2d21c4e4db430817f6398887e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.071472 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22d861810ed846c3b77a452d05e93d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.071417 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 630ae4d66f99474382814cc32cc60ea0 (DEFAULT_TABLE table=default.Table_12da35feb05c42a79377049e12fe73d6 [id=8a26e744abae4c5eb1d18dd994cd6bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.071782 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 630ae4d66f99474382814cc32cc60ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.073391 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f1908c85b5144e90a366ca29ba5d1b9b" } modify_external_catalogs: true
I20251027 21:07:51.073545 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f1908c85b5144e90a366ca29ba5d1b9b" } modify_external_catalogs: true
I20251027 21:07:51.074391 9904 tablet_service.cc:1505] Processing CreateTablet for tablet ccf10ef559f74e2db28023fd520c8de9 (DEFAULT_TABLE table=default.Table_17e67d6acbdd470d9b351522b7908579 [id=bbdedeb45c464c47b96d078308141f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.074720 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccf10ef559f74e2db28023fd520c8de9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.077596 9308 master_service.cc:954] ReplaceTablet: received request to replace tablet 21a8fc4a69f3486e92b1a68d10faed38 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:51.078459 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 24ec2e0ba1994ab59c53977583bd2058 (DEFAULT_TABLE table=default.Table_2992b8e207024e06957ce7befb657acb [id=decffeb1481e4d798cffd8c9883be59b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.078954 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 9cec6294051b41868c9c7c7fadafa138 (table default.Table_5d8e4122af254070a2df7016e521a937 [id=12b1bbc770194455ab7292641980e8e1]) successfully deleted
I20251027 21:07:51.079545 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 6f1d1bd52556463fa306e51eb253af40 (DEFAULT_TABLE table=default.Table_9b06fbc2554c4780a754ed9cfb964fa9 [id=47a1de450a474742ae41e277716f5381]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:51.079689 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet e07b8aa0a41241d79254c4dae8738e79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e07b8aa0a41241d79254c4dae8738e79 already in progress: creating tablet
I20251027 21:07:51.079897 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 78a0a8b05f10477c923ab565839141b8 (table default.Table_108afd04fe5d45c195ba81996dee61d9 [id=02839636442f4955b245a06a594634c1]) successfully deleted
I20251027 21:07:51.079957 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 345d3795a37b4a40b40967cfacf0f893 (DEFAULT_TABLE table=default.Table_ed32ba2bd26544e1a9165f6246ee5cb9 [id=93d6307da3a34f87a67018a06825cd15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.080168 9901 tablet_service.cc:1505] Processing CreateTablet for tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa (DEFAULT_TABLE table=default.Table_8fa71eed37b040fcb5f63d829b4a6315 [id=de5974f2ae38479fa2d403d5ac6e1dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.080395 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 84b9de82d55447bba2a897edb683e5c5 (DEFAULT_TABLE table=default.Table_c73837b0d9514a8980a8a70c61ae8019 [id=6b70bbbf477d4b788b3ffb10c9c7d7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.080650 9900 tablet_service.cc:1505] Processing CreateTablet for tablet dd7029986a5246ee823e5fdb50683118 (DEFAULT_TABLE table=default.Table_79211c9993c54e4ea0455d01fec68b8e [id=421a64f7fb6f499bbf6f20fb38f736d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.081882 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet 65fbcaa5447c48818fabfccac22c4048 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.082326 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ec2e0ba1994ab59c53977583bd2058. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.082432 9612 tablet_service.cc:1505] Processing CreateTablet for tablet 630ae4d66f99474382814cc32cc60ea0 (DEFAULT_TABLE table=default.Table_12da35feb05c42a79377049e12fe73d6 [id=8a26e744abae4c5eb1d18dd994cd6bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.082753 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 630ae4d66f99474382814cc32cc60ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.082751 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd7029986a5246ee823e5fdb50683118. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.083194 9901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.083552 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84b9de82d55447bba2a897edb683e5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.083938 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 345d3795a37b4a40b40967cfacf0f893. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.084333 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f1d1bd52556463fa306e51eb253af40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.084414 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 84b9de82d55447bba2a897edb683e5c5 (DEFAULT_TABLE table=default.Table_c73837b0d9514a8980a8a70c61ae8019 [id=6b70bbbf477d4b788b3ffb10c9c7d7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.084730 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84b9de82d55447bba2a897edb683e5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.086531 9610 tablet_service.cc:1505] Processing CreateTablet for tablet ccf10ef559f74e2db28023fd520c8de9 (DEFAULT_TABLE table=default.Table_17e67d6acbdd470d9b351522b7908579 [id=bbdedeb45c464c47b96d078308141f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.086856 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccf10ef559f74e2db28023fd520c8de9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.090665 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 87d5af4607f04f92abcc7292b4176cbd (DEFAULT_TABLE table=default.Table_2f29d30ca47244898036f6030063a5c6 [id=cdb023218cda4e9ea04a226b37a220d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.090970 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87d5af4607f04f92abcc7292b4176cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.094489 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 53cac7817db440e79e7416d625a9b2c8 (DEFAULT_TABLE table=default.Table_79a3420c8b6542868231c740293ab1fb [id=f3babbbf131543d2813ed9d5882e3d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.094825 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53cac7817db440e79e7416d625a9b2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.095986 10032 tablet_bootstrap.cc:492] T 81e914ecc3c74b869e9fc3ad102887ec P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:51.096168 10032 ts_tablet_manager.cc:1403] T 81e914ecc3c74b869e9fc3ad102887ec P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.153s user 0.001s sys 0.003s
I20251027 21:07:51.097097 10032 raft_consensus.cc:359] T 81e914ecc3c74b869e9fc3ad102887ec P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.097316 10032 raft_consensus.cc:385] T 81e914ecc3c74b869e9fc3ad102887ec P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.097406 10032 raft_consensus.cc:740] T 81e914ecc3c74b869e9fc3ad102887ec P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.097672 10032 consensus_queue.cc:260] T 81e914ecc3c74b869e9fc3ad102887ec P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.097954 10178 raft_consensus.cc:493] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:51.098301 10032 ts_tablet_manager.cc:1434] T 81e914ecc3c74b869e9fc3ad102887ec P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:51.098752 10032 tablet_bootstrap.cc:492] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:07:51.100643 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 87d5af4607f04f92abcc7292b4176cbd consensus state has no leader
I20251027 21:07:51.100594 9608 tablet_service.cc:1505] Processing CreateTablet for tablet d872cf3ef39042ff9a615b55e7195afa (DEFAULT_TABLE table=default.Table_6f1e2491cbc546a7b54ab25da6ebf334 [id=59de715802f547bca16f181917fcc418]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.100948 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d872cf3ef39042ff9a615b55e7195afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.101074 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f802ec6bccdf4ba897356686e0210911" } new_table_name: "default.Table_3895a18e2b7a4456a3681679d7c8ba4f" modify_external_catalogs: true
I20251027 21:07:51.102452 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 343eaeca24dc4816ae032c7e674e9412
I20251027 21:07:51.102670 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_13a109a52d1a42aca531982896796ff7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.103117 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13a109a52d1a42aca531982896796ff7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.103719 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet d8e008c868d04281beba0e5b204eb69a
I20251027 21:07:51.104017 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 21a8fc4a69f3486e92b1a68d10faed38
I20251027 21:07:51.103046 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_bd23960abfd741b49e67220c481d5d17" } new_table_name: "default.Table_4c5fcd020732482597349bbb07468bf5" modify_external_catalogs: true
I20251027 21:07:51.104817 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c400101d2a39490e83722500839198cd
I20251027 21:07:51.104861 10178 raft_consensus.cc:515] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:51.105244 9311 catalog_manager.cc:6719] ReplaceTablet: tablet c400101d2a39490e83722500839198cd deleted and replaced by tablet c9efffa9766e46aca8eb286fc2ad48d2
I20251027 21:07:51.105594 10178 leader_election.cc:290] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:51.106581 10032 tablet_bootstrap.cc:654] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:51.110772 9886 tablet_service.cc:1552] Processing DeleteTablet for tablet 21a8fc4a69f3486e92b1a68d10faed38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.110977 9888 tablet_service.cc:1552] Processing DeleteTablet for tablet c400101d2a39490e83722500839198cd with delete_type TABLET_DATA_DELETED (replaced by tablet c9efffa9766e46aca8eb286fc2ad48d2) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.111040 9886 tablet_service.cc:1552] Processing DeleteTablet for tablet 81e914ecc3c74b869e9fc3ad102887ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.111070 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet d8e008c868d04281beba0e5b204eb69a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.110627 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 343eaeca24dc4816ae032c7e674e9412 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.111629 9889 tablet_service.cc:1505] Processing CreateTablet for tablet d872cf3ef39042ff9a615b55e7195afa (DEFAULT_TABLE table=default.Table_6f1e2491cbc546a7b54ab25da6ebf334 [id=59de715802f547bca16f181917fcc418]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.111967 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d872cf3ef39042ff9a615b55e7195afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.114296 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 21b785277d604543bfa7816ed96d6181
I20251027 21:07:51.116068 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 87d5af4607f04f92abcc7292b4176cbd (DEFAULT_TABLE table=default.Table_2f29d30ca47244898036f6030063a5c6 [id=cdb023218cda4e9ea04a226b37a220d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.117434 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 24ec2e0ba1994ab59c53977583bd2058 (DEFAULT_TABLE table=default.Table_2992b8e207024e06957ce7befb657acb [id=decffeb1481e4d798cffd8c9883be59b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.117866 9598 tablet_service.cc:1505] Processing CreateTablet for tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa (DEFAULT_TABLE table=default.Table_8fa71eed37b040fcb5f63d829b4a6315 [id=de5974f2ae38479fa2d403d5ac6e1dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.118531 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 22d861810ed846c3b77a452d05e93d72 (DEFAULT_TABLE table=default.Table_94d4f5b9325a4c27a6305962fe774808 [id=c92fda1282d7419f94b0004ab8b8403d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.119851 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8ef795c88da49b58082ba1264d83378" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:51.120488 9860 leader_election.cc:343] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:51.120813 9611 tablet_service.cc:1505] Processing CreateTablet for tablet dd7029986a5246ee823e5fdb50683118 (DEFAULT_TABLE table=default.Table_79211c9993c54e4ea0455d01fec68b8e [id=421a64f7fb6f499bbf6f20fb38f736d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.121167 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 345d3795a37b4a40b40967cfacf0f893 (DEFAULT_TABLE table=default.Table_ed32ba2bd26544e1a9165f6246ee5cb9 [id=93d6307da3a34f87a67018a06825cd15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.121671 10299 sys_catalog.cc:263] Verifying existing consensus state
I20251027 21:07:51.121999 9604 tablet_service.cc:1505] Processing CreateTablet for tablet 6f1d1bd52556463fa306e51eb253af40 (DEFAULT_TABLE table=default.Table_9b06fbc2554c4780a754ed9cfb964fa9 [id=47a1de450a474742ae41e277716f5381]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.122370 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87d5af4607f04f92abcc7292b4176cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.122840 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 26872be132fb46ee9e79a9c2e62374ef (DEFAULT_TABLE table=default.Table_b23313ec4d7b4e32a4283dd404578616 [id=6a545a959eb74d0eb589f1be92a0bc0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.123478 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 53cac7817db440e79e7416d625a9b2c8 (DEFAULT_TABLE table=default.Table_79a3420c8b6542868231c740293ab1fb [id=f3babbbf131543d2813ed9d5882e3d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.124119 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 345d3795a37b4a40b40967cfacf0f893. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.125366 10041 tablet_replica.cc:333] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:51.125633 10041 raft_consensus.cc:2243] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.125824 10041 raft_consensus.cc:2272] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.126538 10041 ts_tablet_manager.cc:1916] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.135627 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet 21b785277d604543bfa7816ed96d6181 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.158166 10299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: Bootstrap starting.
I20251027 21:07:51.152954 10041 ts_tablet_manager.cc:1929] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.161772 10041 log.cc:1199] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/d8ef795c88da49b58082ba1264d83378
I20251027 21:07:51.162209 10041 ts_tablet_manager.cc:1950] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:51.163075 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet d8ef795c88da49b58082ba1264d83378 (table default.Table_df95aebbdcf94d8289b235722b7d2c8c [id=98143d42942742568f2c3a1d0fbcc100]) successfully deleted
W20251027 21:07:51.164892 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet e07b8aa0a41241d79254c4dae8738e79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e07b8aa0a41241d79254c4dae8738e79 already in progress: creating tablet
I20251027 21:07:51.166028 10041 tablet_replica.cc:333] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:51.166252 10041 raft_consensus.cc:2243] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.166435 10041 raft_consensus.cc:2272] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.168843 10041 ts_tablet_manager.cc:1916] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.176913 9970 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:07:51.179831 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53cac7817db440e79e7416d625a9b2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.187525 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6df2aec5939c4b3cbfb82b376fe5b53d" } new_table_name: "default.Table_1d1b4e58d3da402188147c3d83dbe958" modify_external_catalogs: true
I20251027 21:07:51.187588 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5fe0ef16a24f4af9ae8e843adbe1643e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.188108 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fe0ef16a24f4af9ae8e843adbe1643e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.189446 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b547b70a001f404fb96dc4f8fa6d70e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.190023 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b547b70a001f404fb96dc4f8fa6d70e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.191606 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_34189a0358a840a19f81cdf18c207177" } new_table_name: "default.Table_dfa3f51576fa4c689b2682b0e277df60" modify_external_catalogs: true
I20251027 21:07:51.193427 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4a455030bbe945c391d97696ee50010e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.193965 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a455030bbe945c391d97696ee50010e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.199429 10041 ts_tablet_manager.cc:1929] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.199625 10041 log.cc:1199] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/cd57440d3b2a4f37a8db7d1fa4bc4dbe
I20251027 21:07:51.200160 10041 ts_tablet_manager.cc:1950] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:51.201383 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe (table default.Table_4c5433970bd64291a2b1289a84f35590 [id=a1b2d46db3764c7f97712a0c5f891881]) successfully deleted
I20251027 21:07:51.201462 10041 tablet_replica.cc:333] T 36e168cc44274dae9682b0c1329d4558 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:51.201712 10041 raft_consensus.cc:2243] T 36e168cc44274dae9682b0c1329d4558 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.202553 10041 raft_consensus.cc:2272] T 36e168cc44274dae9682b0c1329d4558 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.202656 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e65d5d66ce8747a9a1a2abfa04cd2855" } modify_external_catalogs: true
I20251027 21:07:51.203183 10041 ts_tablet_manager.cc:1916] T 36e168cc44274dae9682b0c1329d4558 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.203287 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e65d5d66ce8747a9a1a2abfa04cd2855" } modify_external_catalogs: true
I20251027 21:07:51.204982 10299 log.cc:826] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: Log is configured to *not* fsync() on all Append() calls
I20251027 21:07:51.211764 10041 ts_tablet_manager.cc:1929] T 36e168cc44274dae9682b0c1329d4558 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.212016 10041 log.cc:1199] T 36e168cc44274dae9682b0c1329d4558 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/36e168cc44274dae9682b0c1329d4558
I20251027 21:07:51.212528 10041 ts_tablet_manager.cc:1950] T 36e168cc44274dae9682b0c1329d4558 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:51.213603 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 36e168cc44274dae9682b0c1329d4558 (table default.Table_795e30871da94371958578747d1c898b [id=9a111e9480b84a55b5226f0940d999c4]) successfully deleted
I20251027 21:07:51.213688 10041 tablet_replica.cc:333] T 2d659e4d73ea4ff3975a4b58a794090a P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:51.213974 10041 raft_consensus.cc:2243] T 2d659e4d73ea4ff3975a4b58a794090a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.214254 10041 raft_consensus.cc:2272] T 2d659e4d73ea4ff3975a4b58a794090a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.215003 10041 ts_tablet_manager.cc:1916] T 2d659e4d73ea4ff3975a4b58a794090a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.222893 10041 ts_tablet_manager.cc:1929] T 2d659e4d73ea4ff3975a4b58a794090a P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.223161 10041 log.cc:1199] T 2d659e4d73ea4ff3975a4b58a794090a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/2d659e4d73ea4ff3975a4b58a794090a
I20251027 21:07:51.223690 10041 ts_tablet_manager.cc:1950] T 2d659e4d73ea4ff3975a4b58a794090a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:51.224771 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 2d659e4d73ea4ff3975a4b58a794090a (table default.Table_e9fd9bee6a2a40d7ad92d2d31b03afce [id=af6cb904f8cd4a929be7a978f9bc2001]) successfully deleted
W20251027 21:07:51.225344 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 21a8fc4a69f3486e92b1a68d10faed38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21a8fc4a69f3486e92b1a68d10faed38 already in progress: creating tablet
W20251027 21:07:51.225661 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet c400101d2a39490e83722500839198cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c400101d2a39490e83722500839198cd already in progress: creating tablet
I20251027 21:07:51.225719 10041 tablet_replica.cc:333] T 81e914ecc3c74b869e9fc3ad102887ec P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:51.226118 10041 raft_consensus.cc:2243] T 81e914ecc3c74b869e9fc3ad102887ec P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.226403 10041 raft_consensus.cc:2272] T 81e914ecc3c74b869e9fc3ad102887ec P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.227234 10041 ts_tablet_manager.cc:1916] T 81e914ecc3c74b869e9fc3ad102887ec P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.235273 10041 ts_tablet_manager.cc:1929] T 81e914ecc3c74b869e9fc3ad102887ec P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.235438 10041 log.cc:1199] T 81e914ecc3c74b869e9fc3ad102887ec P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/81e914ecc3c74b869e9fc3ad102887ec
I20251027 21:07:51.235850 10041 ts_tablet_manager.cc:1950] T 81e914ecc3c74b869e9fc3ad102887ec P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:51.236840 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 81e914ecc3c74b869e9fc3ad102887ec (table default.Table_a7f999b292d04332b831f454d4aa6743 [id=a6803bcbb3c2427992f31518cc1de040]) successfully deleted
W20251027 21:07:51.237229 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet d8e008c868d04281beba0e5b204eb69a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8e008c868d04281beba0e5b204eb69a already in progress: creating tablet
I20251027 21:07:51.237434 10041 tablet_replica.cc:333] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
W20251027 21:07:51.237540 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 343eaeca24dc4816ae032c7e674e9412 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 343eaeca24dc4816ae032c7e674e9412 already in progress: creating tablet
I20251027 21:07:51.237779 10041 raft_consensus.cc:2243] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.238085 10041 raft_consensus.cc:2272] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.239041 10041 ts_tablet_manager.cc:1916] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.245443 10178 raft_consensus.cc:493] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:51.245615 10178 raft_consensus.cc:515] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.246424 10178 leader_election.cc:290] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:51.249504 10041 ts_tablet_manager.cc:1929] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251027 21:07:51.249678 10041 log.cc:1199] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/21b785277d604543bfa7816ed96d6181
I20251027 21:07:51.250180 10041 ts_tablet_manager.cc:1950] T 21b785277d604543bfa7816ed96d6181 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:51.254382 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 21b785277d604543bfa7816ed96d6181 (table default.Table_f1908c85b5144e90a366ca29ba5d1b9b [id=e3de2e5ea6484be7af68eb8601778b85]) successfully deleted
I20251027 21:07:51.255890 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_979a3d9dccc74a86a0cf5c725bd8cf2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.256634 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_979a3d9dccc74a86a0cf5c725bd8cf2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:50.957185 10040 ts_tablet_manager.cc:1929] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.258553 10040 log.cc:1199] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/5c87a5450d354506852c4b20bf0541d2
I20251027 21:07:51.258991 10040 ts_tablet_manager.cc:1950] T 5c87a5450d354506852c4b20bf0541d2 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:51.259384 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 169) took 1661 ms. Trace:
I20251027 21:07:51.259479 10040 rpcz_store.cc:276] 1027 21:07:49.597396 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.597466 (+ 70us) service_pool.cc:225] Handling call
1027 21:07:50.942735 (+1345269us) ts_tablet_manager.cc:1151] Deleting tablet 5c87a5450d354506852c4b20bf0541d2
1027 21:07:50.942744 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:50.953870 (+ 11126us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:50.957092 (+ 3222us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:51.259319 (+302227us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1345088}
I20251027 21:07:51.259841 10040 tablet_replica.cc:333] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:50.984319 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet e07b8aa0a41241d79254c4dae8738e79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.260053 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 21b785277d604543bfa7816ed96d6181 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:50.997805 9754 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e5d52b7ec114be5ae68e57280c2eea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.002501 10143 consensus_queue.cc:1048] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:07:51.260560 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 5c87a5450d354506852c4b20bf0541d2 (table default.Table_d1fb7f4ef09e4bfdac00a75f752f0873 [id=1cf515b159954979acd73a0b276e7503]) successfully deleted
I20251027 21:07:51.002944 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet e3c9f64ee0544e59add4f73bca59df98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.069051 9741 tablet_service.cc:1505] Processing CreateTablet for tablet 630ae4d66f99474382814cc32cc60ea0 (DEFAULT_TABLE table=default.Table_12da35feb05c42a79377049e12fe73d6 [id=8a26e744abae4c5eb1d18dd994cd6bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.069936 9748 tablet_service.cc:1505] Processing CreateTablet for tablet ccf10ef559f74e2db28023fd520c8de9 (DEFAULT_TABLE table=default.Table_17e67d6acbdd470d9b351522b7908579 [id=bbdedeb45c464c47b96d078308141f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.070725 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 24ec2e0ba1994ab59c53977583bd2058 (DEFAULT_TABLE table=default.Table_2992b8e207024e06957ce7befb657acb [id=decffeb1481e4d798cffd8c9883be59b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.072395 9750 tablet_service.cc:1505] Processing CreateTablet for tablet d872cf3ef39042ff9a615b55e7195afa (DEFAULT_TABLE table=default.Table_6f1e2491cbc546a7b54ab25da6ebf334 [id=59de715802f547bca16f181917fcc418]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.074180 9744 tablet_service.cc:1505] Processing CreateTablet for tablet 6f1d1bd52556463fa306e51eb253af40 (DEFAULT_TABLE table=default.Table_9b06fbc2554c4780a754ed9cfb964fa9 [id=47a1de450a474742ae41e277716f5381]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.075767 9743 tablet_service.cc:1505] Processing CreateTablet for tablet dd7029986a5246ee823e5fdb50683118 (DEFAULT_TABLE table=default.Table_79211c9993c54e4ea0455d01fec68b8e [id=421a64f7fb6f499bbf6f20fb38f736d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.076898 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 345d3795a37b4a40b40967cfacf0f893 (DEFAULT_TABLE table=default.Table_ed32ba2bd26544e1a9165f6246ee5cb9 [id=93d6307da3a34f87a67018a06825cd15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.078714 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 84b9de82d55447bba2a897edb683e5c5 (DEFAULT_TABLE table=default.Table_c73837b0d9514a8980a8a70c61ae8019 [id=6b70bbbf477d4b788b3ffb10c9c7d7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.079427 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa (DEFAULT_TABLE table=default.Table_8fa71eed37b040fcb5f63d829b4a6315 [id=de5974f2ae38479fa2d403d5ac6e1dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.080626 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 87d5af4607f04f92abcc7292b4176cbd (DEFAULT_TABLE table=default.Table_2f29d30ca47244898036f6030063a5c6 [id=cdb023218cda4e9ea04a226b37a220d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.094524 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 53cac7817db440e79e7416d625a9b2c8 (DEFAULT_TABLE table=default.Table_79a3420c8b6542868231c740293ab1fb [id=f3babbbf131543d2813ed9d5882e3d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.095604 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 26872be132fb46ee9e79a9c2e62374ef (DEFAULT_TABLE table=default.Table_b23313ec4d7b4e32a4283dd404578616 [id=6a545a959eb74d0eb589f1be92a0bc0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.106173 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8ef795c88da49b58082ba1264d83378" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:51.109269 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet 343eaeca24dc4816ae032c7e674e9412 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.109423 9745 tablet_service.cc:1552] Processing DeleteTablet for tablet d8e008c868d04281beba0e5b204eb69a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.109540 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet 21a8fc4a69f3486e92b1a68d10faed38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.109673 9737 tablet_service.cc:1552] Processing DeleteTablet for tablet c400101d2a39490e83722500839198cd with delete_type TABLET_DATA_DELETED (replaced by tablet c9efffa9766e46aca8eb286fc2ad48d2) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:51.264317 9859 leader_election.cc:343] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:51.264483 9859 leader_election.cc:304] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:07:51.264734 9859 raft_consensus.cc:2686] T d8ef795c88da49b58082ba1264d83378 P 0fbd567f929a4f6d8e4d78feef7ed678: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251027 21:07:51.250962 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "33e41145c79b4465b8d564741174b8e0" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:51.255395 10141 raft_consensus.cc:493] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:51.266746 10141 raft_consensus.cc:515] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.267656 10141 leader_election.cc:290] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
W20251027 21:07:51.267992 9859 leader_election.cc:343] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:51.269383 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet d8e008c868d04281beba0e5b204eb69a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.270335 9631 raft_consensus.cc:3060] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:07:51.270540 9711 leader_election.cc:343] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: e3c9f64ee0544e59add4f73bca59df98
I20251027 21:07:51.271261 10040 raft_consensus.cc:2243] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.271476 10040 raft_consensus.cc:2272] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.272073 10040 ts_tablet_manager.cc:1916] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.272080 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 343eaeca24dc4816ae032c7e674e9412 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.272286 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "33e41145c79b4465b8d564741174b8e0" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:51.275696 9860 leader_election.cc:343] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:51.275864 9860 leader_election.cc:304] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:51.276345 10178 raft_consensus.cc:2749] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:51.276638 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3c9f64ee0544e59add4f73bca59df98" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:51.277851 9710 leader_election.cc:343] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:51.277987 9710 leader_election.cc:304] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
W20251027 21:07:51.278491 9710 consensus_peers.cc:597] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 -> Peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Couldn't send request to peer 51902b6b61ff4d5584ff21896e847e14. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9cec6294051b41868c9c7c7fadafa138. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:07:51.279817 10141 raft_consensus.cc:2749] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:51.280077 10198 raft_consensus.cc:493] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:51.280263 10198 raft_consensus.cc:515] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:51.281653 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3dfef8b741c42f68b664d7e0eab49c6" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:51.282294 9566 leader_election.cc:343] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: Tablet not RUNNING: STOPPED
I20251027 21:07:51.284255 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22d861810ed846c3b77a452d05e93d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.285156 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd7029986a5246ee823e5fdb50683118. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.285820 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26872be132fb46ee9e79a9c2e62374ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.286798 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.287734 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ec2e0ba1994ab59c53977583bd2058. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.288285 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f1d1bd52556463fa306e51eb253af40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.290057 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 21b785277d604543bfa7816ed96d6181 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.296955 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 21a8fc4a69f3486e92b1a68d10faed38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.297165 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet c400101d2a39490e83722500839198cd with delete_type TABLET_DATA_DELETED (replaced by tablet c9efffa9766e46aca8eb286fc2ad48d2) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.297821 10043 ts_tablet_manager.cc:1929] T 44783760defd433a972c9cd83933249c P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.297989 10043 log.cc:1199] T 44783760defd433a972c9cd83933249c P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/44783760defd433a972c9cd83933249c
I20251027 21:07:51.298449 10043 ts_tablet_manager.cc:1950] T 44783760defd433a972c9cd83933249c P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:51.300972 9747 tablet_service.cc:1505] Processing CreateTablet for tablet 22d861810ed846c3b77a452d05e93d72 (DEFAULT_TABLE table=default.Table_94d4f5b9325a4c27a6305962fe774808 [id=c92fda1282d7419f94b0004ab8b8403d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.302567 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 26872be132fb46ee9e79a9c2e62374ef (DEFAULT_TABLE table=default.Table_b23313ec4d7b4e32a4283dd404578616 [id=6a545a959eb74d0eb589f1be92a0bc0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.303067 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26872be132fb46ee9e79a9c2e62374ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.313131 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 2532bf7190064443a898e47e843fd9ac (DEFAULT_TABLE table=default.Table_60b6845211be4b7798d0690248d2b718 [id=c6f20d8985134d9a8d3c5bb4a51b45fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.313541 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2532bf7190064443a898e47e843fd9ac. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:51.315239 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 995284883d834bbb9ef583b736ca8402 consensus state has no leader
I20251027 21:07:51.315723 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_506c51fbcfa745eb83b0b15a29c72e13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.316339 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_506c51fbcfa745eb83b0b15a29c72e13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.318039 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6d0035eeace14307bbb048e1f584b0c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.318600 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d0035eeace14307bbb048e1f584b0c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:51.319725 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca477937278f45b4b115fec715f32a92 has no consensus state
I20251027 21:07:51.347589 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 2532bf7190064443a898e47e843fd9ac (DEFAULT_TABLE table=default.Table_60b6845211be4b7798d0690248d2b718 [id=c6f20d8985134d9a8d3c5bb4a51b45fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.364362 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8fa71eed37b040fcb5f63d829b4a6315" } modify_external_catalogs: true
I20251027 21:07:51.364552 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8fa71eed37b040fcb5f63d829b4a6315" } modify_external_catalogs: true
W20251027 21:07:51.369812 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9339c6b1154400eace53d765528fdf9 consensus state has no leader
I20251027 21:07:51.370216 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_44053023614445a79986595acb8e2e60" } new_table_name: "default.Table_318a68ed05144ed9aeb2db53570ab9c3" modify_external_catalogs: true
I20251027 21:07:51.371747 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_79211c9993c54e4ea0455d01fec68b8e" } modify_external_catalogs: true
I20251027 21:07:51.371892 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_79211c9993c54e4ea0455d01fec68b8e" } modify_external_catalogs: true
I20251027 21:07:51.373394 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3f5e27698f4c4da88d9bacc9dd719304" } modify_external_catalogs: true
I20251027 21:07:51.373557 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3f5e27698f4c4da88d9bacc9dd719304" } modify_external_catalogs: true
W20251027 21:07:51.374869 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bbb4116d1edd492893adfbf5fb3ddeef has no consensus state
I20251027 21:07:51.375253 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0f7c274ead214bb68565a54bc2da4f63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.375787 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f7c274ead214bb68565a54bc2da4f63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.377271 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d19e7f3fb9784eb0b338e23019d1a745"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.377717 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d19e7f3fb9784eb0b338e23019d1a745 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.379930 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_bef3b1992e374f98a2e2ccd86f02ae75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.380394 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bef3b1992e374f98a2e2ccd86f02ae75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.381578 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 0ac7111d13b6460a9f4c77e94e09d417
I20251027 21:07:51.382308 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_627b1e3af2d34bc9b4508237abf020f5" } modify_external_catalogs: true
I20251027 21:07:51.382452 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_627b1e3af2d34bc9b4508237abf020f5" } modify_external_catalogs: true
I20251027 21:07:51.383936 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ac7111d13b6460a9f4c77e94e09d417 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.384022 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ac7111d13b6460a9f4c77e94e09d417 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.384307 10041 tablet_replica.cc:333] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:51.384560 10041 raft_consensus.cc:2243] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.384788 10041 raft_consensus.cc:2272] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.386307 10041 ts_tablet_manager.cc:1916] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.395361 10041 ts_tablet_manager.cc:1929] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 21:07:51.395537 10041 log.cc:1199] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/0ac7111d13b6460a9f4c77e94e09d417
I20251027 21:07:51.396745 10041 ts_tablet_manager.cc:1950] T 0ac7111d13b6460a9f4c77e94e09d417 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:51.397954 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 0ac7111d13b6460a9f4c77e94e09d417 (table default.Table_e65d5d66ce8747a9a1a2abfa04cd2855 [id=e2b68fb34c3b48e8952ef6d4201fd93f]) successfully deleted
I20251027 21:07:51.308256 10198 leader_election.cc:290] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:07:51.311477 9568 leader_election.cc:343] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: b3dfef8b741c42f68b664d7e0eab49c6
I20251027 21:07:51.312160 10029 tablet_bootstrap.cc:492] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:51.319265 9682 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:07:51.400714 10198 raft_consensus.cc:493] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:51.400884 10198 raft_consensus.cc:515] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.402495 10029 ts_tablet_manager.cc:1403] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.347s user 0.029s sys 0.049s
I20251027 21:07:51.403118 9568 leader_election.cc:304] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:51.404500 10198 leader_election.cc:290] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:51.405107 10029 raft_consensus.cc:359] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.405275 10029 raft_consensus.cc:385] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.405366 10029 raft_consensus.cc:740] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.405571 10029 consensus_queue.cc:260] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.406013 10029 ts_tablet_manager.cc:1434] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:07:51.406287 10029 tablet_bootstrap.cc:492] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:51.408948 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65fbcaa5447c48818fabfccac22c4048" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:51.409179 9774 raft_consensus.cc:2468] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
W20251027 21:07:51.409512 9568 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 97122.6us
I20251027 21:07:51.409879 10029 tablet_bootstrap.cc:654] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:51.411007 10198 raft_consensus.cc:2749] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:51.411193 10198 raft_consensus.cc:493] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:51.411307 10198 raft_consensus.cc:515] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:51.411906 10029 tablet_bootstrap.cc:492] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:51.412056 10029 ts_tablet_manager.cc:1403] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20251027 21:07:51.412762 10029 raft_consensus.cc:359] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.412891 9566 leader_election.cc:304] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:07:51.412947 10029 raft_consensus.cc:385] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.413017 10029 raft_consensus.cc:740] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.413214 10029 consensus_queue.cc:260] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.413283 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbada3946dab4739a8369032ff3b8a48" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:51.413513 9774 raft_consensus.cc:2393] T cbada3946dab4739a8369032ff3b8a48 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51902b6b61ff4d5584ff21896e847e14 in current term 1: Already voted for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in this term.
I20251027 21:07:51.411912 10198 leader_election.cc:290] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:51.417200 10198 raft_consensus.cc:2804] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:51.417330 10029 ts_tablet_manager.cc:1434] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251027 21:07:51.417413 10198 raft_consensus.cc:493] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:51.417536 10198 raft_consensus.cc:3060] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:51.417703 10029 tablet_bootstrap.cc:492] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:51.420121 10029 tablet_bootstrap.cc:654] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:51.427094 10198 raft_consensus.cc:515] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.429689 10032 tablet_bootstrap.cc:492] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:51.429867 10032 ts_tablet_manager.cc:1403] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.331s user 0.056s sys 0.061s
I20251027 21:07:51.430712 10032 raft_consensus.cc:359] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:51.430897 10032 raft_consensus.cc:385] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.430991 10032 raft_consensus.cc:740] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.414295 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 630ae4d66f99474382814cc32cc60ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.430739 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65fbcaa5447c48818fabfccac22c4048" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:51.431989 9774 raft_consensus.cc:3060] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:51.434265 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f1d1bd52556463fa306e51eb253af40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.434612 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22d861810ed846c3b77a452d05e93d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.434835 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26872be132fb46ee9e79a9c2e62374ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.436864 9774 raft_consensus.cc:2468] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:51.438274 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd7029986a5246ee823e5fdb50683118. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.438725 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccf10ef559f74e2db28023fd520c8de9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.439110 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 345d3795a37b4a40b40967cfacf0f893. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.439071 9566 leader_election.cc:304] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:07:51.439520 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84b9de82d55447bba2a897edb683e5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.439878 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ec2e0ba1994ab59c53977583bd2058. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.440490 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2532bf7190064443a898e47e843fd9ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.440469 10310 raft_consensus.cc:2804] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:51.440886 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.441488 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d872cf3ef39042ff9a615b55e7195afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.442651 9312 master_service.cc:954] ReplaceTablet: received request to replace tablet 77797eea3e9c4f8099dc9d58f116200b from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:51.442931 10310 raft_consensus.cc:697] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:07:51.443070 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53cac7817db440e79e7416d625a9b2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.443444 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87d5af4607f04f92abcc7292b4176cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.444325 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa
W20251027 21:07:51.445212 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8328318a4f14b38a4ce36d183e99dfb has no consensus state
I20251027 21:07:51.445576 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet dd7029986a5246ee823e5fdb50683118
I20251027 21:07:51.446926 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.446943 9745 tablet_service.cc:1552] Processing DeleteTablet for tablet dd7029986a5246ee823e5fdb50683118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.447182 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.447201 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet dd7029986a5246ee823e5fdb50683118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.447228 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet adaa4f66bbab491f8030d439d85cc7d1
I20251027 21:07:51.448143 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet adaa4f66bbab491f8030d439d85cc7d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.449007 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet adaa4f66bbab491f8030d439d85cc7d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.449793 10198 leader_election.cc:290] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:51.451341 10032 consensus_queue.cc:260] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:51.454962 9970 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:07:51.456115 10032 ts_tablet_manager.cc:1434] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.026s user 0.003s sys 0.000s
I20251027 21:07:51.456447 10032 tablet_bootstrap.cc:492] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:51.458027 10032 tablet_bootstrap.cc:654] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:51.462743 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 970fbf01a4c3406dafe1ea4cd005ea23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.448644 10310 consensus_queue.cc:237] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.477236 10043 tablet_replica.cc:333] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:51.479414 10043 raft_consensus.cc:2243] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.480695 10043 raft_consensus.cc:2272] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.481638 10043 ts_tablet_manager.cc:1916] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.494200 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.494843 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet dd7029986a5246ee823e5fdb50683118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.496244 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet adaa4f66bbab491f8030d439d85cc7d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.498301 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ac7111d13b6460a9f4c77e94e09d417 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.498706 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 2532bf7190064443a898e47e843fd9ac (DEFAULT_TABLE table=default.Table_60b6845211be4b7798d0690248d2b718 [id=c6f20d8985134d9a8d3c5bb4a51b45fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.499073 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2532bf7190064443a898e47e843fd9ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.500507 9304 master_service.cc:954] ReplaceTablet: received request to replace tablet 2b247829a11349b6835a2f18906e20a9 from {username='slave'} at 127.0.0.1:41710
W20251027 21:07:51.501538 9568 leader_election.cc:343] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 65fbcaa5447c48818fabfccac22c4048
I20251027 21:07:51.501741 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 44783760defd433a972c9cd83933249c (table default.Table_44e3f093c8d7476a98074b07828da3ab [id=30fbbddfaed646e3b2d80f788f774741]) successfully deleted
I20251027 21:07:51.512082 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet a9bf3a093be8498c9dd5511b53436b16
I20251027 21:07:51.513594 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet a9bf3a093be8498c9dd5511b53436b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.513950 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet a9bf3a093be8498c9dd5511b53436b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.515161 9820 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:07:51.520423 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 77797eea3e9c4f8099dc9d58f116200b
I20251027 21:07:51.520818 9312 catalog_manager.cc:6719] ReplaceTablet: tablet 77797eea3e9c4f8099dc9d58f116200b deleted and replaced by tablet 631ff72613734913818a3cad24f20dbc
I20251027 21:07:51.528349 9743 tablet_service.cc:1552] Processing DeleteTablet for tablet 77797eea3e9c4f8099dc9d58f116200b with delete_type TABLET_DATA_DELETED (replaced by tablet 631ff72613734913818a3cad24f20dbc) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.535414 10031 tablet_bootstrap.cc:492] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:51.535699 10031 ts_tablet_manager.cc:1403] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.585s user 0.021s sys 0.025s
I20251027 21:07:51.536594 10031 raft_consensus.cc:359] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.536857 10031 raft_consensus.cc:385] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.537088 10031 raft_consensus.cc:740] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.537601 10031 consensus_queue.cc:260] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.539916 9312 master_service.cc:954] ReplaceTablet: received request to replace tablet c49775486ff24686b0f45c47e2757aba from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:51.546514 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet 1776590d8387494da5dfbd08db17f4b1 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:51.547194 9312 master_service.cc:954] ReplaceTablet: received request to replace tablet 648f506ec87c4fb8a3f6d4591873dbb8 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:51.548434 10031 ts_tablet_manager.cc:1434] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251027 21:07:51.552829 9820 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:07:51.553484 10040 ts_tablet_manager.cc:1929] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.553609 10040 log.cc:1199] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/b3dfef8b741c42f68b664d7e0eab49c6
I20251027 21:07:51.553963 10040 ts_tablet_manager.cc:1950] T b3dfef8b741c42f68b664d7e0eab49c6 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:51.554297 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 44783760defd433a972c9cd83933249c with delete_type TABLET_DATA_DELETED (replaced by tablet ffe1182e8f4e4a44b9a12e0a5192989f) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.555915 10031 tablet_bootstrap.cc:492] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:51.556985 10178 raft_consensus.cc:493] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:51.557155 10178 raft_consensus.cc:515] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:51.558437 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa already in progress: creating tablet
W20251027 21:07:51.558669 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet dd7029986a5246ee823e5fdb50683118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd7029986a5246ee823e5fdb50683118 already in progress: creating tablet
W20251027 21:07:51.558900 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet adaa4f66bbab491f8030d439d85cc7d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet adaa4f66bbab491f8030d439d85cc7d1 already in progress: creating tablet
I20251027 21:07:51.559458 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "970fbf01a4c3406dafe1ea4cd005ea23" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:51.559733 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d19e7f3fb9784eb0b338e23019d1a745" } new_table_name: "default.Table_f3c2de263e8849caaef87b570fc69459" modify_external_catalogs: true
I20251027 21:07:51.559955 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_30780e9f9901401ba8240a70a20ad769"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.560523 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30780e9f9901401ba8240a70a20ad769 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.562471 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c24f2baac2d6497ead37ab560db1221b" } new_table_name: "default.Table_0c82ef1916f14d07929b28bffd296f9f" modify_external_catalogs: true
I20251027 21:07:51.564159 10041 tablet_replica.cc:333] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
W20251027 21:07:51.565764 9711 consensus_peers.cc:597] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba8b125474bc42a5946e8f2dd718f7f4. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:07:51.566581 10178 leader_election.cc:290] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:51.566869 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 77797eea3e9c4f8099dc9d58f116200b with delete_type TABLET_DATA_DELETED (replaced by tablet 631ff72613734913818a3cad24f20dbc) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.567837 10032 tablet_bootstrap.cc:492] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
W20251027 21:07:51.567999 9859 leader_election.cc:343] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:51.568259 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet a9bf3a093be8498c9dd5511b53436b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.568981 10032 ts_tablet_manager.cc:1403] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.112s user 0.000s sys 0.003s
I20251027 21:07:51.569780 10032 raft_consensus.cc:359] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:51.569893 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e6b58b1f4f63461aa96e416cae879be8 has no consensus state
I20251027 21:07:51.570844 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 170) took 1973 ms. Trace:
I20251027 21:07:51.570971 10040 rpcz_store.cc:276] 1027 21:07:49.597514 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.597693 (+ 179us) service_pool.cc:225] Handling call
1027 21:07:51.259760 (+1662067us) ts_tablet_manager.cc:1151] Deleting tablet b3dfef8b741c42f68b664d7e0eab49c6
1027 21:07:51.259769 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:51.509335 (+249566us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:51.553350 (+ 44015us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:51.570798 (+ 17448us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11207,"tablet-delete.queue_time_us":1661854}
I20251027 21:07:51.571318 10040 tablet_replica.cc:333] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:51.571533 10040 raft_consensus.cc:2243] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.571686 10040 raft_consensus.cc:2272] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.572182 10040 ts_tablet_manager.cc:1916] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.573199 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet b3dfef8b741c42f68b664d7e0eab49c6 (table default.Table_be23b67279484b3db60ef67b1b157253 [id=9e10359cf7a640e9b16f01096987ca59]) successfully deleted
I20251027 21:07:51.573478 10032 raft_consensus.cc:385] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.573580 10032 raft_consensus.cc:740] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.574429 10032 consensus_queue.cc:260] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:51.574940 10032 ts_tablet_manager.cc:1434] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251027 21:07:51.576880 10032 tablet_bootstrap.cc:492] T f6249fc5c6174e8e81b9d02d598d5ebd P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:51.578981 10040 ts_tablet_manager.cc:1929] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.579133 10040 log.cc:1199] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/63662f175a1c450f8f67f37da971db37
I20251027 21:07:51.579480 10040 ts_tablet_manager.cc:1950] T 63662f175a1c450f8f67f37da971db37 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:51.580163 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 148) took 1985 ms. Trace:
I20251027 21:07:51.580255 10040 rpcz_store.cc:276] 1027 21:07:49.594689 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.594771 (+ 82us) service_pool.cc:225] Handling call
1027 21:07:51.571257 (+1976486us) ts_tablet_manager.cc:1151] Deleting tablet 63662f175a1c450f8f67f37da971db37
1027 21:07:51.571266 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:51.575388 (+ 4122us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:51.578881 (+ 3493us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:51.580102 (+ 1221us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1973227}
I20251027 21:07:51.580691 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 157) took 1984 ms. Trace:
I20251027 21:07:51.580786 10040 rpcz_store.cc:276] 1027 21:07:49.595800 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.595854 (+ 54us) service_pool.cc:225] Handling call
1027 21:07:51.580507 (+1984653us) ts_tablet_manager.cc:1151] Deleting tablet cad3a3e910ac42399cf371ad82c89ff7
1027 21:07:51.580516 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:51.580640 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1982203}
I20251027 21:07:51.580873 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 63662f175a1c450f8f67f37da971db37 (table default.Table_d7caf208b9864991a0e50ced803eb60d [id=4142b5bb00004c83afe20e6982409849]) successfully deleted
I20251027 21:07:51.581128 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 162) took 1984 ms. Trace:
I20251027 21:07:51.581207 10040 rpcz_store.cc:276] 1027 21:07:49.596490 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.596649 (+ 159us) service_pool.cc:225] Handling call
1027 21:07:51.580999 (+1984350us) ts_tablet_manager.cc:1151] Deleting tablet 6db0f31c12d14e66a4ec7fc722d62c3d
1027 21:07:51.581007 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:51.581081 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1981919}
I20251027 21:07:51.581557 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 158) took 1985 ms. Trace:
W20251027 21:07:51.581669 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet cad3a3e910ac42399cf371ad82c89ff7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cad3a3e910ac42399cf371ad82c89ff7 already in progress: creating tablet
I20251027 21:07:51.581638 10040 rpcz_store.cc:276] 1027 21:07:49.595955 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.596864 (+ 909us) service_pool.cc:225] Handling call
1027 21:07:51.581434 (+1984570us) ts_tablet_manager.cc:1151] Deleting tablet 81e914ecc3c74b869e9fc3ad102887ec
1027 21:07:51.581443 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:51.581513 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1982244}
I20251027 21:07:51.581895 10040 tablet_replica.cc:333] T cbada3946dab4739a8369032ff3b8a48 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
W20251027 21:07:51.581935 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 6db0f31c12d14e66a4ec7fc722d62c3d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6db0f31c12d14e66a4ec7fc722d62c3d already in progress: creating tablet
I20251027 21:07:51.582047 10040 raft_consensus.cc:2243] T cbada3946dab4739a8369032ff3b8a48 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.582217 10040 raft_consensus.cc:2272] T cbada3946dab4739a8369032ff3b8a48 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.582482 10178 raft_consensus.cc:493] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:51.582711 10040 ts_tablet_manager.cc:1916] T cbada3946dab4739a8369032ff3b8a48 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.584106 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 2b247829a11349b6835a2f18906e20a9
I20251027 21:07:51.584540 9304 catalog_manager.cc:6719] ReplaceTablet: tablet 2b247829a11349b6835a2f18906e20a9 deleted and replaced by tablet fde0980aa64c4496861a662f7ed3ef75
W20251027 21:07:51.587749 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 81e914ecc3c74b869e9fc3ad102887ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81e914ecc3c74b869e9fc3ad102887ec already in progress: creating tablet
I20251027 21:07:51.588325 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b247829a11349b6835a2f18906e20a9 with delete_type TABLET_DATA_DELETED (replaced by tablet fde0980aa64c4496861a662f7ed3ef75) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.588563 10178 raft_consensus.cc:515] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:51.589232 10178 leader_election.cc:290] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:51.589951 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_69f618a7f3c74a1d87cb5e0c17f38f94" } modify_external_catalogs: true
I20251027 21:07:51.590085 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_69f618a7f3c74a1d87cb5e0c17f38f94" } modify_external_catalogs: true
I20251027 21:07:51.590535 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_cf6134c7114f4ad5b61f1bda6ffa5015"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.590977 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf6134c7114f4ad5b61f1bda6ffa5015 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.591563 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10a44aecfade42969454126b990eca24" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:51.591980 10041 raft_consensus.cc:2243] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.592049 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5877315061df40eb80ba9d966632471a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:51.592171 10041 raft_consensus.cc:2272] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:07:51.592527 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5877315061df40eb80ba9d966632471a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.593485 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6f1e2491cbc546a7b54ab25da6ebf334" } new_table_name: "default.Table_1e71059726554fec9b06fa78154e9ad5" modify_external_catalogs: true
W20251027 21:07:51.594215 9859 leader_election.cc:343] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:51.594575 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a6cf2819ed9241cea09738477423c84a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:51.594962 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b247829a11349b6835a2f18906e20a9 with delete_type TABLET_DATA_DELETED (replaced by tablet fde0980aa64c4496861a662f7ed3ef75) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:51.595033 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6cf2819ed9241cea09738477423c84a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.595773 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 10a44aecfade42969454126b990eca24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.596344 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_857e62d8fe554d7886a5de26057618f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.596722 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_857e62d8fe554d7886a5de26057618f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.597697 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_af921638ac16405c9835ce2b0446cec2" } new_table_name: "default.Table_dbda62167b92488788bb65bb4b75b73a" modify_external_catalogs: true
I20251027 21:07:51.597783 10041 ts_tablet_manager.cc:1916] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:51.600144 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e841c54a94d649c19b639f5021f150cd consensus state has no leader
I20251027 21:07:51.601603 10040 ts_tablet_manager.cc:1929] T cbada3946dab4739a8369032ff3b8a48 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.601712 10040 log.cc:1199] T cbada3946dab4739a8369032ff3b8a48 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/cbada3946dab4739a8369032ff3b8a48
I20251027 21:07:51.602028 10032 tablet_bootstrap.cc:654] T f6249fc5c6174e8e81b9d02d598d5ebd P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:51.602031 10040 ts_tablet_manager.cc:1950] T cbada3946dab4739a8369032ff3b8a48 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:51.602746 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 172) took 2005 ms. Trace:
I20251027 21:07:51.602850 10040 rpcz_store.cc:276] 1027 21:07:49.597702 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.597917 (+ 215us) service_pool.cc:225] Handling call
1027 21:07:51.581828 (+1983911us) ts_tablet_manager.cc:1151] Deleting tablet cbada3946dab4739a8369032ff3b8a48
1027 21:07:51.581836 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:51.598766 (+ 16930us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:51.601505 (+ 2739us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:51.602715 (+ 1210us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1953780}
I20251027 21:07:51.603122 10040 tablet_replica.cc:333] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:51.603307 10040 raft_consensus.cc:2243] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:07:51.603608 10040 pending_rounds.cc:70] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
I20251027 21:07:51.603693 10040 pending_rounds.cc:77] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510612896256000 op_type: NO_OP noop_request { }
I20251027 21:07:51.603791 10040 raft_consensus.cc:2889] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:07:51.603912 10040 raft_consensus.cc:2272] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.604156 10041 ts_tablet_manager.cc:1929] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.604282 10041 log.cc:1199] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/970fbf01a4c3406dafe1ea4cd005ea23
I20251027 21:07:51.604665 10040 ts_tablet_manager.cc:1916] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.605656 10041 ts_tablet_manager.cc:1950] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:51.606658 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 970fbf01a4c3406dafe1ea4cd005ea23 (table default.Table_dddd93d5fb3c4727b64ded3ff6cc3382 [id=741c9ce04c9d47d9a016bce1f4403ec7]) successfully deleted
W20251027 21:07:51.606904 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 77797eea3e9c4f8099dc9d58f116200b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77797eea3e9c4f8099dc9d58f116200b already in progress: creating tablet
W20251027 21:07:51.607148 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet a9bf3a093be8498c9dd5511b53436b16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9bf3a093be8498c9dd5511b53436b16 already in progress: creating tablet
I20251027 21:07:51.607352 10041 tablet_replica.cc:333] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
W20251027 21:07:51.607416 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 2b247829a11349b6835a2f18906e20a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b247829a11349b6835a2f18906e20a9 already in progress: creating tablet
I20251027 21:07:51.607631 10041 raft_consensus.cc:2243] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.607823 10041 raft_consensus.cc:2272] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.608565 10041 ts_tablet_manager.cc:1916] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.609256 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 648f506ec87c4fb8a3f6d4591873dbb8
I20251027 21:07:51.609699 9312 catalog_manager.cc:6719] ReplaceTablet: tablet 648f506ec87c4fb8a3f6d4591873dbb8 deleted and replaced by tablet 96a680e8f16343b1908829a67e960b60
I20251027 21:07:51.610780 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 648f506ec87c4fb8a3f6d4591873dbb8 with delete_type TABLET_DATA_DELETED (replaced by tablet 96a680e8f16343b1908829a67e960b60) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.611271 10312 raft_consensus.cc:493] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:51.611423 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 648f506ec87c4fb8a3f6d4591873dbb8 with delete_type TABLET_DATA_DELETED (replaced by tablet 96a680e8f16343b1908829a67e960b60) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.611485 10312 raft_consensus.cc:515] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.612229 10312 leader_election.cc:290] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:51.612994 10043 ts_tablet_manager.cc:1929] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.613474 10029 tablet_bootstrap.cc:492] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:51.613617 10029 ts_tablet_manager.cc:1403] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.196s user 0.009s sys 0.003s
I20251027 21:07:51.614377 10029 raft_consensus.cc:359] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.614537 10029 raft_consensus.cc:385] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.614621 10029 raft_consensus.cc:740] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.614852 10029 consensus_queue.cc:260] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.615357 10029 ts_tablet_manager.cc:1434] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:51.615234 10031 tablet_bootstrap.cc:654] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:51.615653 10029 tablet_bootstrap.cc:492] T 36e168cc44274dae9682b0c1329d4558 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:51.616289 10041 ts_tablet_manager.cc:1929] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.616480 10041 log.cc:1199] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/10a44aecfade42969454126b990eca24
I20251027 21:07:51.616890 10041 ts_tablet_manager.cc:1950] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:51.617187 10029 tablet_bootstrap.cc:654] T 36e168cc44274dae9682b0c1329d4558 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:51.617686 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 10a44aecfade42969454126b990eca24 (table default.Table_b0f0589285ac4a56b7a2b23ace6248f6 [id=43ebd8ba9e694d17a9def870c95d7aa6]) successfully deleted
W20251027 21:07:51.617969 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 648f506ec87c4fb8a3f6d4591873dbb8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 648f506ec87c4fb8a3f6d4591873dbb8 already in progress: creating tablet
I20251027 21:07:51.618189 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet 77797eea3e9c4f8099dc9d58f116200b with delete_type TABLET_DATA_DELETED (replaced by tablet 631ff72613734913818a3cad24f20dbc) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.618309 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b247829a11349b6835a2f18906e20a9 with delete_type TABLET_DATA_DELETED (replaced by tablet fde0980aa64c4496861a662f7ed3ef75) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.618505 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "970fbf01a4c3406dafe1ea4cd005ea23" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:51.618965 10043 log.cc:1199] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/b3dfef8b741c42f68b664d7e0eab49c6
I20251027 21:07:51.619413 10043 ts_tablet_manager.cc:1950] T b3dfef8b741c42f68b664d7e0eab49c6 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
W20251027 21:07:51.620608 9860 leader_election.cc:343] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:51.620651 10043 tablet_replica.cc:333] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:51.620743 9860 leader_election.cc:304] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:51.620898 10043 raft_consensus.cc:2243] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251027 21:07:51.620914 9860 raft_consensus.cc:2686] T 970fbf01a4c3406dafe1ea4cd005ea23 P 0fbd567f929a4f6d8e4d78feef7ed678: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251027 21:07:51.621423 10043 raft_consensus.cc:2272] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.621572 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69b1fb42428748239d8939d98428d8fe" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:51.622047 10043 ts_tablet_manager.cc:1916] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:51.622474 9566 leader_election.cc:343] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:51.623054 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10a44aecfade42969454126b990eca24" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:51.626619 9860 leader_election.cc:343] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:51.626739 9860 leader_election.cc:304] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:07:51.626938 9860 raft_consensus.cc:2686] T 10a44aecfade42969454126b990eca24 P 0fbd567f929a4f6d8e4d78feef7ed678: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251027 21:07:51.627735 9890 tablet_service.cc:1505] Processing CreateTablet for tablet a8328318a4f14b38a4ce36d183e99dfb (DEFAULT_TABLE table=default.Table_318a68ed05144ed9aeb2db53570ab9c3 [id=287175ad40544f73a586c736fcce910b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.628105 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8328318a4f14b38a4ce36d183e99dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.628491 9749 tablet_service.cc:1505] Processing CreateTablet for tablet a8328318a4f14b38a4ce36d183e99dfb (DEFAULT_TABLE table=default.Table_318a68ed05144ed9aeb2db53570ab9c3 [id=287175ad40544f73a586c736fcce910b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.628640 9753 tablet_service.cc:1505] Processing CreateTablet for tablet a339c0dee0994788b8644912f74c1b82 (DEFAULT_TABLE table=default.Table_4a455030bbe945c391d97696ee50010e [id=fb030c6e79b44af4b3469a20e09a8e47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.627784 9889 tablet_service.cc:1505] Processing CreateTablet for tablet a339c0dee0994788b8644912f74c1b82 (DEFAULT_TABLE table=default.Table_4a455030bbe945c391d97696ee50010e [id=fb030c6e79b44af4b3469a20e09a8e47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.628939 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8328318a4f14b38a4ce36d183e99dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.629153 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a339c0dee0994788b8644912f74c1b82. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:51.629236 9568 leader_election.cc:343] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: cbada3946dab4739a8369032ff3b8a48
I20251027 21:07:51.629355 9568 leader_election.cc:304] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
W20251027 21:07:51.629753 9568 leader_election.cc:343] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 65fbcaa5447c48818fabfccac22c4048
W20251027 21:07:51.630126 9568 consensus_peers.cc:597] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ac7111d13b6460a9f4c77e94e09d417. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:07:51.630434 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 6600774e4c7b48479a2fdb02614ba505 (DEFAULT_TABLE table=default.Table_11d4ee667e1c4d9badc00349e1c61463 [id=914b728e358e4fdba6d38d6c7beadaca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.630805 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6600774e4c7b48479a2fdb02614ba505. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.630846 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet 648f506ec87c4fb8a3f6d4591873dbb8 with delete_type TABLET_DATA_DELETED (replaced by tablet 96a680e8f16343b1908829a67e960b60) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.631054 10312 raft_consensus.cc:3060] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:51.633906 10312 raft_consensus.cc:2749] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:51.634348 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a339c0dee0994788b8644912f74c1b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.634477 9741 tablet_service.cc:1505] Processing CreateTablet for tablet 6600774e4c7b48479a2fdb02614ba505 (DEFAULT_TABLE table=default.Table_11d4ee667e1c4d9badc00349e1c61463 [id=914b728e358e4fdba6d38d6c7beadaca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.634640 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet b3dfef8b741c42f68b664d7e0eab49c6 (table default.Table_be23b67279484b3db60ef67b1b157253 [id=9e10359cf7a640e9b16f01096987ca59]) successfully deleted
I20251027 21:07:51.634943 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6600774e4c7b48479a2fdb02614ba505. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.636034 9604 tablet_service.cc:1505] Processing CreateTablet for tablet a339c0dee0994788b8644912f74c1b82 (DEFAULT_TABLE table=default.Table_4a455030bbe945c391d97696ee50010e [id=fb030c6e79b44af4b3469a20e09a8e47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.636372 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a339c0dee0994788b8644912f74c1b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.641674 10043 ts_tablet_manager.cc:1929] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.641837 10043 log.cc:1199] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/69b1fb42428748239d8939d98428d8fe
I20251027 21:07:51.643350 9606 tablet_service.cc:1505] Processing CreateTablet for tablet a8328318a4f14b38a4ce36d183e99dfb (DEFAULT_TABLE table=default.Table_318a68ed05144ed9aeb2db53570ab9c3 [id=287175ad40544f73a586c736fcce910b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.643672 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8328318a4f14b38a4ce36d183e99dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.644776 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 4debc3f13f3a42939b6b83f6523aab7c (DEFAULT_TABLE table=default.Table_4bd57498972a413ca2bb7de2cd588e8b [id=5ebbc228b61046f8aef2d6fe13f12e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.645121 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4debc3f13f3a42939b6b83f6523aab7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.647149 9753 tablet_service.cc:1505] Processing CreateTablet for tablet ca477937278f45b4b115fec715f32a92 (DEFAULT_TABLE table=default.Table_4c5fcd020732482597349bbb07468bf5 [id=fc5a0b5a96e6400da438307d26f73614]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.647486 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca477937278f45b4b115fec715f32a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.648556 9898 tablet_service.cc:1505] Processing CreateTablet for tablet e9edb62a8e4b4dfea6a31a2202545a6e (DEFAULT_TABLE table=default.Table_2d874641d88e434c8022d7026802972f [id=7b94fb706c8e429b911e1e19265a3883]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.648876 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9edb62a8e4b4dfea6a31a2202545a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.648972 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 6600774e4c7b48479a2fdb02614ba505 (DEFAULT_TABLE table=default.Table_11d4ee667e1c4d9badc00349e1c61463 [id=914b728e358e4fdba6d38d6c7beadaca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.649307 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6600774e4c7b48479a2fdb02614ba505. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:51.649204 9568 leader_election.cc:343] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 69b1fb42428748239d8939d98428d8fe
I20251027 21:07:51.651891 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d2ff301b3e2b4602a2483859973a02d6" } modify_external_catalogs: true
I20251027 21:07:51.652066 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d2ff301b3e2b4602a2483859973a02d6" } modify_external_catalogs: true
I20251027 21:07:51.651975 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_294e3c4092274dc1a146d3f604bc2179"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.652525 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_294e3c4092274dc1a146d3f604bc2179 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.652690 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4c5fcd020732482597349bbb07468bf5" } new_table_name: "default.Table_c76923735ab34c5fbbbba0443cbc54c5" modify_external_catalogs: true
I20251027 21:07:51.652534 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2d874641d88e434c8022d7026802972f" } modify_external_catalogs: true
I20251027 21:07:51.653697 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2d874641d88e434c8022d7026802972f" } modify_external_catalogs: true
I20251027 21:07:51.653882 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a8906ade8f93405bb76d1ca9b7a0afa6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.654330 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8906ade8f93405bb76d1ca9b7a0afa6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.654734 9890 tablet_service.cc:1505] Processing CreateTablet for tablet d56d323ef0e74344a479940b93fea11a (DEFAULT_TABLE table=default.Table_31d10565065d494194f0dda082708ac0 [id=71e02b21c9654e45a41d3e450ac3fbe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.655120 9741 tablet_service.cc:1505] Processing CreateTablet for tablet d56d323ef0e74344a479940b93fea11a (DEFAULT_TABLE table=default.Table_31d10565065d494194f0dda082708ac0 [id=71e02b21c9654e45a41d3e450ac3fbe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.655200 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d56d323ef0e74344a479940b93fea11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.655486 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d56d323ef0e74344a479940b93fea11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.649626 9568 leader_election.cc:304] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:51.656035 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 4debc3f13f3a42939b6b83f6523aab7c (DEFAULT_TABLE table=default.Table_4bd57498972a413ca2bb7de2cd588e8b [id=5ebbc228b61046f8aef2d6fe13f12e57]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:51.656647 9568 raft_consensus.cc:2686] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251027 21:07:51.656816 10043 ts_tablet_manager.cc:1950] T 69b1fb42428748239d8939d98428d8fe P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:51.658430 10043 tablet_replica.cc:333] T 0c1a6a7616f24f69b83ec8a282746e92 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:51.660470 9891 tablet_service.cc:1505] Processing CreateTablet for tablet ca477937278f45b4b115fec715f32a92 (DEFAULT_TABLE table=default.Table_4c5fcd020732482597349bbb07468bf5 [id=fc5a0b5a96e6400da438307d26f73614]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.661000 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4debc3f13f3a42939b6b83f6523aab7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.662247 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca477937278f45b4b115fec715f32a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.665412 10032 tablet_bootstrap.cc:492] T f6249fc5c6174e8e81b9d02d598d5ebd P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:51.665887 10032 ts_tablet_manager.cc:1403] T f6249fc5c6174e8e81b9d02d598d5ebd P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.091s user 0.001s sys 0.005s
I20251027 21:07:51.667150 10032 raft_consensus.cc:359] T f6249fc5c6174e8e81b9d02d598d5ebd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:51.667657 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2994a62669894e3baf66db0e580ece68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.668227 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2994a62669894e3baf66db0e580ece68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.668437 9753 tablet_service.cc:1505] Processing CreateTablet for tablet e9edb62a8e4b4dfea6a31a2202545a6e (DEFAULT_TABLE table=default.Table_2d874641d88e434c8022d7026802972f [id=7b94fb706c8e429b911e1e19265a3883]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.668769 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9edb62a8e4b4dfea6a31a2202545a6e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:51.669476 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d872cf3ef39042ff9a615b55e7195afa consensus state has no leader
I20251027 21:07:51.670198 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_673503358b54481c8877034c26349796" } modify_external_catalogs: true
I20251027 21:07:51.670356 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_673503358b54481c8877034c26349796" } modify_external_catalogs: true
I20251027 21:07:51.671041 10032 raft_consensus.cc:385] T f6249fc5c6174e8e81b9d02d598d5ebd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.671207 10032 raft_consensus.cc:740] T f6249fc5c6174e8e81b9d02d598d5ebd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.671401 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 67bb12de9f044c73ae9ab26d10498c90 (DEFAULT_TABLE table=default.Table_0ce16ec9dc8b44eab80cedc074577cba [id=c22fee46f319480e88ab7745f6a95126]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.671738 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67bb12de9f044c73ae9ab26d10498c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.672535 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 67bb12de9f044c73ae9ab26d10498c90 (DEFAULT_TABLE table=default.Table_0ce16ec9dc8b44eab80cedc074577cba [id=c22fee46f319480e88ab7745f6a95126]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.673888 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67bb12de9f044c73ae9ab26d10498c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.676640 10043 raft_consensus.cc:2243] T 0c1a6a7616f24f69b83ec8a282746e92 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.678731 10032 consensus_queue.cc:260] T f6249fc5c6174e8e81b9d02d598d5ebd P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:51.679869 9897 tablet_service.cc:1505] Processing CreateTablet for tablet a26ded6012174348b7295fc7c0e05e6e (DEFAULT_TABLE table=default.Table_69f618a7f3c74a1d87cb5e0c17f38f94 [id=d37bad05f12e4d0da8934baeb7627de6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.680299 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a26ded6012174348b7295fc7c0e05e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.683077 9601 tablet_service.cc:1505] Processing CreateTablet for tablet ca477937278f45b4b115fec715f32a92 (DEFAULT_TABLE table=default.Table_4c5fcd020732482597349bbb07468bf5 [id=fc5a0b5a96e6400da438307d26f73614]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.683100 9594 tablet_service.cc:1505] Processing CreateTablet for tablet d56d323ef0e74344a479940b93fea11a (DEFAULT_TABLE table=default.Table_31d10565065d494194f0dda082708ac0 [id=71e02b21c9654e45a41d3e450ac3fbe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.683431 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca477937278f45b4b115fec715f32a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.683655 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d56d323ef0e74344a479940b93fea11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.685424 9606 tablet_service.cc:1505] Processing CreateTablet for tablet e9edb62a8e4b4dfea6a31a2202545a6e (DEFAULT_TABLE table=default.Table_2d874641d88e434c8022d7026802972f [id=7b94fb706c8e429b911e1e19265a3883]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.685834 10043 raft_consensus.cc:2272] T 0c1a6a7616f24f69b83ec8a282746e92 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.690454 10032 ts_tablet_manager.cc:1434] T f6249fc5c6174e8e81b9d02d598d5ebd P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20251027 21:07:51.691699 10032 tablet_bootstrap.cc:492] T d89c3c588a1747d7875ebf45924fb890 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:07:51.692154 10299 log.cc:927] Time spent T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20251027 21:07:51.692416 9749 tablet_service.cc:1505] Processing CreateTablet for tablet a26ded6012174348b7295fc7c0e05e6e (DEFAULT_TABLE table=default.Table_69f618a7f3c74a1d87cb5e0c17f38f94 [id=d37bad05f12e4d0da8934baeb7627de6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.692816 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a26ded6012174348b7295fc7c0e05e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.693997 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 32f80614e30d41128f9fb2e0ea165223 (DEFAULT_TABLE table=default.Table_b547b70a001f404fb96dc4f8fa6d70e9 [id=e1f4a77dfc0a497080c3c0ad4fe2e370]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.694501 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32f80614e30d41128f9fb2e0ea165223. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.694887 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9edb62a8e4b4dfea6a31a2202545a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.695993 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 69b1fb42428748239d8939d98428d8fe (table default.Table_5d5f327d728f4250841a9f4f10a2b533 [id=33dbf5fe986c415cb54fbf1b574651ac]) successfully deleted
I20251027 21:07:51.696383 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 67bb12de9f044c73ae9ab26d10498c90 (DEFAULT_TABLE table=default.Table_0ce16ec9dc8b44eab80cedc074577cba [id=c22fee46f319480e88ab7745f6a95126]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.696707 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67bb12de9f044c73ae9ab26d10498c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.700147 10043 ts_tablet_manager.cc:1916] T 0c1a6a7616f24f69b83ec8a282746e92 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.700228 9601 tablet_service.cc:1505] Processing CreateTablet for tablet a26ded6012174348b7295fc7c0e05e6e (DEFAULT_TABLE table=default.Table_69f618a7f3c74a1d87cb5e0c17f38f94 [id=d37bad05f12e4d0da8934baeb7627de6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.700613 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a26ded6012174348b7295fc7c0e05e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.700750 9604 tablet_service.cc:1505] Processing CreateTablet for tablet 4debc3f13f3a42939b6b83f6523aab7c (DEFAULT_TABLE table=default.Table_4bd57498972a413ca2bb7de2cd588e8b [id=5ebbc228b61046f8aef2d6fe13f12e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.701539 10032 tablet_bootstrap.cc:654] T d89c3c588a1747d7875ebf45924fb890 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:51.703505 10029 tablet_bootstrap.cc:492] T 36e168cc44274dae9682b0c1329d4558 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:51.704401 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 32f80614e30d41128f9fb2e0ea165223 (DEFAULT_TABLE table=default.Table_b547b70a001f404fb96dc4f8fa6d70e9 [id=e1f4a77dfc0a497080c3c0ad4fe2e370]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.705032 9741 tablet_service.cc:1505] Processing CreateTablet for tablet c9efffa9766e46aca8eb286fc2ad48d2 (DEFAULT_TABLE table=default.Table_97b01c1840c543838263ca4222695f54 [id=bea51a58fda64d40916f306c4aa8ab4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.705691 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9efffa9766e46aca8eb286fc2ad48d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.706669 10029 ts_tablet_manager.cc:1403] T 36e168cc44274dae9682b0c1329d4558 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.089s user 0.004s sys 0.019s
I20251027 21:07:51.707397 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32f80614e30d41128f9fb2e0ea165223. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.707789 10029 raft_consensus.cc:359] T 36e168cc44274dae9682b0c1329d4558 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.707938 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4debc3f13f3a42939b6b83f6523aab7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.708163 10299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=37 ignored=0} inserts{seen=39 ignored=0} mutations{seen=22 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251027 21:07:51.708352 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 32f80614e30d41128f9fb2e0ea165223 (DEFAULT_TABLE table=default.Table_b547b70a001f404fb96dc4f8fa6d70e9 [id=e1f4a77dfc0a497080c3c0ad4fe2e370]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.708858 10299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: Bootstrap complete.
I20251027 21:07:51.705588 9897 tablet_service.cc:1505] Processing CreateTablet for tablet c9efffa9766e46aca8eb286fc2ad48d2 (DEFAULT_TABLE table=default.Table_97b01c1840c543838263ca4222695f54 [id=bea51a58fda64d40916f306c4aa8ab4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.709667 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9efffa9766e46aca8eb286fc2ad48d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.713861 10029 raft_consensus.cc:385] T 36e168cc44274dae9682b0c1329d4558 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.714418 10029 raft_consensus.cc:740] T 36e168cc44274dae9682b0c1329d4558 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.715210 10029 consensus_queue.cc:260] T 36e168cc44274dae9682b0c1329d4558 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.715806 10029 ts_tablet_manager.cc:1434] T 36e168cc44274dae9682b0c1329d4558 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20251027 21:07:51.717228 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32f80614e30d41128f9fb2e0ea165223. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.717533 10029 tablet_bootstrap.cc:492] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:51.723516 9898 tablet_service.cc:1505] Processing CreateTablet for tablet c3b9a461567a48fcb4af514ecdf618b4 (DEFAULT_TABLE table=default.Table_9e88d5f78e27447c8072ce0563686ac8 [id=56f8db49ad8c4670a73a29bf4ed613e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.723601 10029 tablet_bootstrap.cc:654] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:51.724025 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3b9a461567a48fcb4af514ecdf618b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.725385 10040 ts_tablet_manager.cc:1929] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:07:51.725502 10040 log.cc:1199] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/ba8b125474bc42a5946e8f2dd718f7f4
I20251027 21:07:51.725953 10040 ts_tablet_manager.cc:1950] T ba8b125474bc42a5946e8f2dd718f7f4 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:51.726281 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 173) took 2098 ms. Trace:
I20251027 21:07:51.726397 10040 rpcz_store.cc:276] 1027 21:07:49.627632 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.627688 (+ 56us) service_pool.cc:225] Handling call
1027 21:07:51.603069 (+1975381us) ts_tablet_manager.cc:1151] Deleting tablet ba8b125474bc42a5946e8f2dd718f7f4
1027 21:07:51.603076 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:51.608146 (+ 5070us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:51.725270 (+117124us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:51.726253 (+ 983us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1916379}
I20251027 21:07:51.726490 10324 tablet.cc:1722] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:07:51.726759 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet ba8b125474bc42a5946e8f2dd718f7f4 (table default.Table_16dd2e0479ea434dbbd0b900b340fc9f [id=e7b7bdb2ea7f468c98bfe88f900a5be8]) successfully deleted
I20251027 21:07:51.726917 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 164) took 2130 ms. Trace:
I20251027 21:07:51.727484 9753 tablet_service.cc:1505] Processing CreateTablet for tablet c3b9a461567a48fcb4af514ecdf618b4 (DEFAULT_TABLE table=default.Table_9e88d5f78e27447c8072ce0563686ac8 [id=56f8db49ad8c4670a73a29bf4ed613e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.727835 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3b9a461567a48fcb4af514ecdf618b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.728722 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 8b30052ff19744a9bf4bfaaa1bc77b36 (DEFAULT_TABLE table=default.Table_5dc88891b75b4f1ca7b3ffba0d2b1fda [id=b46f1e4f1e6d4bd8b4a3edd1d6787612]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.729050 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b30052ff19744a9bf4bfaaa1bc77b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.730974 10032 tablet_bootstrap.cc:492] T d89c3c588a1747d7875ebf45924fb890 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:51.731155 10032 ts_tablet_manager.cc:1403] T d89c3c588a1747d7875ebf45924fb890 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.001s
I20251027 21:07:51.731884 10032 raft_consensus.cc:359] T d89c3c588a1747d7875ebf45924fb890 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:51.732123 10032 raft_consensus.cc:385] T d89c3c588a1747d7875ebf45924fb890 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.732211 10032 raft_consensus.cc:740] T d89c3c588a1747d7875ebf45924fb890 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.727012 10040 rpcz_store.cc:276] 1027 21:07:49.596705 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.596857 (+ 152us) service_pool.cc:225] Handling call
1027 21:07:51.726791 (+2129934us) ts_tablet_manager.cc:1151] Deleting tablet c49775486ff24686b0f45c47e2757aba
1027 21:07:51.726800 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:51.726885 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2037645}
I20251027 21:07:51.739027 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 163) took 2142 ms. Trace:
I20251027 21:07:51.739140 10040 rpcz_store.cc:276] 1027 21:07:49.596594 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.598300 (+ 1706us) service_pool.cc:225] Handling call
1027 21:07:51.738897 (+2140597us) ts_tablet_manager.cc:1151] Deleting tablet a9cc64ad85334daa9a09538e24ebd8a5
1027 21:07:51.738907 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:51.738992 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2036519}
I20251027 21:07:51.740159 10040 tablet_replica.cc:333] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:51.740365 10040 raft_consensus.cc:2243] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.740516 10040 raft_consensus.cc:2272] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.741227 10029 tablet_bootstrap.cc:492] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:51.741278 9900 tablet_service.cc:1505] Processing CreateTablet for tablet 8b30052ff19744a9bf4bfaaa1bc77b36 (DEFAULT_TABLE table=default.Table_5dc88891b75b4f1ca7b3ffba0d2b1fda [id=b46f1e4f1e6d4bd8b4a3edd1d6787612]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.741405 10029 ts_tablet_manager.cc:1403] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20251027 21:07:51.741622 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b30052ff19744a9bf4bfaaa1bc77b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.742408 10029 raft_consensus.cc:359] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.742630 10029 raft_consensus.cc:385] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.742765 10029 raft_consensus.cc:740] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.743031 10029 consensus_queue.cc:260] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.744751 10324 tablet.cc:1717] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14: Skipping requested alter to schema version 2, tablet already version 2
I20251027 21:07:51.745630 10029 ts_tablet_manager.cc:1434] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251027 21:07:51.745983 10029 tablet_bootstrap.cc:492] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:51.746385 9892 tablet_service.cc:1505] Processing CreateTablet for tablet da26c83976df40328cf9b11734fcfa44 (DEFAULT_TABLE table=default.Table_ac66ea1435fe49518c073dfc9b2368c8 [id=985d97a0b33f4ef79a9b0df362a75c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.746748 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da26c83976df40328cf9b11734fcfa44. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.747731 10029 tablet_bootstrap.cc:654] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:51.753131 10029 tablet_bootstrap.cc:492] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:51.753311 10029 ts_tablet_manager.cc:1403] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20251027 21:07:51.754423 10029 raft_consensus.cc:359] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:51.754576 10029 raft_consensus.cc:385] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.754658 10029 raft_consensus.cc:740] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.755386 10040 ts_tablet_manager.cc:1916] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:51.758109 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c49775486ff24686b0f45c47e2757aba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c49775486ff24686b0f45c47e2757aba already in progress: creating tablet
W20251027 21:07:51.758581 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet a9cc64ad85334daa9a09538e24ebd8a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9cc64ad85334daa9a09538e24ebd8a5 already in progress: creating tablet
I20251027 21:07:51.760149 10040 ts_tablet_manager.cc:1929] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.760293 10040 log.cc:1199] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/84305bcf21b64d2c8c1df015a2e9615c
I20251027 21:07:51.754905 10029 consensus_queue.cc:260] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:51.761251 10029 ts_tablet_manager.cc:1434] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251027 21:07:51.761552 10029 tablet_bootstrap.cc:492] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:51.762388 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e9edb62a8e4b4dfea6a31a2202545a6e
I20251027 21:07:51.760641 10040 ts_tablet_manager.cc:1950] T 84305bcf21b64d2c8c1df015a2e9615c P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:51.764268 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_24aac5cecb8745d7be92287080a1b6e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.764801 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24aac5cecb8745d7be92287080a1b6e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.765657 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d5b5337ae438439d984f34a4e0af6c93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.766409 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8856c5dc8f144358445165324bfb477 consensus state has no leader
I20251027 21:07:51.766597 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 165) took 2169 ms. Trace:
I20251027 21:07:51.766680 10040 rpcz_store.cc:276] 1027 21:07:49.596851 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.598335 (+ 1484us) service_pool.cc:225] Handling call
1027 21:07:51.740103 (+2141768us) ts_tablet_manager.cc:1151] Deleting tablet 84305bcf21b64d2c8c1df015a2e9615c
1027 21:07:51.740112 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:51.757369 (+ 17257us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:51.760058 (+ 2689us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:51.766547 (+ 6489us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14430,"tablet-delete.queue_time_us":1921987}
I20251027 21:07:51.766767 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_506c51fbcfa745eb83b0b15a29c72e13" } modify_external_catalogs: true
I20251027 21:07:51.766927 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_506c51fbcfa745eb83b0b15a29c72e13" } modify_external_catalogs: true
I20251027 21:07:51.767349 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet dc589852710342f7bd017193e4075303
I20251027 21:07:51.767935 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet 10a44aecfade42969454126b990eca24 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:51.768260 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet a26ded6012174348b7295fc7c0e05e6e
I20251027 21:07:51.769169 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 84305bcf21b64d2c8c1df015a2e9615c (table default.Table_41a04598f12247fbac5f4103d2cbf22d [id=2e28a4d0f9204351b93335736a9abe18]) successfully deleted
W20251027 21:07:51.769411 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca477937278f45b4b115fec715f32a92 consensus state has no leader
I20251027 21:07:51.769496 9741 tablet_service.cc:1552] Processing DeleteTablet for tablet a26ded6012174348b7295fc7c0e05e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.769681 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet dc589852710342f7bd017193e4075303 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.769865 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet e9edb62a8e4b4dfea6a31a2202545a6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.770819 10029 tablet_bootstrap.cc:654] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:51.773160 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 174) took 1938 ms. Trace:
I20251027 21:07:51.773267 10040 rpcz_store.cc:276] 1027 21:07:49.834698 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:49.835498 (+ 800us) service_pool.cc:225] Handling call
1027 21:07:51.768590 (+1933092us) ts_tablet_manager.cc:1151] Deleting tablet 583fba7c750e4ac7b768da447002c211
1027 21:07:51.773006 (+ 4416us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:51.773108 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1922654}
I20251027 21:07:51.773774 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 188) took 1925 ms. Trace:
I20251027 21:07:51.773860 10040 rpcz_store.cc:276] 1027 21:07:49.848577 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:50.110087 (+261510us) service_pool.cc:225] Handling call
1027 21:07:51.773604 (+1663517us) ts_tablet_manager.cc:1151] Deleting tablet 10a44aecfade42969454126b990eca24
1027 21:07:51.773613 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:51.773689 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1655631}
I20251027 21:07:51.774416 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 189) took 1925 ms. Trace:
I20251027 21:07:51.774503 10040 rpcz_store.cc:276] 1027 21:07:49.848654 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:50.118000 (+269346us) service_pool.cc:225] Handling call
1027 21:07:51.774231 (+1656231us) ts_tablet_manager.cc:1151] Deleting tablet b87e809ff897417c88e5edc19e2d591e
1027 21:07:51.774238 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:51.774353 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1633070}
I20251027 21:07:51.774971 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 190) took 1926 ms. Trace:
I20251027 21:07:51.775066 10040 rpcz_store.cc:276] 1027 21:07:49.848752 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:50.141182 (+292430us) service_pool.cc:225] Handling call
1027 21:07:51.774831 (+1633649us) ts_tablet_manager.cc:1151] Deleting tablet 4e8c83b501d8426fa0cad6a89f9a0b9b
1027 21:07:51.774838 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:51.774912 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1606668}
W20251027 21:07:51.775738 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 583fba7c750e4ac7b768da447002c211 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 583fba7c750e4ac7b768da447002c211 already in progress: creating tablet
I20251027 21:07:51.775835 10031 tablet_bootstrap.cc:492] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
W20251027 21:07:51.775960 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 10a44aecfade42969454126b990eca24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10a44aecfade42969454126b990eca24 already in progress: creating tablet
I20251027 21:07:51.776000 10031 ts_tablet_manager.cc:1403] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.220s user 0.000s sys 0.005s
W20251027 21:07:51.776183 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet b87e809ff897417c88e5edc19e2d591e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b87e809ff897417c88e5edc19e2d591e already in progress: creating tablet
I20251027 21:07:51.776186 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 199) took 1853 ms. Trace:
I20251027 21:07:51.776285 10040 rpcz_store.cc:276] 1027 21:07:49.922934 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:50.439447 (+516513us) service_pool.cc:225] Handling call
1027 21:07:51.775974 (+1336527us) ts_tablet_manager.cc:1151] Deleting tablet 49b3d23798b740008c98cd0754ae33f4
1027 21:07:51.775982 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:51.776132 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1329620}
W20251027 21:07:51.776397 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 4e8c83b501d8426fa0cad6a89f9a0b9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e8c83b501d8426fa0cad6a89f9a0b9b already in progress: creating tablet
W20251027 21:07:51.776600 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 49b3d23798b740008c98cd0754ae33f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49b3d23798b740008c98cd0754ae33f4 already in progress: creating tablet
I20251027 21:07:51.776831 10031 raft_consensus.cc:359] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.777029 10031 raft_consensus.cc:385] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.777112 10031 raft_consensus.cc:740] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.777370 10031 consensus_queue.cc:260] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.777760 10040 tablet_replica.cc:333] T 87ec5a6b9b3c425da4e96cc6736d6310 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:51.778021 10031 ts_tablet_manager.cc:1434] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251027 21:07:51.778585 10040 raft_consensus.cc:2243] T 87ec5a6b9b3c425da4e96cc6736d6310 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.778767 10040 raft_consensus.cc:2272] T 87ec5a6b9b3c425da4e96cc6736d6310 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.779747 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_762d79ee19da4feb90eca3f718458fc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.780403 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_762d79ee19da4feb90eca3f718458fc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.780436 10040 ts_tablet_manager.cc:1916] T 87ec5a6b9b3c425da4e96cc6736d6310 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.781030 10031 tablet_bootstrap.cc:492] T 69b1fb42428748239d8939d98428d8fe P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:51.781580 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_857e62d8fe554d7886a5de26057618f7" } new_table_name: "default.Table_d11118d727fb4579a6465bd6deebaa43" modify_external_catalogs: true
W20251027 21:07:51.784464 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5b5337ae438439d984f34a4e0af6c93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.786612 9901 tablet_service.cc:1552] Processing DeleteTablet for tablet a26ded6012174348b7295fc7c0e05e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.787066 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 2a4aa1fe05114c1287ca580c1f411d3f (DEFAULT_TABLE table=default.Table_757ff3ea762e447ba76041b257234644 [id=63bc2d46b97e4054b557025436c371d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.787441 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a4aa1fe05114c1287ca580c1f411d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.790375 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 980eb6755aee4bcfb2ea44d84009021f (DEFAULT_TABLE table=default.Table_8a1e00e5debb481cbc1fdef16d74527b [id=42c8fedcabe0435dbaa11427141228ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.790715 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 980eb6755aee4bcfb2ea44d84009021f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.794266 9741 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c1a6a7616f24f69b83ec8a282746e92 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d659e4d73ea4ff3975a4b58a794090a) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.794667 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 2a4aa1fe05114c1287ca580c1f411d3f (DEFAULT_TABLE table=default.Table_757ff3ea762e447ba76041b257234644 [id=63bc2d46b97e4054b557025436c371d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.795145 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a4aa1fe05114c1287ca580c1f411d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.798398 9753 tablet_service.cc:1505] Processing CreateTablet for tablet da26c83976df40328cf9b11734fcfa44 (DEFAULT_TABLE table=default.Table_ac66ea1435fe49518c073dfc9b2368c8 [id=985d97a0b33f4ef79a9b0df362a75c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.799386 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet dc589852710342f7bd017193e4075303 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.799940 9900 tablet_service.cc:1552] Processing DeleteTablet for tablet f6249fc5c6174e8e81b9d02d598d5ebd with delete_type TABLET_DATA_DELETED (replaced by tablet 7d8825b80cec4ba68da077ca3b8d89db) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.800132 10041 tablet_replica.cc:333] T f6249fc5c6174e8e81b9d02d598d5ebd P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:51.800285 10041 raft_consensus.cc:2243] T f6249fc5c6174e8e81b9d02d598d5ebd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.800436 10041 raft_consensus.cc:2272] T f6249fc5c6174e8e81b9d02d598d5ebd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.801054 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da26c83976df40328cf9b11734fcfa44. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.801133 10041 ts_tablet_manager.cc:1916] T f6249fc5c6174e8e81b9d02d598d5ebd P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.806263 9309 master_service.cc:954] ReplaceTablet: received request to replace tablet 69b1fb42428748239d8939d98428d8fe from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:51.807281 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5dc2aa9f965e456dbf9b04d9032d4967" } new_table_name: "default.Table_660eb77e69884fe5bd5e65b47ea91dde" modify_external_catalogs: true
W20251027 21:07:51.808466 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet a26ded6012174348b7295fc7c0e05e6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a26ded6012174348b7295fc7c0e05e6e already in progress: creating tablet
W20251027 21:07:51.808828 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet dc589852710342f7bd017193e4075303 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc589852710342f7bd017193e4075303 already in progress: creating tablet
I20251027 21:07:51.812194 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7b970942803b42f29feb36d6d845167a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.812670 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b970942803b42f29feb36d6d845167a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.814074 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet ae124514964440c5b88b16ac9f8a2567 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:51.814709 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 8b30052ff19744a9bf4bfaaa1bc77b36 (DEFAULT_TABLE table=default.Table_5dc88891b75b4f1ca7b3ffba0d2b1fda [id=b46f1e4f1e6d4bd8b4a3edd1d6787612]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.814806 9598 tablet_service.cc:1505] Processing CreateTablet for tablet da26c83976df40328cf9b11734fcfa44 (DEFAULT_TABLE table=default.Table_ac66ea1435fe49518c073dfc9b2368c8 [id=985d97a0b33f4ef79a9b0df362a75c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.815120 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b30052ff19744a9bf4bfaaa1bc77b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.815372 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da26c83976df40328cf9b11734fcfa44. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.815987 9606 tablet_service.cc:1505] Processing CreateTablet for tablet c3b9a461567a48fcb4af514ecdf618b4 (DEFAULT_TABLE table=default.Table_9e88d5f78e27447c8072ce0563686ac8 [id=56f8db49ad8c4670a73a29bf4ed613e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.816468 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3b9a461567a48fcb4af514ecdf618b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.819145 10041 ts_tablet_manager.cc:1929] T f6249fc5c6174e8e81b9d02d598d5ebd P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.819348 10041 log.cc:1199] T f6249fc5c6174e8e81b9d02d598d5ebd P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/f6249fc5c6174e8e81b9d02d598d5ebd
I20251027 21:07:51.819747 10041 ts_tablet_manager.cc:1950] T f6249fc5c6174e8e81b9d02d598d5ebd P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:51.821201 10040 ts_tablet_manager.cc:1929] T 87ec5a6b9b3c425da4e96cc6736d6310 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:07:51.821190 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet f6249fc5c6174e8e81b9d02d598d5ebd (table default.Table_943417713b3b456ab4ca691d2fb168fd [id=1cb32eea25114c6dbcb3b28044036c08]) successfully deleted
I20251027 21:07:51.821321 10040 log.cc:1199] T 87ec5a6b9b3c425da4e96cc6736d6310 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/87ec5a6b9b3c425da4e96cc6736d6310
I20251027 21:07:51.822526 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 132a01405a63447ab27b2137b803224f
I20251027 21:07:51.823261 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 132a01405a63447ab27b2137b803224f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.823338 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c3f0afa8e76249b69b9fbe4b3f67d020"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:51.823632 10040 ts_tablet_manager.cc:1950] T 87ec5a6b9b3c425da4e96cc6736d6310 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
W20251027 21:07:51.823787 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3f0afa8e76249b69b9fbe4b3f67d020 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.823907 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 200) took 1900 ms. Trace:
I20251027 21:07:51.824018 10040 rpcz_store.cc:276] 1027 21:07:49.923016 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:50.440496 (+517480us) service_pool.cc:225] Handling call
1027 21:07:51.777684 (+1337188us) ts_tablet_manager.cc:1151] Deleting tablet 87ec5a6b9b3c425da4e96cc6736d6310
1027 21:07:51.777693 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:51.786401 (+ 8708us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:51.821088 (+ 34687us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:51.823870 (+ 2782us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":614,"tablet-delete.queue_time_us":1328990}
I20251027 21:07:51.824795 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet e9edb62a8e4b4dfea6a31a2202545a6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.826771 10031 tablet_bootstrap.cc:654] T 69b1fb42428748239d8939d98428d8fe P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:51.828307 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet 132a01405a63447ab27b2137b803224f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.828420 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 87ec5a6b9b3c425da4e96cc6736d6310 (table default.Table_0c05919c1a104864a4e1cd182ae59e88 [id=16df16d9dc6144cabc7f453c319255d9]) successfully deleted
I20251027 21:07:51.828586 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 204) took 1831 ms. Trace:
I20251027 21:07:51.828698 10040 rpcz_store.cc:276] 1027 21:07:49.997112 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:50.460423 (+463311us) service_pool.cc:225] Handling call
1027 21:07:51.828452 (+1368029us) ts_tablet_manager.cc:1151] Deleting tablet 970fbf01a4c3406dafe1ea4cd005ea23
1027 21:07:51.828460 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:51.828558 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1365906}
W20251027 21:07:51.829048 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 970fbf01a4c3406dafe1ea4cd005ea23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 970fbf01a4c3406dafe1ea4cd005ea23 already in progress: creating tablet
I20251027 21:07:51.829607 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 205) took 1832 ms. Trace:
I20251027 21:07:51.829996 9306 master_service.cc:954] ReplaceTablet: received request to replace tablet 26872be132fb46ee9e79a9c2e62374ef from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:51.829720 10040 rpcz_store.cc:276] 1027 21:07:49.997226 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:50.462576 (+465350us) service_pool.cc:225] Handling call
1027 21:07:51.829500 (+1366924us) ts_tablet_manager.cc:1151] Deleting tablet 9248f458baca409aa3505cc0db321b34
1027 21:07:51.829508 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:51.829579 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1365437}
W20251027 21:07:51.830732 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 9248f458baca409aa3505cc0db321b34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9248f458baca409aa3505cc0db321b34 already in progress: creating tablet
W20251027 21:07:51.831441 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet e9edb62a8e4b4dfea6a31a2202545a6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9edb62a8e4b4dfea6a31a2202545a6e already in progress: creating tablet
W20251027 21:07:51.832140 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 132a01405a63447ab27b2137b803224f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 132a01405a63447ab27b2137b803224f already in progress: creating tablet
I20251027 21:07:51.831879 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 206) took 1466 ms. Trace:
I20251027 21:07:51.832381 10040 rpcz_store.cc:276] 1027 21:07:50.365079 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:50.464094 (+ 99015us) service_pool.cc:225] Handling call
1027 21:07:51.831737 (+1367643us) ts_tablet_manager.cc:1151] Deleting tablet 49d9e5c5960a4534a6603875006e6266
1027 21:07:51.831745 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:51.831845 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1367412}
W20251027 21:07:51.832772 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 49d9e5c5960a4534a6603875006e6266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49d9e5c5960a4534a6603875006e6266 already in progress: creating tablet
I20251027 21:07:51.833086 10040 tablet_replica.cc:333] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:51.833323 10040 raft_consensus.cc:2243] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:07:51.833709 10040 pending_rounds.cc:70] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
I20251027 21:07:51.833801 10040 pending_rounds.cc:77] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510609310023680 op_type: NO_OP noop_request { }
I20251027 21:07:51.833911 10040 raft_consensus.cc:2889] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:07:51.834036 10040 raft_consensus.cc:2272] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.834584 10040 ts_tablet_manager.cc:1916] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.836100 9303 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 90a8be40f21a4356b22b4ab90b0bddea
W20251027 21:07:51.836685 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1d64a13667194c12864fa388eb88a380 has no consensus state
W20251027 21:07:51.840624 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f34512059f0c47fc9af7db2645f86b40 consensus state has no leader
I20251027 21:07:51.843253 9611 tablet_service.cc:1552] Processing DeleteTablet for tablet e9edb62a8e4b4dfea6a31a2202545a6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.843490 9605 tablet_service.cc:1505] Processing CreateTablet for tablet c9efffa9766e46aca8eb286fc2ad48d2 (DEFAULT_TABLE table=default.Table_97b01c1840c543838263ca4222695f54 [id=bea51a58fda64d40916f306c4aa8ab4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.732455 10032 consensus_queue.cc:260] T d89c3c588a1747d7875ebf45924fb890 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:51.845132 10032 ts_tablet_manager.cc:1434] T d89c3c588a1747d7875ebf45924fb890 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.114s user 0.001s sys 0.001s
I20251027 21:07:51.845544 10032 tablet_bootstrap.cc:492] T 019a892bfdd54e62bc3560a341d5c2da P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:51.847317 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet ae124514964440c5b88b16ac9f8a2567
I20251027 21:07:51.847793 9307 catalog_manager.cc:6719] ReplaceTablet: tablet ae124514964440c5b88b16ac9f8a2567 deleted and replaced by tablet 01e9db9eaf8e41c6a2ee13b9ed127421
I20251027 21:07:51.848834 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet 49d9e5c5960a4534a6603875006e6266 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:51.850441 9890 tablet_service.cc:1505] Processing CreateTablet for tablet bbb4116d1edd492893adfbf5fb3ddeef (DEFAULT_TABLE table=default.Table_dfa3f51576fa4c689b2682b0e277df60 [id=088a35e69cc840c0b9386df033a885a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.851050 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbb4116d1edd492893adfbf5fb3ddeef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.851109 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet ae124514964440c5b88b16ac9f8a2567 with delete_type TABLET_DATA_DELETED (replaced by tablet 01e9db9eaf8e41c6a2ee13b9ed127421) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.851315 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 9fde4fd91943498d8888d1783005e492 (DEFAULT_TABLE table=default.Table_c6c89ff2d21c4e4db430817f6398887e [id=3a0c8b19df934d1580349b061fe0e8f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.851498 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 980eb6755aee4bcfb2ea44d84009021f (DEFAULT_TABLE table=default.Table_8a1e00e5debb481cbc1fdef16d74527b [id=42c8fedcabe0435dbaa11427141228ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.851766 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fde4fd91943498d8888d1783005e492. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.851842 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 980eb6755aee4bcfb2ea44d84009021f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.852236 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 94f2819dd0394530ae93754bdd1e482f (DEFAULT_TABLE table=default.Table_979a3d9dccc74a86a0cf5c725bd8cf2b [id=63147f61013c4d6dbc96fd25769b6136]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.852629 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94f2819dd0394530ae93754bdd1e482f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.854379 9741 tablet_service.cc:1505] Processing CreateTablet for tablet 94f2819dd0394530ae93754bdd1e482f (DEFAULT_TABLE table=default.Table_979a3d9dccc74a86a0cf5c725bd8cf2b [id=63147f61013c4d6dbc96fd25769b6136]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.854708 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94f2819dd0394530ae93754bdd1e482f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.855454 9750 tablet_service.cc:1505] Processing CreateTablet for tablet bbb4116d1edd492893adfbf5fb3ddeef (DEFAULT_TABLE table=default.Table_dfa3f51576fa4c689b2682b0e277df60 [id=088a35e69cc840c0b9386df033a885a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.855719 10141 raft_consensus.cc:493] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:51.855847 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbb4116d1edd492893adfbf5fb3ddeef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.855911 10141 raft_consensus.cc:515] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.856612 10141 leader_election.cc:290] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:51.857831 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c1a6a7616f24f69b83ec8a282746e92" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:51.858434 9710 leader_election.cc:343] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: Tablet not RUNNING: STOPPED
I20251027 21:07:51.860352 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 26872be132fb46ee9e79a9c2e62374ef
I20251027 21:07:51.860157 10299 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 3 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:51.860764 9306 catalog_manager.cc:6719] ReplaceTablet: tablet 26872be132fb46ee9e79a9c2e62374ef deleted and replaced by tablet 6409f45a907b4389bac84093cdc3c30f
I20251027 21:07:51.861584 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 26872be132fb46ee9e79a9c2e62374ef with delete_type TABLET_DATA_DELETED (replaced by tablet 6409f45a907b4389bac84093cdc3c30f) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.862423 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9efffa9766e46aca8eb286fc2ad48d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.863466 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet d89c3c588a1747d7875ebf45924fb890 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:51.863615 9711 leader_election.cc:343] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 0c1a6a7616f24f69b83ec8a282746e92
I20251027 21:07:51.863751 9711 leader_election.cc:304] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:51.863960 10141 raft_consensus.cc:2749] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:51.863699 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet 26872be132fb46ee9e79a9c2e62374ef with delete_type TABLET_DATA_DELETED (replaced by tablet 6409f45a907b4389bac84093cdc3c30f) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.864408 10041 tablet_replica.cc:333] T d89c3c588a1747d7875ebf45924fb890 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:51.864583 9611 tablet_service.cc:1552] Processing DeleteTablet for tablet dc589852710342f7bd017193e4075303 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.864691 10041 raft_consensus.cc:2243] T d89c3c588a1747d7875ebf45924fb890 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.864841 10041 raft_consensus.cc:2272] T d89c3c588a1747d7875ebf45924fb890 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.866699 10041 ts_tablet_manager.cc:1916] T d89c3c588a1747d7875ebf45924fb890 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.867228 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 2a4aa1fe05114c1287ca580c1f411d3f (DEFAULT_TABLE table=default.Table_757ff3ea762e447ba76041b257234644 [id=63bc2d46b97e4054b557025436c371d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.867311 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 9fde4fd91943498d8888d1783005e492 (DEFAULT_TABLE table=default.Table_c6c89ff2d21c4e4db430817f6398887e [id=3a0c8b19df934d1580349b061fe0e8f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.867656 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fde4fd91943498d8888d1783005e492. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.867677 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a4aa1fe05114c1287ca580c1f411d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.868876 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 980eb6755aee4bcfb2ea44d84009021f (DEFAULT_TABLE table=default.Table_8a1e00e5debb481cbc1fdef16d74527b [id=42c8fedcabe0435dbaa11427141228ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.870253 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 980eb6755aee4bcfb2ea44d84009021f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.870910 9737 tablet_service.cc:1505] Processing CreateTablet for tablet aaa8910d9fdb4146945621ec4ebf969e (DEFAULT_TABLE table=default.Table_5fe0ef16a24f4af9ae8e843adbe1643e [id=bb676562b97f48088767941239cfbf33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.871240 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaa8910d9fdb4146945621ec4ebf969e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.871294 10031 tablet_bootstrap.cc:492] T 69b1fb42428748239d8939d98428d8fe P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:51.871439 10031 ts_tablet_manager.cc:1403] T 69b1fb42428748239d8939d98428d8fe P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.090s user 0.000s sys 0.003s
I20251027 21:07:51.872797 10031 raft_consensus.cc:359] T 69b1fb42428748239d8939d98428d8fe P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.872946 10031 raft_consensus.cc:385] T 69b1fb42428748239d8939d98428d8fe P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.872911 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_16dd2e0479ea434dbbd0b900b340fc9f" } new_table_name: "default.Table_ed23972576b74b5fbc7167cf7b541bfc" modify_external_catalogs: true
I20251027 21:07:51.873023 10031 raft_consensus.cc:740] T 69b1fb42428748239d8939d98428d8fe P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.873441 10031 consensus_queue.cc:260] T 69b1fb42428748239d8939d98428d8fe P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.874385 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 04c579bb008448e9baa841135b6dbc61 (DEFAULT_TABLE table=default.Table_13a109a52d1a42aca531982896796ff7 [id=73cba162a3af4c0fa5dfde81e1a81257]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.874763 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04c579bb008448e9baa841135b6dbc61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.875178 10031 ts_tablet_manager.cc:1434] T 69b1fb42428748239d8939d98428d8fe P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251027 21:07:51.877476 9746 tablet_service.cc:1552] Processing DeleteTablet for tablet 69b1fb42428748239d8939d98428d8fe with delete_type TABLET_DATA_DELETED (replaced by tablet 49b3d23798b740008c98cd0754ae33f4) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.878368 10032 tablet_bootstrap.cc:654] T 019a892bfdd54e62bc3560a341d5c2da P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:51.879437 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet a26ded6012174348b7295fc7c0e05e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.880925 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_00deabd570914e54b7fa9c4fc4eb02c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:51.882325 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 59b6818d29624da5a49dd1a7d20994c6 (DEFAULT_TABLE table=default.Table_9904948d307e479ea184d6dc441b5e61 [id=500c0fe58264475087ef89652bf83263]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:51.883154 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00deabd570914e54b7fa9c4fc4eb02c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.886216 10031 tablet_bootstrap.cc:492] T b1bdb2be5032415187c2826c14e75f07 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:51.886859 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 59b6818d29624da5a49dd1a7d20994c6 (DEFAULT_TABLE table=default.Table_9904948d307e479ea184d6dc441b5e61 [id=500c0fe58264475087ef89652bf83263]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.887224 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59b6818d29624da5a49dd1a7d20994c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.888051 10031 tablet_bootstrap.cc:654] T b1bdb2be5032415187c2826c14e75f07 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:51.892086 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 631ff72613734913818a3cad24f20dbc has no consensus state
I20251027 21:07:51.892467 10031 tablet_bootstrap.cc:492] T b1bdb2be5032415187c2826c14e75f07 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:51.892714 10031 ts_tablet_manager.cc:1403] T b1bdb2be5032415187c2826c14e75f07 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20251027 21:07:51.895424 10031 raft_consensus.cc:359] T b1bdb2be5032415187c2826c14e75f07 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.895900 10031 raft_consensus.cc:385] T b1bdb2be5032415187c2826c14e75f07 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.896224 10031 raft_consensus.cc:740] T b1bdb2be5032415187c2826c14e75f07 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.896637 10031 consensus_queue.cc:260] T b1bdb2be5032415187c2826c14e75f07 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.896996 9904 tablet_service.cc:1505] Processing CreateTablet for tablet aaa8910d9fdb4146945621ec4ebf969e (DEFAULT_TABLE table=default.Table_5fe0ef16a24f4af9ae8e843adbe1643e [id=bb676562b97f48088767941239cfbf33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.897447 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 with delete_type TABLET_DATA_DELETED (replaced by tablet e07b8aa0a41241d79254c4dae8738e79) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.897591 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet e3c9f64ee0544e59add4f73bca59df98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.898752 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet d8ef795c88da49b58082ba1264d83378 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.899504 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 04c579bb008448e9baa841135b6dbc61 (DEFAULT_TABLE table=default.Table_13a109a52d1a42aca531982896796ff7 [id=73cba162a3af4c0fa5dfde81e1a81257]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.899899 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59b6818d29624da5a49dd1a7d20994c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.900745 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04c579bb008448e9baa841135b6dbc61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.904193 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaa8910d9fdb4146945621ec4ebf969e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.905221 10041 ts_tablet_manager.cc:1929] T d89c3c588a1747d7875ebf45924fb890 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.905393 10041 log.cc:1199] T d89c3c588a1747d7875ebf45924fb890 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/d89c3c588a1747d7875ebf45924fb890
I20251027 21:07:51.905766 10041 ts_tablet_manager.cc:1950] T d89c3c588a1747d7875ebf45924fb890 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:51.906363 10031 ts_tablet_manager.cc:1434] T b1bdb2be5032415187c2826c14e75f07 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251027 21:07:51.907431 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet b1bdb2be5032415187c2826c14e75f07 with delete_type TABLET_DATA_DELETED (replaced by tablet f1aa5f7f76cb4a1ba15f016942f17067) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:51.908829 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet 36e168cc44274dae9682b0c1329d4558 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.910755 10031 tablet_bootstrap.cc:492] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:51.910782 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet b1bdb2be5032415187c2826c14e75f07 with delete_type TABLET_DATA_DELETED (replaced by tablet f1aa5f7f76cb4a1ba15f016942f17067) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.911834 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet ae124514964440c5b88b16ac9f8a2567 with delete_type TABLET_DATA_DELETED (replaced by tablet 01e9db9eaf8e41c6a2ee13b9ed127421) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.912549 10031 tablet_bootstrap.cc:654] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:51.913318 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet 132a01405a63447ab27b2137b803224f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.913720 10032 tablet_bootstrap.cc:492] T 019a892bfdd54e62bc3560a341d5c2da P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:51.913902 10032 ts_tablet_manager.cc:1403] T 019a892bfdd54e62bc3560a341d5c2da P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.002s
I20251027 21:07:51.915979 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet ae124514964440c5b88b16ac9f8a2567 with delete_type TABLET_DATA_DELETED (replaced by tablet 01e9db9eaf8e41c6a2ee13b9ed127421) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.917150 10032 raft_consensus.cc:359] T 019a892bfdd54e62bc3560a341d5c2da P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:51.918253 10032 raft_consensus.cc:385] T 019a892bfdd54e62bc3560a341d5c2da P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.918342 10032 raft_consensus.cc:740] T 019a892bfdd54e62bc3560a341d5c2da P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.918588 10032 consensus_queue.cc:260] T 019a892bfdd54e62bc3560a341d5c2da P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:51.919173 10032 ts_tablet_manager.cc:1434] T 019a892bfdd54e62bc3560a341d5c2da P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251027 21:07:51.921837 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet d89c3c588a1747d7875ebf45924fb890 (table default.Table_f05923c6594b43378fa156d0ab2bd1da [id=ad541dd12124493583f4fdba032b7ed4]) successfully deleted
W20251027 21:07:51.922372 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 26872be132fb46ee9e79a9c2e62374ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26872be132fb46ee9e79a9c2e62374ef already in progress: creating tablet
W20251027 21:07:51.922667 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet ae124514964440c5b88b16ac9f8a2567 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae124514964440c5b88b16ac9f8a2567 already in progress: creating tablet
I20251027 21:07:51.930531 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 94f2819dd0394530ae93754bdd1e482f (DEFAULT_TABLE table=default.Table_979a3d9dccc74a86a0cf5c725bd8cf2b [id=63147f61013c4d6dbc96fd25769b6136]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.936038 10032 tablet_bootstrap.cc:492] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:51.937868 10032 tablet_bootstrap.cc:654] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:51.944439 9604 tablet_service.cc:1505] Processing CreateTablet for tablet bbb4116d1edd492893adfbf5fb3ddeef (DEFAULT_TABLE table=default.Table_dfa3f51576fa4c689b2682b0e277df60 [id=088a35e69cc840c0b9386df033a885a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.932191 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet 019a892bfdd54e62bc3560a341d5c2da with delete_type TABLET_DATA_DELETED (replaced by tablet ccaf5c2f8e544842880ba8ba27300ae6) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:51.938628 9970 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:07:51.947657 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet 26872be132fb46ee9e79a9c2e62374ef with delete_type TABLET_DATA_DELETED (replaced by tablet 6409f45a907b4389bac84093cdc3c30f) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:51.949180 10041 tablet_replica.cc:333] T 019a892bfdd54e62bc3560a341d5c2da P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:51.949381 10041 raft_consensus.cc:2243] T 019a892bfdd54e62bc3560a341d5c2da P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:51.949553 10041 raft_consensus.cc:2272] T 019a892bfdd54e62bc3560a341d5c2da P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:51.950321 10032 tablet_bootstrap.cc:492] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:51.950297 10041 ts_tablet_manager.cc:1916] T 019a892bfdd54e62bc3560a341d5c2da P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:51.959795 10041 ts_tablet_manager.cc:1929] T 019a892bfdd54e62bc3560a341d5c2da P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.960156 10041 log.cc:1199] T 019a892bfdd54e62bc3560a341d5c2da P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/019a892bfdd54e62bc3560a341d5c2da
I20251027 21:07:51.960569 10041 ts_tablet_manager.cc:1950] T 019a892bfdd54e62bc3560a341d5c2da P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:51.961602 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 019a892bfdd54e62bc3560a341d5c2da (table default.Table_18e48b818fc040bfa4272f08e203dc3e [id=2b0d4e085a8e42879c67963231a24b03]) successfully deleted
I20251027 21:07:51.961659 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 9fde4fd91943498d8888d1783005e492 (DEFAULT_TABLE table=default.Table_c6c89ff2d21c4e4db430817f6398887e [id=3a0c8b19df934d1580349b061fe0e8f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.960031 10032 ts_tablet_manager.cc:1403] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20251027 21:07:51.961993 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94f2819dd0394530ae93754bdd1e482f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.962674 10032 raft_consensus.cc:359] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:51.962850 10032 raft_consensus.cc:385] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.962921 10032 raft_consensus.cc:740] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.963129 10032 consensus_queue.cc:260] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:51.963577 10032 ts_tablet_manager.cc:1434] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:07:51.963900 10032 tablet_bootstrap.cc:492] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:51.966077 9307 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 8fcfa6b85e7b46d78a84947e9f11ae46 (table default.Table_4723d51b934b4f6a90c790ee4bc7ea86 [id=e68b63cf420d49289a5bd1db81e1cbc1]). Expected version 2 got 0
I20251027 21:07:51.966332 9600 tablet_service.cc:1505] Processing CreateTablet for tablet aaa8910d9fdb4146945621ec4ebf969e (DEFAULT_TABLE table=default.Table_5fe0ef16a24f4af9ae8e843adbe1643e [id=bb676562b97f48088767941239cfbf33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.966663 10032 tablet_bootstrap.cc:654] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:51.966683 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaa8910d9fdb4146945621ec4ebf969e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.968168 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_917a7cf5e19547888798a43ded99f4b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.968674 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_917a7cf5e19547888798a43ded99f4b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.970224 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 04c579bb008448e9baa841135b6dbc61 (DEFAULT_TABLE table=default.Table_13a109a52d1a42aca531982896796ff7 [id=73cba162a3af4c0fa5dfde81e1a81257]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.970549 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04c579bb008448e9baa841135b6dbc61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.973798 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2ff5a5b701c540b8a9eb8b87aace0e77" } modify_external_catalogs: true
I20251027 21:07:51.973990 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2ff5a5b701c540b8a9eb8b87aace0e77" } modify_external_catalogs: true
I20251027 21:07:51.974014 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c76923735ab34c5fbbbba0443cbc54c5" } new_table_name: "default.Table_69627cab65c2407b9226a05106b7bdf4" modify_external_catalogs: true
I20251027 21:07:51.974861 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 59b6818d29624da5a49dd1a7d20994c6 (DEFAULT_TABLE table=default.Table_9904948d307e479ea184d6dc441b5e61 [id=500c0fe58264475087ef89652bf83263]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:51.975239 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fde4fd91943498d8888d1783005e492. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.979794 10029 tablet_bootstrap.cc:492] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:51.980000 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59b6818d29624da5a49dd1a7d20994c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.980368 10043 ts_tablet_manager.cc:1929] T 0c1a6a7616f24f69b83ec8a282746e92 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:51.983291 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4a455030bbe945c391d97696ee50010e" } modify_external_catalogs: true
I20251027 21:07:51.983481 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4a455030bbe945c391d97696ee50010e" } modify_external_catalogs: true
I20251027 21:07:51.984273 10029 ts_tablet_manager.cc:1403] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.220s user 0.009s sys 0.008s
W20251027 21:07:51.985989 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca477937278f45b4b115fec715f32a92 consensus state has no leader
I20251027 21:07:51.986238 10032 tablet_bootstrap.cc:492] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:51.987154 10032 ts_tablet_manager.cc:1403] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20251027 21:07:51.987486 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_052e3f023dfd462884130780e8c57936"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:51.987952 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_052e3f023dfd462884130780e8c57936 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.988152 10032 raft_consensus.cc:359] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:51.988293 10032 raft_consensus.cc:385] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.988368 10032 raft_consensus.cc:740] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.988924 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbb4116d1edd492893adfbf5fb3ddeef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:51.988991 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_31d10565065d494194f0dda082708ac0" } new_table_name: "default.Table_5e48ead40e6e4f6f9ae841b8fd6d3629" modify_external_catalogs: true
I20251027 21:07:51.990449 10032 consensus_queue.cc:260] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:51.990671 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9c027ff744824da9b9cac5574017e7fd" } modify_external_catalogs: true
I20251027 21:07:51.990804 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9c027ff744824da9b9cac5574017e7fd" } modify_external_catalogs: true
I20251027 21:07:51.991326 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_03200469a2e740baab131dd8bcbe9da7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:51.991441 10032 ts_tablet_manager.cc:1434] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251027 21:07:51.991693 10032 tablet_bootstrap.cc:492] T 601c74459d3a4a41b812fdcdf8b0a62c P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:07:51.991784 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03200469a2e740baab131dd8bcbe9da7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:51.997844 10312 raft_consensus.cc:493] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:51.998023 10312 raft_consensus.cc:515] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:51.998734 10312 leader_election.cc:290] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:51.999408 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8ef795c88da49b58082ba1264d83378" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:51.999406 10029 raft_consensus.cc:359] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:51.999552 10029 raft_consensus.cc:385] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:51.999603 10032 tablet_bootstrap.cc:654] T 601c74459d3a4a41b812fdcdf8b0a62c P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:51.999629 10029 raft_consensus.cc:740] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:51.999817 10029 consensus_queue.cc:260] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
W20251027 21:07:52.000190 9566 leader_election.cc:343] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:52.000284 10029 ts_tablet_manager.cc:1434] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.016s user 0.000s sys 0.000s
I20251027 21:07:52.000559 10029 tablet_bootstrap.cc:492] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:52.002701 10043 log.cc:1199] T 0c1a6a7616f24f69b83ec8a282746e92 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/0c1a6a7616f24f69b83ec8a282746e92
I20251027 21:07:52.003180 10043 ts_tablet_manager.cc:1950] T 0c1a6a7616f24f69b83ec8a282746e92 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:52.004148 10043 tablet_replica.cc:333] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:52.004374 10043 raft_consensus.cc:2243] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.004489 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 9b04349b15f3494bb0933db56b7b24bc
I20251027 21:07:52.004647 10043 raft_consensus.cc:2272] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.005213 10043 ts_tablet_manager.cc:1916] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.005455 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b04349b15f3494bb0933db56b7b24bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.005832 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b04349b15f3494bb0933db56b7b24bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:52.006062 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 9b04349b15f3494bb0933db56b7b24bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b04349b15f3494bb0933db56b7b24bc already in progress: creating tablet
I20251027 21:07:52.011138 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 0c1a6a7616f24f69b83ec8a282746e92 (table default.Table_e9fd9bee6a2a40d7ad92d2d31b03afce [id=af6cb904f8cd4a929be7a978f9bc2001]) successfully deleted
I20251027 21:07:52.013543 10040 ts_tablet_manager.cc:1929] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:07:52.013691 10040 log.cc:1199] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/9cec6294051b41868c9c7c7fadafa138
I20251027 21:07:52.014097 10040 ts_tablet_manager.cc:1950] T 9cec6294051b41868c9c7c7fadafa138 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:52.014894 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 207) took 1635 ms. Trace:
I20251027 21:07:52.015014 10040 rpcz_store.cc:276] 1027 21:07:50.379628 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:50.464341 (+ 84713us) service_pool.cc:225] Handling call
1027 21:07:51.833023 (+1368682us) ts_tablet_manager.cc:1151] Deleting tablet 9cec6294051b41868c9c7c7fadafa138
1027 21:07:51.833030 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:51.850376 (+ 17346us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:52.013412 (+163036us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:52.014862 (+ 1450us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1368496}
I20251027 21:07:52.015455 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 9cec6294051b41868c9c7c7fadafa138 (table default.Table_5d8e4122af254070a2df7016e521a937 [id=12b1bbc770194455ab7292641980e8e1]) successfully deleted
I20251027 21:07:52.022022 9306 master_service.cc:954] ReplaceTablet: received request to replace tablet c400101d2a39490e83722500839198cd from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:52.022531 10040 tablet_replica.cc:333] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
W20251027 21:07:52.024859 9568 leader_election.cc:343] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: d8ef795c88da49b58082ba1264d83378
I20251027 21:07:52.025108 10029 tablet_bootstrap.cc:654] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.025337 9568 leader_election.cc:304] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
W20251027 21:07:52.025945 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d56d323ef0e74344a479940b93fea11a consensus state has no leader
I20251027 21:07:52.026401 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b04349b15f3494bb0933db56b7b24bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.027992 10040 raft_consensus.cc:2243] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.027959 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1ecaac632a024248b04e7c40e2c01d2c" } modify_external_catalogs: true
I20251027 21:07:52.028128 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1ecaac632a024248b04e7c40e2c01d2c" } modify_external_catalogs: true
I20251027 21:07:52.028264 10040 raft_consensus.cc:2272] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.028844 10040 ts_tablet_manager.cc:1916] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.030627 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_605ef0bcf3ad4d919afc3a73ac0ab926"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.031082 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_605ef0bcf3ad4d919afc3a73ac0ab926 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.032348 10312 raft_consensus.cc:2749] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:52.032608 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet 61cd8ee8e1f8459eb7af5dd48c018e02 with delete_type TABLET_DATA_DELETED (replaced by tablet 49d9e5c5960a4534a6603875006e6266) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.035511 10031 tablet_bootstrap.cc:492] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:52.035653 10031 ts_tablet_manager.cc:1403] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.125s user 0.014s sys 0.018s
I20251027 21:07:52.038095 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet abc7458fcfb94985acf76d77c0292b08
I20251027 21:07:52.038990 10031 raft_consensus.cc:359] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.039177 10031 raft_consensus.cc:385] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.039426 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet abc7458fcfb94985acf76d77c0292b08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.039716 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet abc7458fcfb94985acf76d77c0292b08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:52.040086 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet abc7458fcfb94985acf76d77c0292b08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abc7458fcfb94985acf76d77c0292b08 already in progress: creating tablet
W20251027 21:07:52.040350 9299 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet a8328318a4f14b38a4ce36d183e99dfb (table default.Table_318a68ed05144ed9aeb2db53570ab9c3 [id=287175ad40544f73a586c736fcce910b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a8328318a4f14b38a4ce36d183e99dfb consensus state has no leader
I20251027 21:07:52.039253 10031 raft_consensus.cc:740] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.041905 10031 consensus_queue.cc:260] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.042181 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e5aa7dc0a7da4c1cb2887c10710fbdf1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.042733 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5aa7dc0a7da4c1cb2887c10710fbdf1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.044437 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0c82ef1916f14d07929b28bffd296f9f" } new_table_name: "default.Table_ac768571ea3e43e4af7551ee98801793" modify_external_catalogs: true
I20251027 21:07:52.044679 10031 ts_tablet_manager.cc:1434] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251027 21:07:52.044965 10031 tablet_bootstrap.cc:492] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:52.045725 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_cfb36d57b28d49f88f50a05650a40b49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.046218 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cfb36d57b28d49f88f50a05650a40b49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.049095 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 with delete_type TABLET_DATA_DELETED (replaced by tablet e07b8aa0a41241d79254c4dae8738e79) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.051759 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_38e99066d2014865a10ad4f614cf0339"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.052266 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38e99066d2014865a10ad4f614cf0339 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.056247 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5877315061df40eb80ba9d966632471a" } modify_external_catalogs: true
I20251027 21:07:52.056418 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5877315061df40eb80ba9d966632471a" } modify_external_catalogs: true
I20251027 21:07:52.056483 10299 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e86ca5e83dd4df5b72f045e7b787e81, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.057247 10299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 37, Last appended: 3.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:52.057893 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet abc7458fcfb94985acf76d77c0292b08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.060421 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet a339c0dee0994788b8644912f74c1b82
I20251027 21:07:52.061208 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet a339c0dee0994788b8644912f74c1b82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.061569 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet a339c0dee0994788b8644912f74c1b82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.061684 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet a287439ea6c9469da5fb56d84ee68048
I20251027 21:07:52.062202 10043 ts_tablet_manager.cc:1929] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.062340 10043 log.cc:1199] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/cbada3946dab4739a8369032ff3b8a48
I20251027 21:07:52.062355 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet a287439ea6c9469da5fb56d84ee68048 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.062687 10043 ts_tablet_manager.cc:1950] T cbada3946dab4739a8369032ff3b8a48 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
W20251027 21:07:52.063040 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet a339c0dee0994788b8644912f74c1b82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a339c0dee0994788b8644912f74c1b82 already in progress: creating tablet
I20251027 21:07:52.063457 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 220) took 1022 ms. Trace:
I20251027 21:07:52.063606 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet a287439ea6c9469da5fb56d84ee68048 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:52.065779 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet a287439ea6c9469da5fb56d84ee68048 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a287439ea6c9469da5fb56d84ee68048 already in progress: creating tablet
I20251027 21:07:52.066622 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c814f5ad5b6f46f2aa8fc2ecf7ce2d73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.067117 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c814f5ad5b6f46f2aa8fc2ecf7ce2d73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.068351 10043 rpcz_store.cc:276] 1027 21:07:51.041414 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:51.041457 (+ 43us) service_pool.cc:225] Handling call
1027 21:07:52.004062 (+962605us) ts_tablet_manager.cc:1151] Deleting tablet cbada3946dab4739a8369032ff3b8a48
1027 21:07:52.004081 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:52.044276 (+ 40195us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:52.047594 (+ 3318us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:52.063424 (+ 15830us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":955649}
I20251027 21:07:52.069514 10029 tablet_bootstrap.cc:492] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:52.069720 10029 ts_tablet_manager.cc:1403] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.069s user 0.001s sys 0.005s
I20251027 21:07:52.070526 10029 raft_consensus.cc:359] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.070776 10029 raft_consensus.cc:385] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.070894 10029 raft_consensus.cc:740] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.071172 10029 consensus_queue.cc:260] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.075776 10029 ts_tablet_manager.cc:1434] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251027 21:07:52.077001 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d276ca9d745f41a58deb7846600ddcc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.077468 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d276ca9d745f41a58deb7846600ddcc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.078531 10029 tablet_bootstrap.cc:492] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:52.080049 10029 tablet_bootstrap.cc:654] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.080894 10043 tablet_replica.cc:333] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:52.081061 10043 raft_consensus.cc:2243] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:07:52.081256 10043 pending_rounds.cc:70] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14: Trying to abort 1 pending ops.
I20251027 21:07:52.081313 10043 pending_rounds.cc:77] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510615990067200 op_type: NO_OP noop_request { }
I20251027 21:07:52.081393 10043 raft_consensus.cc:2889] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:07:52.081480 10043 raft_consensus.cc:2272] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251027 21:07:52.089083 9711 consensus_peers.cc:597] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21b785277d604543bfa7816ed96d6181. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:07:52.089448 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5d7b7b6b754149609ebbb51ebf00b364" } new_table_name: "default.Table_c78ef9ebe86b4da08443e37c83d4e80d" modify_external_catalogs: true
I20251027 21:07:52.093367 10040 ts_tablet_manager.cc:1929] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.093516 10040 log.cc:1199] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/65fbcaa5447c48818fabfccac22c4048
I20251027 21:07:52.093853 10040 ts_tablet_manager.cc:1950] T 65fbcaa5447c48818fabfccac22c4048 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:52.094482 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8d288192611b43af971f1e081bb362ad" } new_table_name: "default.Table_89407f0c58ec4ee7995b892701a6377f" modify_external_catalogs: true
I20251027 21:07:52.095199 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet a339c0dee0994788b8644912f74c1b82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.095757 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet a287439ea6c9469da5fb56d84ee68048 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.096333 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet cad3a3e910ac42399cf371ad82c89ff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.098657 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet cbada3946dab4739a8369032ff3b8a48 (table default.Table_4f6664b7160e4549a50589454ff10313 [id=67b71acdadc64445bb2492e5d424cb2d]) successfully deleted
W20251027 21:07:52.100878 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e841c54a94d649c19b639f5021f150cd consensus state has no leader
I20251027 21:07:52.101245 10029 tablet_bootstrap.cc:492] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:52.101369 10029 ts_tablet_manager.cc:1403] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20251027 21:07:52.102037 10029 raft_consensus.cc:359] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.102248 10029 raft_consensus.cc:385] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.102334 10029 raft_consensus.cc:740] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.102521 10029 consensus_queue.cc:260] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.105500 10029 ts_tablet_manager.cc:1434] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251027 21:07:52.106122 9682 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:07:52.107926 10029 tablet_bootstrap.cc:492] T 94a00ff844f0415fa419613d70a318e6 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:52.110255 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 212) took 1639 ms. Trace:
I20251027 21:07:52.110359 10040 rpcz_store.cc:276] 1027 21:07:50.470507 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:50.569505 (+ 98998us) service_pool.cc:225] Handling call
1027 21:07:52.022465 (+1452960us) ts_tablet_manager.cc:1151] Deleting tablet 65fbcaa5447c48818fabfccac22c4048
1027 21:07:52.022476 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:52.055731 (+ 33255us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:52.093250 (+ 37519us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:52.110174 (+ 16924us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1403394}
I20251027 21:07:52.114972 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 213) took 1644 ms. Trace:
I20251027 21:07:52.115028 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 65fbcaa5447c48818fabfccac22c4048 (table default.Table_c93601bc03f1438496c9610e8c23b5a6 [id=4369b0d36cc14f48a5f41f214aa4fe4c]) successfully deleted
I20251027 21:07:52.115085 10040 rpcz_store.cc:276] 1027 21:07:50.470618 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:50.615401 (+144783us) service_pool.cc:225] Handling call
1027 21:07:52.114790 (+1499389us) ts_tablet_manager.cc:1151] Deleting tablet 6f1129eecfb54448965222269d9c32b7
1027 21:07:52.114800 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:52.114914 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1449996}
I20251027 21:07:52.115491 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 216) took 1644 ms. Trace:
W20251027 21:07:52.115535 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 6f1129eecfb54448965222269d9c32b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f1129eecfb54448965222269d9c32b7 already in progress: creating tablet
I20251027 21:07:52.115588 10040 rpcz_store.cc:276] 1027 21:07:50.470900 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:50.620512 (+149612us) service_pool.cc:225] Handling call
1027 21:07:52.115326 (+1494814us) ts_tablet_manager.cc:1151] Deleting tablet 9c239216cb8846f0b1aaa360ddb83321
1027 21:07:52.115336 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:52.115437 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1449858}
W20251027 21:07:52.116008 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 9c239216cb8846f0b1aaa360ddb83321 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c239216cb8846f0b1aaa360ddb83321 already in progress: creating tablet
I20251027 21:07:52.116457 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 221) took 1371 ms. Trace:
I20251027 21:07:52.116552 10040 rpcz_store.cc:276] 1027 21:07:50.744138 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:50.745229 (+ 1091us) service_pool.cc:225] Handling call
1027 21:07:52.115804 (+1370575us) ts_tablet_manager.cc:1151] Deleting tablet 94e94cd50c91429d8da9804149501c86
1027 21:07:52.115813 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:52.115885 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1370353}
I20251027 21:07:52.116904 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 220) took 1372 ms. Trace:
I20251027 21:07:52.116991 10040 rpcz_store.cc:276] 1027 21:07:50.744024 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:50.748227 (+ 4203us) service_pool.cc:225] Handling call
1027 21:07:52.116768 (+1368541us) ts_tablet_manager.cc:1151] Deleting tablet e0968c2f25b64f6685f38a999ee9d60e
1027 21:07:52.116778 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:52.116851 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1368332}
I20251027 21:07:52.117332 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 222) took 1373 ms. Trace:
I20251027 21:07:52.117425 10040 rpcz_store.cc:276] 1027 21:07:50.744279 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:50.750331 (+ 6052us) service_pool.cc:225] Handling call
1027 21:07:52.117202 (+1366871us) ts_tablet_manager.cc:1151] Deleting tablet 9882b234e79940088be5f604c41576b7
1027 21:07:52.117210 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:52.117283 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1366635}
W20251027 21:07:52.117563 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet e0968c2f25b64f6685f38a999ee9d60e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0968c2f25b64f6685f38a999ee9d60e already in progress: creating tablet
I20251027 21:07:52.117786 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 223) took 1293 ms. Trace:
I20251027 21:07:52.117875 10040 rpcz_store.cc:276] 1027 21:07:50.824366 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:50.824522 (+ 156us) service_pool.cc:225] Handling call
1027 21:07:52.117619 (+1293097us) ts_tablet_manager.cc:1151] Deleting tablet c5f2e60caa824b5db2330b423221873c
1027 21:07:52.117627 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:52.117735 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1275352}
W20251027 21:07:52.118048 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 9882b234e79940088be5f604c41576b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9882b234e79940088be5f604c41576b7 already in progress: creating tablet
I20251027 21:07:52.118304 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 224) took 1134 ms. Trace:
I20251027 21:07:52.118394 10040 rpcz_store.cc:276] 1027 21:07:50.984178 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:50.984232 (+ 54us) service_pool.cc:225] Handling call
1027 21:07:52.118119 (+1133887us) ts_tablet_manager.cc:1151] Deleting tablet e07b8aa0a41241d79254c4dae8738e79
1027 21:07:52.118127 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:52.118246 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":858158}
W20251027 21:07:52.118600 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c5f2e60caa824b5db2330b423221873c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5f2e60caa824b5db2330b423221873c already in progress: creating tablet
I20251027 21:07:52.118695 10040 tablet_replica.cc:333] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:52.118880 10040 raft_consensus.cc:2243] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
W20251027 21:07:52.119024 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet e07b8aa0a41241d79254c4dae8738e79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e07b8aa0a41241d79254c4dae8738e79 already in progress: creating tablet
I20251027 21:07:52.119398 10040 raft_consensus.cc:2272] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.119632 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c4369a10bf846fb94c9b71a70f21600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.119946 10040 ts_tablet_manager.cc:1916] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.120040 10031 tablet_bootstrap.cc:654] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.121454 10043 ts_tablet_manager.cc:1916] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:52.122331 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet efc0019840244d14bb95ae581793e74b consensus state has no leader
I20251027 21:07:52.123000 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 5c34c8f1433b40d491ae05e7c6e60ba1
I20251027 21:07:52.123847 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c34c8f1433b40d491ae05e7c6e60ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.123965 10029 tablet_bootstrap.cc:654] T 94a00ff844f0415fa419613d70a318e6 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.124514 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c34c8f1433b40d491ae05e7c6e60ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:52.125101 9299 catalog_manager.cc:4952] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 5c34c8f1433b40d491ae05e7c6e60ba1 because the tablet was not found. No further retry: Not found: Tablet not found: 5c34c8f1433b40d491ae05e7c6e60ba1
I20251027 21:07:52.126344 10340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
W20251027 21:07:52.126535 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 87d5af4607f04f92abcc7292b4176cbd consensus state has no leader
I20251027 21:07:52.126641 10340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 21:07:52.127779 10299 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 21:07:52.120216 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f2775123ffc8464eaf6d5e7d4ee692b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:52.137281 10031 tablet_bootstrap.cc:492] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:52.137287 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c34c8f1433b40d491ae05e7c6e60ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.137516 10031 ts_tablet_manager.cc:1403] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.093s user 0.001s sys 0.003s
W20251027 21:07:52.137645 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2775123ffc8464eaf6d5e7d4ee692b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.138473 10031 raft_consensus.cc:359] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.138761 10031 raft_consensus.cc:385] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.138880 10031 raft_consensus.cc:740] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.139238 10031 consensus_queue.cc:260] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.143065 10029 tablet_bootstrap.cc:492] T 94a00ff844f0415fa419613d70a318e6 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:52.143728 10029 ts_tablet_manager.cc:1403] T 94a00ff844f0415fa419613d70a318e6 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.002s
I20251027 21:07:52.144439 10029 raft_consensus.cc:359] T 94a00ff844f0415fa419613d70a318e6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.144605 10029 raft_consensus.cc:385] T 94a00ff844f0415fa419613d70a318e6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.144675 10029 raft_consensus.cc:740] T 94a00ff844f0415fa419613d70a318e6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.144901 10029 consensus_queue.cc:260] T 94a00ff844f0415fa419613d70a318e6 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.146656 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet efc0019840244d14bb95ae581793e74b from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:52.147046 10031 ts_tablet_manager.cc:1434] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251027 21:07:52.149398 9820 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:07:52.149418 10029 ts_tablet_manager.cc:1434] T 94a00ff844f0415fa419613d70a318e6 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251027 21:07:52.150918 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 36e168cc44274dae9682b0c1329d4558 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.152662 10029 tablet_bootstrap.cc:492] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:52.154286 10029 tablet_bootstrap.cc:654] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.155444 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet 94a00ff844f0415fa419613d70a318e6 with delete_type TABLET_DATA_DELETED (replaced by tablet efc0019840244d14bb95ae581793e74b) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.157593 10031 tablet_bootstrap.cc:492] T cad3a3e910ac42399cf371ad82c89ff7 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:52.159852 10031 tablet_bootstrap.cc:654] T cad3a3e910ac42399cf371ad82c89ff7 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.160209 9523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 5c34c8f1433b40d491ae05e7c6e60ba1
I20251027 21:07:52.162017 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c34c8f1433b40d491ae05e7c6e60ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:52.162761 9299 catalog_manager.cc:4952] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 5c34c8f1433b40d491ae05e7c6e60ba1 because the tablet was not found. No further retry: Not found: Tablet not found: 5c34c8f1433b40d491ae05e7c6e60ba1
I20251027 21:07:52.162922 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c34c8f1433b40d491ae05e7c6e60ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.165482 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet efc0019840244d14bb95ae581793e74b
I20251027 21:07:52.166265 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet efc0019840244d14bb95ae581793e74b with delete_type TABLET_DATA_DELETED (replaced by tablet d2203ee62f7e404c95c5f86b5ae85feb) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.166918 9307 catalog_manager.cc:6719] ReplaceTablet: tablet efc0019840244d14bb95ae581793e74b deleted and replaced by tablet d2203ee62f7e404c95c5f86b5ae85feb
I20251027 21:07:52.166949 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet efc0019840244d14bb95ae581793e74b with delete_type TABLET_DATA_DELETED (replaced by tablet d2203ee62f7e404c95c5f86b5ae85feb) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:52.167613 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet efc0019840244d14bb95ae581793e74b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efc0019840244d14bb95ae581793e74b already in progress: creating tablet
I20251027 21:07:52.168104 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet efc0019840244d14bb95ae581793e74b with delete_type TABLET_DATA_DELETED (replaced by tablet d2203ee62f7e404c95c5f86b5ae85feb) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.168682 10043 ts_tablet_manager.cc:1929] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:07:52.168874 10043 log.cc:1199] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/65fbcaa5447c48818fabfccac22c4048
I20251027 21:07:52.169409 10043 ts_tablet_manager.cc:1950] T 65fbcaa5447c48818fabfccac22c4048 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:52.170182 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c34c8f1433b40d491ae05e7c6e60ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.170123 9889 tablet_service.cc:1505] Processing CreateTablet for tablet cef2bfee222f45e6b6f20c6d63ff44d4 (DEFAULT_TABLE table=default.Table_c3f0afa8e76249b69b9fbe4b3f67d020 [id=43518e3f60164df7a812afb41f6c4936]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.170274 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 221) took 1089 ms. Trace:
I20251027 21:07:52.170568 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef2bfee222f45e6b6f20c6d63ff44d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.170475 10043 rpcz_store.cc:276] 1027 21:07:51.080569 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:51.081798 (+ 1229us) service_pool.cc:225] Handling call
1027 21:07:52.080843 (+999045us) ts_tablet_manager.cc:1151] Deleting tablet 65fbcaa5447c48818fabfccac22c4048
1027 21:07:52.080850 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:52.125296 (+ 44446us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:52.134363 (+ 9067us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:52.170239 (+ 35876us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":998842}
I20251027 21:07:52.170877 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 65fbcaa5447c48818fabfccac22c4048 (table default.Table_c93601bc03f1438496c9610e8c23b5a6 [id=4369b0d36cc14f48a5f41f214aa4fe4c]) successfully deleted
W20251027 21:07:52.171412 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet d8e008c868d04281beba0e5b204eb69a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8e008c868d04281beba0e5b204eb69a already in progress: creating tablet
W20251027 21:07:52.171933 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 343eaeca24dc4816ae032c7e674e9412 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 343eaeca24dc4816ae032c7e674e9412 already in progress: creating tablet
I20251027 21:07:52.172184 10043 tablet_replica.cc:333] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:52.172369 10043 raft_consensus.cc:2243] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.172550 10043 raft_consensus.cc:2272] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.173386 10043 ts_tablet_manager.cc:1916] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.173763 9750 tablet_service.cc:1505] Processing CreateTablet for tablet cef2bfee222f45e6b6f20c6d63ff44d4 (DEFAULT_TABLE table=default.Table_c3f0afa8e76249b69b9fbe4b3f67d020 [id=43518e3f60164df7a812afb41f6c4936]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.174096 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef2bfee222f45e6b6f20c6d63ff44d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.178586 10040 ts_tablet_manager.cc:1929] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251027 21:07:52.178716 10040 log.cc:1199] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/21b785277d604543bfa7816ed96d6181
I20251027 21:07:52.179158 10040 ts_tablet_manager.cc:1950] T 21b785277d604543bfa7816ed96d6181 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:52.179929 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 245) took 1065 ms. Trace:
I20251027 21:07:52.180047 10040 rpcz_store.cc:276] 1027 21:07:51.114756 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:51.259980 (+145224us) service_pool.cc:225] Handling call
1027 21:07:52.118616 (+858636us) ts_tablet_manager.cc:1151] Deleting tablet 21b785277d604543bfa7816ed96d6181
1027 21:07:52.118625 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:52.160864 (+ 42239us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:52.178467 (+ 17603us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:52.179888 (+ 1421us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":858474}
I20251027 21:07:52.180413 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 226) took 1182 ms. Trace:
I20251027 21:07:52.180482 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 21b785277d604543bfa7816ed96d6181 (table default.Table_f1908c85b5144e90a366ca29ba5d1b9b [id=e3de2e5ea6484be7af68eb8601778b85]) successfully deleted
I20251027 21:07:52.183928 10031 tablet_bootstrap.cc:492] T cad3a3e910ac42399cf371ad82c89ff7 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:52.184132 10031 ts_tablet_manager.cc:1403] T cad3a3e910ac42399cf371ad82c89ff7 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.005s
I20251027 21:07:52.185839 9604 tablet_service.cc:1505] Processing CreateTablet for tablet cef2bfee222f45e6b6f20c6d63ff44d4 (DEFAULT_TABLE table=default.Table_c3f0afa8e76249b69b9fbe4b3f67d020 [id=43518e3f60164df7a812afb41f6c4936]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.186111 9889 tablet_service.cc:1505] Processing CreateTablet for tablet f775a7e42bdc4910bdb604b26a28a7f4 (DEFAULT_TABLE table=default.Table_7b970942803b42f29feb36d6d845167a [id=20a729dd21c440a882083323ddb2561e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.186235 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef2bfee222f45e6b6f20c6d63ff44d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.186473 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f775a7e42bdc4910bdb604b26a28a7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.180522 10040 rpcz_store.cc:276] 1027 21:07:50.997558 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:50.997708 (+ 150us) service_pool.cc:225] Handling call
1027 21:07:52.180312 (+1182604us) ts_tablet_manager.cc:1151] Deleting tablet 6e5d52b7ec114be5ae68e57280c2eea8
1027 21:07:52.180320 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:52.180384 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":920003}
I20251027 21:07:52.188395 10040 tablet_replica.cc:333] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
W20251027 21:07:52.188519 9297 catalog_manager.cc:4952] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 6e5d52b7ec114be5ae68e57280c2eea8 because the tablet was not found. No further retry: Not found: Tablet not found: 6e5d52b7ec114be5ae68e57280c2eea8
I20251027 21:07:52.188596 10040 raft_consensus.cc:2243] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.187740 9750 tablet_service.cc:1505] Processing CreateTablet for tablet f775a7e42bdc4910bdb604b26a28a7f4 (DEFAULT_TABLE table=default.Table_7b970942803b42f29feb36d6d845167a [id=20a729dd21c440a882083323ddb2561e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.188747 10040 raft_consensus.cc:2272] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.188910 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f775a7e42bdc4910bdb604b26a28a7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.189247 10040 ts_tablet_manager.cc:1916] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.190732 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet 21a8fc4a69f3486e92b1a68d10faed38 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:52.193310 10031 raft_consensus.cc:359] T cad3a3e910ac42399cf371ad82c89ff7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.193604 10031 raft_consensus.cc:385] T cad3a3e910ac42399cf371ad82c89ff7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.193754 10031 raft_consensus.cc:740] T cad3a3e910ac42399cf371ad82c89ff7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.194057 10031 consensus_queue.cc:260] T cad3a3e910ac42399cf371ad82c89ff7 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.196215 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5e48ead40e6e4f6f9ae841b8fd6d3629" } new_table_name: "default.Table_cf11a8c344c544c5ba54672fee38e4aa" modify_external_catalogs: true
I20251027 21:07:52.199317 10031 ts_tablet_manager.cc:1434] T cad3a3e910ac42399cf371ad82c89ff7 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251027 21:07:52.202236 10031 tablet_bootstrap.cc:492] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:52.205224 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet cad3a3e910ac42399cf371ad82c89ff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.207013 10312 raft_consensus.cc:493] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:52.207152 10031 tablet_bootstrap.cc:654] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.207206 10312 raft_consensus.cc:515] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.207782 10312 leader_election.cc:290] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:07:52.208926 9568 leader_election.cc:343] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: e3c9f64ee0544e59add4f73bca59df98
I20251027 21:07:52.209028 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3c9f64ee0544e59add4f73bca59df98" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:52.209481 9566 leader_election.cc:343] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: Tablet not RUNNING: STOPPED
I20251027 21:07:52.209589 9566 leader_election.cc:304] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:52.210052 10312 raft_consensus.cc:2749] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:52.210722 10043 ts_tablet_manager.cc:1929] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251027 21:07:52.210749 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b4d6f7e145e14280ac10fa486bd81385"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:52.211143 10043 log.cc:1199] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/21b785277d604543bfa7816ed96d6181
W20251027 21:07:52.211251 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4d6f7e145e14280ac10fa486bd81385 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.211707 10043 ts_tablet_manager.cc:1950] T 21b785277d604543bfa7816ed96d6181 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:52.212774 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 21b785277d604543bfa7816ed96d6181 (table default.Table_f1908c85b5144e90a366ca29ba5d1b9b [id=e3de2e5ea6484be7af68eb8601778b85]) successfully deleted
I20251027 21:07:52.213728 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_11fbadd9eed14cdba7664715e389f7e5" } modify_external_catalogs: true
I20251027 21:07:52.213886 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_11fbadd9eed14cdba7664715e389f7e5" } modify_external_catalogs: true
I20251027 21:07:52.217060 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 237) took 1037 ms. Trace:
I20251027 21:07:52.217247 10043 rpcz_store.cc:276] 1027 21:07:51.289242 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:51.296882 (+ 7640us) service_pool.cc:225] Handling call
1027 21:07:52.215737 (+918855us) ts_tablet_manager.cc:1151] Deleting tablet 21a8fc4a69f3486e92b1a68d10faed38
1027 21:07:52.216735 (+ 998us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:52.216857 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":918679}
W20251027 21:07:52.217948 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 21a8fc4a69f3486e92b1a68d10faed38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21a8fc4a69f3486e92b1a68d10faed38 already in progress: creating tablet
W20251027 21:07:52.218617 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d56d323ef0e74344a479940b93fea11a consensus state has no leader
W20251027 21:07:52.218844 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet c400101d2a39490e83722500839198cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c400101d2a39490e83722500839198cd already in progress: creating tablet
I20251027 21:07:52.219237 10040 ts_tablet_manager.cc:1929] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.219385 10040 log.cc:1199] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/e3c9f64ee0544e59add4f73bca59df98
I20251027 21:07:52.219743 10040 ts_tablet_manager.cc:1950] T e3c9f64ee0544e59add4f73bca59df98 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
W20251027 21:07:52.219952 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa already in progress: creating tablet
I20251027 21:07:52.221458 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 76e4fb2bea7f4a4696fe9f6bb892bccc (DEFAULT_TABLE table=default.Table_24aac5cecb8745d7be92287080a1b6e0 [id=080bf50609c045bd9b05f4e9144feac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.221835 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76e4fb2bea7f4a4696fe9f6bb892bccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.221872 9604 tablet_service.cc:1505] Processing CreateTablet for tablet f775a7e42bdc4910bdb604b26a28a7f4 (DEFAULT_TABLE table=default.Table_7b970942803b42f29feb36d6d845167a [id=20a729dd21c440a882083323ddb2561e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.222637 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f775a7e42bdc4910bdb604b26a28a7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.223099 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 76e4fb2bea7f4a4696fe9f6bb892bccc (DEFAULT_TABLE table=default.Table_24aac5cecb8745d7be92287080a1b6e0 [id=080bf50609c045bd9b05f4e9144feac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.223445 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76e4fb2bea7f4a4696fe9f6bb892bccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.225193 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 01e9db9eaf8e41c6a2ee13b9ed127421 (DEFAULT_TABLE table=default.Table_11fbadd9eed14cdba7664715e389f7e5 [id=83ec44dba5c444aba48846811a5f5fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.225366 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 01e9db9eaf8e41c6a2ee13b9ed127421 (DEFAULT_TABLE table=default.Table_11fbadd9eed14cdba7664715e389f7e5 [id=83ec44dba5c444aba48846811a5f5fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.225521 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01e9db9eaf8e41c6a2ee13b9ed127421. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.225714 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01e9db9eaf8e41c6a2ee13b9ed127421. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.226454 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 76e4fb2bea7f4a4696fe9f6bb892bccc (DEFAULT_TABLE table=default.Table_24aac5cecb8745d7be92287080a1b6e0 [id=080bf50609c045bd9b05f4e9144feac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.226794 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76e4fb2bea7f4a4696fe9f6bb892bccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.230355 10312 raft_consensus.cc:493] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:52.230543 10312 raft_consensus.cc:515] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.231189 10312 leader_election.cc:290] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:52.231758 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cad3a3e910ac42399cf371ad82c89ff7" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:52.231909 10312 raft_consensus.cc:493] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:52.231963 9774 raft_consensus.cc:2468] T cad3a3e910ac42399cf371ad82c89ff7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:52.232026 10312 raft_consensus.cc:515] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.232528 10312 leader_election.cc:290] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:52.232718 9566 leader_election.cc:304] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:07:52.233356 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1bdb2be5032415187c2826c14e75f07" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:52.233513 9774 raft_consensus.cc:2468] T b1bdb2be5032415187c2826c14e75f07 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:52.233996 9566 leader_election.cc:304] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:07:52.234457 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7797741574f1487ba7211ddb3bb032dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.234589 9568 leader_election.cc:343] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: cad3a3e910ac42399cf371ad82c89ff7
I20251027 21:07:52.235113 10312 raft_consensus.cc:2804] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251027 21:07:52.235179 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7797741574f1487ba7211ddb3bb032dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.235242 10312 raft_consensus.cc:493] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:52.235337 10312 raft_consensus.cc:3060] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:52.236692 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_faf6fa3adb644cdf984ac3704b878293"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.237403 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_faf6fa3adb644cdf984ac3704b878293 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.238587 10312 raft_consensus.cc:515] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.238970 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_605ef0bcf3ad4d919afc3a73ac0ab926" } new_table_name: "default.Table_e6955caf15904f3bae744add8221897f" modify_external_catalogs: true
I20251027 21:07:52.239176 10312 leader_election.cc:290] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:52.239392 10312 raft_consensus.cc:2804] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:52.239483 10312 raft_consensus.cc:493] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:52.239598 10312 raft_consensus.cc:3060] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:52.241432 10043 tablet_replica.cc:333] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:52.242342 10043 raft_consensus.cc:2243] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
W20251027 21:07:52.242529 9568 leader_election.cc:343] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: b1bdb2be5032415187c2826c14e75f07
W20251027 21:07:52.243252 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet dd7029986a5246ee823e5fdb50683118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd7029986a5246ee823e5fdb50683118 already in progress: creating tablet
W20251027 21:07:52.243526 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet adaa4f66bbab491f8030d439d85cc7d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet adaa4f66bbab491f8030d439d85cc7d1 already in progress: creating tablet
W20251027 21:07:52.243870 9568 leader_election.cc:343] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: cad3a3e910ac42399cf371ad82c89ff7
I20251027 21:07:52.244387 10032 tablet_bootstrap.cc:492] T 601c74459d3a4a41b812fdcdf8b0a62c P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:52.244885 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 01e9db9eaf8e41c6a2ee13b9ed127421 (DEFAULT_TABLE table=default.Table_11fbadd9eed14cdba7664715e389f7e5 [id=83ec44dba5c444aba48846811a5f5fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.245240 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01e9db9eaf8e41c6a2ee13b9ed127421. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.245498 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 01e9db9eaf8e41c6a2ee13b9ed127421
I20251027 21:07:52.245675 10031 tablet_bootstrap.cc:492] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:52.245816 10031 ts_tablet_manager.cc:1403] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.000s
I20251027 21:07:52.246591 10031 raft_consensus.cc:359] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:52.246829 10031 raft_consensus.cc:385] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.246935 10031 raft_consensus.cc:740] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.247201 10031 consensus_queue.cc:260] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:52.248422 10141 raft_consensus.cc:493] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:52.248574 10141 raft_consensus.cc:515] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.248700 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet 01e9db9eaf8e41c6a2ee13b9ed127421 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.243551 10043 raft_consensus.cc:2272] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.249168 10141 leader_election.cc:290] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:52.249400 10043 ts_tablet_manager.cc:1916] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.250288 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet 01e9db9eaf8e41c6a2ee13b9ed127421 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:52.251521 9299 catalog_manager.cc:4952] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 01e9db9eaf8e41c6a2ee13b9ed127421 because the tablet was not found. No further retry: Not found: Tablet not found: 01e9db9eaf8e41c6a2ee13b9ed127421
W20251027 21:07:52.251909 9710 leader_election.cc:343] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 44783760defd433a972c9cd83933249c
I20251027 21:07:52.244588 10032 ts_tablet_manager.cc:1403] T 601c74459d3a4a41b812fdcdf8b0a62c P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.253s user 0.002s sys 0.078s
W20251027 21:07:52.252048 9711 leader_election.cc:343] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 44783760defd433a972c9cd83933249c
I20251027 21:07:52.252216 9711 leader_election.cc:304] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:52.252741 10141 raft_consensus.cc:2749] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:52.253217 10032 raft_consensus.cc:359] T 601c74459d3a4a41b812fdcdf8b0a62c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.253400 10032 raft_consensus.cc:385] T 601c74459d3a4a41b812fdcdf8b0a62c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.253520 10032 raft_consensus.cc:740] T 601c74459d3a4a41b812fdcdf8b0a62c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.253815 10032 consensus_queue.cc:260] T 601c74459d3a4a41b812fdcdf8b0a62c P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.254539 10032 ts_tablet_manager.cc:1434] T 601c74459d3a4a41b812fdcdf8b0a62c P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:52.254935 10032 tablet_bootstrap.cc:492] T efc0019840244d14bb95ae581793e74b P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:52.256724 10032 tablet_bootstrap.cc:654] T efc0019840244d14bb95ae581793e74b P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.257191 10312 raft_consensus.cc:515] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
W20251027 21:07:52.258747 9568 leader_election.cc:343] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: b1bdb2be5032415187c2826c14e75f07
I20251027 21:07:52.259083 10031 ts_tablet_manager.cc:1434] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20251027 21:07:52.259231 10312 leader_election.cc:290] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:52.259383 10312 raft_consensus.cc:493] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:52.259512 10312 raft_consensus.cc:515] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.260313 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5d8837ff3c44a66a1f933d80540ba0c" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:52.260504 9924 raft_consensus.cc:2468] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:52.260869 9568 leader_election.cc:304] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:07:52.260942 10031 tablet_bootstrap.cc:492] T 9c4369a10bf846fb94c9b71a70f21600 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:52.261479 10312 leader_election.cc:290] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:52.261649 10312 raft_consensus.cc:2804] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:52.261735 10312 raft_consensus.cc:493] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:52.261837 10312 raft_consensus.cc:3060] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:52.263329 10031 tablet_bootstrap.cc:654] T 9c4369a10bf846fb94c9b71a70f21600 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.266614 10032 tablet_bootstrap.cc:492] T efc0019840244d14bb95ae581793e74b P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:52.267220 10032 ts_tablet_manager.cc:1403] T efc0019840244d14bb95ae581793e74b P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.003s
I20251027 21:07:52.267791 10312 raft_consensus.cc:515] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.268815 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5d8837ff3c44a66a1f933d80540ba0c" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:07:52.268986 9924 raft_consensus.cc:3060] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:52.270980 10043 ts_tablet_manager.cc:1929] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 21:07:52.271102 10043 log.cc:1199] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/0ac7111d13b6460a9f4c77e94e09d417
I20251027 21:07:52.272046 9924 raft_consensus.cc:2468] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:52.272047 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e4d0e9336a834d6d8ab29ac0b1de7ace" } modify_external_catalogs: true
I20251027 21:07:52.272218 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e4d0e9336a834d6d8ab29ac0b1de7ace" } modify_external_catalogs: true
I20251027 21:07:52.272490 9568 leader_election.cc:304] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:07:52.268657 10032 raft_consensus.cc:359] T efc0019840244d14bb95ae581793e74b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.273046 10032 raft_consensus.cc:385] T efc0019840244d14bb95ae581793e74b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.273432 10043 ts_tablet_manager.cc:1950] T 0ac7111d13b6460a9f4c77e94e09d417 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:52.273301 10032 raft_consensus.cc:740] T efc0019840244d14bb95ae581793e74b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.274633 10032 consensus_queue.cc:260] T efc0019840244d14bb95ae581793e74b P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.275238 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 0ac7111d13b6460a9f4c77e94e09d417 (table default.Table_e65d5d66ce8747a9a1a2abfa04cd2855 [id=e2b68fb34c3b48e8952ef6d4201fd93f]) successfully deleted
I20251027 21:07:52.275852 10032 ts_tablet_manager.cc:1434] T efc0019840244d14bb95ae581793e74b P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251027 21:07:52.276124 10031 tablet_bootstrap.cc:492] T 9c4369a10bf846fb94c9b71a70f21600 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:52.276273 10031 ts_tablet_manager.cc:1403] T 9c4369a10bf846fb94c9b71a70f21600 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20251027 21:07:52.276685 10032 tablet_bootstrap.cc:492] T 2b247829a11349b6835a2f18906e20a9 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:52.277128 10031 raft_consensus.cc:359] T 9c4369a10bf846fb94c9b71a70f21600 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.277333 10031 raft_consensus.cc:385] T 9c4369a10bf846fb94c9b71a70f21600 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.277436 10031 raft_consensus.cc:740] T 9c4369a10bf846fb94c9b71a70f21600 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.277686 10031 consensus_queue.cc:260] T 9c4369a10bf846fb94c9b71a70f21600 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
W20251027 21:07:52.277784 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 520f7142f8af4599ac4880b0692e3151 has no consensus state
I20251027 21:07:52.278434 10031 ts_tablet_manager.cc:1434] T 9c4369a10bf846fb94c9b71a70f21600 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:52.278755 10031 tablet_bootstrap.cc:492] T e5d8837ff3c44a66a1f933d80540ba0c P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:52.278842 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 227) took 1276 ms. Trace:
I20251027 21:07:52.278978 10040 rpcz_store.cc:276] 1027 21:07:51.002812 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:51.002882 (+ 70us) service_pool.cc:225] Handling call
1027 21:07:52.188331 (+1185449us) ts_tablet_manager.cc:1151] Deleting tablet e3c9f64ee0544e59add4f73bca59df98
1027 21:07:52.188340 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:52.202939 (+ 14599us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:52.219127 (+ 16188us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:52.278780 (+ 59653us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":927626}
I20251027 21:07:52.279398 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 241) took 1170 ms. Trace:
I20251027 21:07:52.280308 10032 tablet_bootstrap.cc:654] T 2b247829a11349b6835a2f18906e20a9 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.279506 10040 rpcz_store.cc:276] 1027 21:07:51.108804 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:51.109205 (+ 401us) service_pool.cc:225] Handling call
1027 21:07:52.279254 (+1170049us) ts_tablet_manager.cc:1151] Deleting tablet 343eaeca24dc4816ae032c7e674e9412
1027 21:07:52.279262 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:52.279341 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1015339}
I20251027 21:07:52.283077 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet e3c9f64ee0544e59add4f73bca59df98 (table default.Table_79febe3afdb94e8781e2633f126402dc [id=0e1ad615b4494a6dbb8c1390c1b8416f]) successfully deleted
W20251027 21:07:52.283390 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 343eaeca24dc4816ae032c7e674e9412 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 343eaeca24dc4816ae032c7e674e9412 already in progress: creating tablet
I20251027 21:07:52.283463 10312 leader_election.cc:290] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:07:52.284266 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet a9bf3a093be8498c9dd5511b53436b16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9bf3a093be8498c9dd5511b53436b16 already in progress: creating tablet
W20251027 21:07:52.285178 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 77797eea3e9c4f8099dc9d58f116200b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77797eea3e9c4f8099dc9d58f116200b already in progress: creating tablet
I20251027 21:07:52.285440 10312 raft_consensus.cc:2804] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:52.285637 10312 raft_consensus.cc:697] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:07:52.285876 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet d8ef795c88da49b58082ba1264d83378 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:47 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:52.286314 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 2b247829a11349b6835a2f18906e20a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b247829a11349b6835a2f18906e20a9 already in progress: creating tablet
I20251027 21:07:52.286391 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 242) took 1177 ms. Trace:
I20251027 21:07:52.286489 10040 rpcz_store.cc:276] 1027 21:07:51.108939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:51.109366 (+ 427us) service_pool.cc:225] Handling call
1027 21:07:52.281281 (+1171915us) ts_tablet_manager.cc:1151] Deleting tablet d8e008c868d04281beba0e5b204eb69a
1027 21:07:52.286080 (+ 4799us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:52.286337 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1017305}
I20251027 21:07:52.286949 10312 consensus_queue.cc:237] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
W20251027 21:07:52.287650 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 648f506ec87c4fb8a3f6d4591873dbb8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 648f506ec87c4fb8a3f6d4591873dbb8 already in progress: creating tablet
W20251027 21:07:52.288343 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet e9edb62a8e4b4dfea6a31a2202545a6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9edb62a8e4b4dfea6a31a2202545a6e already in progress: creating tablet
W20251027 21:07:52.289312 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet dc589852710342f7bd017193e4075303 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc589852710342f7bd017193e4075303 already in progress: creating tablet
I20251027 21:07:52.290306 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 243) took 1181 ms. Trace:
I20251027 21:07:52.290396 10040 rpcz_store.cc:276] 1027 21:07:51.109041 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:51.109497 (+ 456us) service_pool.cc:225] Handling call
1027 21:07:52.286738 (+1177241us) ts_tablet_manager.cc:1151] Deleting tablet 21a8fc4a69f3486e92b1a68d10faed38
1027 21:07:52.290164 (+ 3426us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:52.290254 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1022721}
W20251027 21:07:52.290597 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet a26ded6012174348b7295fc7c0e05e6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a26ded6012174348b7295fc7c0e05e6e already in progress: creating tablet
I20251027 21:07:52.291145 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 244) took 1181 ms. Trace:
I20251027 21:07:52.291247 10040 rpcz_store.cc:276] 1027 21:07:51.109157 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:51.109605 (+ 448us) service_pool.cc:225] Handling call
1027 21:07:52.290612 (+1181007us) ts_tablet_manager.cc:1151] Deleting tablet c400101d2a39490e83722500839198cd
1027 21:07:52.291001 (+ 389us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:52.291079 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1026546}
I20251027 21:07:52.291481 10043 tablet_replica.cc:333] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:52.291772 10043 raft_consensus.cc:2243] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.291914 10031 tablet_bootstrap.cc:654] T e5d8837ff3c44a66a1f933d80540ba0c P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.292001 10040 tablet_replica.cc:333] T 0ac7111d13b6460a9f4c77e94e09d417 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:52.292196 10040 raft_consensus.cc:2243] T 0ac7111d13b6460a9f4c77e94e09d417 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.292393 10040 raft_consensus.cc:2272] T 0ac7111d13b6460a9f4c77e94e09d417 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.293498 10040 ts_tablet_manager.cc:1916] T 0ac7111d13b6460a9f4c77e94e09d417 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.292418 10043 raft_consensus.cc:2272] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.294119 10043 ts_tablet_manager.cc:1916] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:52.294548 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet d8e008c868d04281beba0e5b204eb69a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8e008c868d04281beba0e5b204eb69a already in progress: creating tablet
W20251027 21:07:52.294937 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 21a8fc4a69f3486e92b1a68d10faed38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21a8fc4a69f3486e92b1a68d10faed38 already in progress: creating tablet
W20251027 21:07:52.295332 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c400101d2a39490e83722500839198cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c400101d2a39490e83722500839198cd already in progress: creating tablet
I20251027 21:07:52.295913 10032 tablet_bootstrap.cc:492] T 2b247829a11349b6835a2f18906e20a9 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:52.296463 10032 ts_tablet_manager.cc:1403] T 2b247829a11349b6835a2f18906e20a9 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20251027 21:07:52.297349 10032 raft_consensus.cc:359] T 2b247829a11349b6835a2f18906e20a9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.297726 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 601c74459d3a4a41b812fdcdf8b0a62c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.298663 10041 tablet_replica.cc:333] T 601c74459d3a4a41b812fdcdf8b0a62c P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:52.298851 10041 raft_consensus.cc:2243] T 601c74459d3a4a41b812fdcdf8b0a62c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.298992 10041 raft_consensus.cc:2272] T 601c74459d3a4a41b812fdcdf8b0a62c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.299700 10041 ts_tablet_manager.cc:1916] T 601c74459d3a4a41b812fdcdf8b0a62c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.302057 9309 catalog_manager.cc:5649] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } }
I20251027 21:07:52.303894 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c04ff0f25ba04d25958346fc631b2802" } modify_external_catalogs: true
I20251027 21:07:52.304083 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c04ff0f25ba04d25958346fc631b2802" } modify_external_catalogs: true
I20251027 21:07:52.306798 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b730ec55ffb24974928bd0b0d9416460"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.307304 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b730ec55ffb24974928bd0b0d9416460 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.307359 10031 tablet_bootstrap.cc:492] T e5d8837ff3c44a66a1f933d80540ba0c P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:52.307493 10031 ts_tablet_manager.cc:1403] T e5d8837ff3c44a66a1f933d80540ba0c P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20251027 21:07:52.308175 10031 raft_consensus.cc:359] T e5d8837ff3c44a66a1f933d80540ba0c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.308388 10031 raft_consensus.cc:385] T e5d8837ff3c44a66a1f933d80540ba0c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.308527 10031 raft_consensus.cc:740] T e5d8837ff3c44a66a1f933d80540ba0c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.308840 10031 consensus_queue.cc:260] T e5d8837ff3c44a66a1f933d80540ba0c P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.309037 10041 ts_tablet_manager.cc:1929] T 601c74459d3a4a41b812fdcdf8b0a62c P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.309204 10041 log.cc:1199] T 601c74459d3a4a41b812fdcdf8b0a62c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/601c74459d3a4a41b812fdcdf8b0a62c
I20251027 21:07:52.309551 10031 ts_tablet_manager.cc:1434] T e5d8837ff3c44a66a1f933d80540ba0c P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:52.309651 10041 ts_tablet_manager.cc:1950] T 601c74459d3a4a41b812fdcdf8b0a62c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:52.309844 10031 tablet_bootstrap.cc:492] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:52.310766 10032 raft_consensus.cc:385] T 2b247829a11349b6835a2f18906e20a9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.310778 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 601c74459d3a4a41b812fdcdf8b0a62c (table default.Table_0ac2c74fb86c48459bad366bd8efccd5 [id=7c97543d81eb4a1fa0c81678857179ba]) successfully deleted
I20251027 21:07:52.311024 10032 raft_consensus.cc:740] T 2b247829a11349b6835a2f18906e20a9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.311620 10031 tablet_bootstrap.cc:654] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.311512 10032 consensus_queue.cc:260] T 2b247829a11349b6835a2f18906e20a9 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.312399 10032 ts_tablet_manager.cc:1434] T 2b247829a11349b6835a2f18906e20a9 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20251027 21:07:52.313196 10032 tablet_bootstrap.cc:492] T 49b3d23798b740008c98cd0754ae33f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:52.315124 10032 tablet_bootstrap.cc:654] T 49b3d23798b740008c98cd0754ae33f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.315326 10043 ts_tablet_manager.cc:1929] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.315551 10043 log.cc:1199] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/35e5b7bbab7f4ee9bbf23d913c0aee76
I20251027 21:07:52.316013 10043 ts_tablet_manager.cc:1950] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:52.317461 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 (table default.Table_e68b56d3ddd94f308243ec6a03ea9e6e [id=0a94e88b69094ffe8be3367470be86da]) successfully deleted
I20251027 21:07:52.317992 10043 tablet_replica.cc:333] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:52.318293 10043 raft_consensus.cc:2243] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.318583 10043 raft_consensus.cc:2272] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.319259 10043 ts_tablet_manager.cc:1916] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.321125 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e9fd9bee6a2a40d7ad92d2d31b03afce" } modify_external_catalogs: true
I20251027 21:07:52.321367 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e9fd9bee6a2a40d7ad92d2d31b03afce" } modify_external_catalogs: true
I20251027 21:07:52.322646 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c4369a10bf846fb94c9b71a70f21600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.324824 10031 tablet_bootstrap.cc:492] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:52.325006 10031 ts_tablet_manager.cc:1403] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.004s
I20251027 21:07:52.325758 10031 raft_consensus.cc:359] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.326494 10031 raft_consensus.cc:385] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.326695 10031 raft_consensus.cc:740] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.327086 10031 consensus_queue.cc:260] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.328485 10032 tablet_bootstrap.cc:492] T 49b3d23798b740008c98cd0754ae33f4 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:52.328542 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 01e9db9eaf8e41c6a2ee13b9ed127421 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.328632 10032 ts_tablet_manager.cc:1403] T 49b3d23798b740008c98cd0754ae33f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20251027 21:07:52.329587 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 33e41145c79b4465b8d564741174b8e0
I20251027 21:07:52.330511 10032 raft_consensus.cc:359] T 49b3d23798b740008c98cd0754ae33f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.330705 10032 raft_consensus.cc:385] T 49b3d23798b740008c98cd0754ae33f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.330717 10031 ts_tablet_manager.cc:1434] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251027 21:07:52.330729 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet 33e41145c79b4465b8d564741174b8e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.330823 10032 raft_consensus.cc:740] T 49b3d23798b740008c98cd0754ae33f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.331025 10032 consensus_queue.cc:260] T 49b3d23798b740008c98cd0754ae33f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.331285 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 33e41145c79b4465b8d564741174b8e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.332449 10041 tablet_replica.cc:333] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:52.332621 10041 raft_consensus.cc:2243] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.332772 10041 raft_consensus.cc:2272] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.333238 10041 ts_tablet_manager.cc:1916] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.333384 10031 tablet_bootstrap.cc:492] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:52.334972 10031 tablet_bootstrap.cc:654] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.335872 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 33e41145c79b4465b8d564741174b8e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.338752 10043 ts_tablet_manager.cc:1929] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.340502 10043 log.cc:1199] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/e3c9f64ee0544e59add4f73bca59df98
I20251027 21:07:52.341035 10043 ts_tablet_manager.cc:1950] T e3c9f64ee0544e59add4f73bca59df98 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:52.341837 10041 ts_tablet_manager.cc:1929] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.342007 10041 log.cc:1199] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/33e41145c79b4465b8d564741174b8e0
I20251027 21:07:52.342440 10041 ts_tablet_manager.cc:1950] T 33e41145c79b4465b8d564741174b8e0 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:52.342481 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet e3c9f64ee0544e59add4f73bca59df98 (table default.Table_79febe3afdb94e8781e2633f126402dc [id=0e1ad615b4494a6dbb8c1390c1b8416f]) successfully deleted
I20251027 21:07:52.342406 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_340ce2d28b454548af6c487cb35008ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.342954 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_340ce2d28b454548af6c487cb35008ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.343664 10043 tablet_replica.cc:333] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:52.343937 10032 ts_tablet_manager.cc:1434] T 49b3d23798b740008c98cd0754ae33f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251027 21:07:52.344002 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 33e41145c79b4465b8d564741174b8e0 (table default.Table_e4d0e9336a834d6d8ab29ac0b1de7ace [id=af5b5b94fcef4677affe7e7afaa7f023]) successfully deleted
I20251027 21:07:52.344126 10043 raft_consensus.cc:2243] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.342953 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 3103c8e5a7c744be90be097179904ba3
I20251027 21:07:52.344283 10043 raft_consensus.cc:2272] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.345059 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet 3103c8e5a7c744be90be097179904ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.345093 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 3103c8e5a7c744be90be097179904ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.346055 10043 ts_tablet_manager.cc:1916] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:52.346158 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 3103c8e5a7c744be90be097179904ba3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3103c8e5a7c744be90be097179904ba3 already in progress: creating tablet
I20251027 21:07:52.346217 10032 tablet_bootstrap.cc:492] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:52.347159 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b78874d9b16b4c76bdff139266ed9f32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.347697 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b78874d9b16b4c76bdff139266ed9f32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.349395 10040 ts_tablet_manager.cc:1929] T 0ac7111d13b6460a9f4c77e94e09d417 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 21:07:52.349900 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_35709e995c574069b4df49b470f87b26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:52.350478 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_13a109a52d1a42aca531982896796ff7" } new_table_name: "default.Table_fe308a2645c14e6c8efc7ce330b5add7" modify_external_catalogs: true
I20251027 21:07:52.350659 10032 tablet_bootstrap.cc:654] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.351181 10040 log.cc:1199] T 0ac7111d13b6460a9f4c77e94e09d417 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/0ac7111d13b6460a9f4c77e94e09d417
I20251027 21:07:52.351626 10040 ts_tablet_manager.cc:1950] T 0ac7111d13b6460a9f4c77e94e09d417 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:52.351809 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_932e4079f74d40509aa6c2a1b8139d16" } new_table_name: "default.Table_bc5ec745ebe54017b6bc9635581c813d" modify_external_catalogs: true
I20251027 21:07:52.352602 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 0ac7111d13b6460a9f4c77e94e09d417 (table default.Table_e65d5d66ce8747a9a1a2abfa04cd2855 [id=e2b68fb34c3b48e8952ef6d4201fd93f]) successfully deleted
I20251027 21:07:52.352789 10031 tablet_bootstrap.cc:492] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
W20251027 21:07:52.352986 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35709e995c574069b4df49b470f87b26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.353014 10031 ts_tablet_manager.cc:1403] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.005s
I20251027 21:07:52.353729 10031 raft_consensus.cc:359] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.353921 10031 raft_consensus.cc:385] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.354035 10031 raft_consensus.cc:740] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.354101 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_44ef0d8b29e34b8fb63bf6a7532a47fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:52.354377 10031 consensus_queue.cc:260] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:52.354730 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44ef0d8b29e34b8fb63bf6a7532a47fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:52.355726 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet dd7029986a5246ee823e5fdb50683118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd7029986a5246ee823e5fdb50683118 already in progress: creating tablet
W20251027 21:07:52.356071 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa already in progress: creating tablet
I20251027 21:07:52.356304 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_913666abca6044f9aacb48e76118aac5" } modify_external_catalogs: true
I20251027 21:07:52.355459 10031 ts_tablet_manager.cc:1434] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:52.356568 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_913666abca6044f9aacb48e76118aac5" } modify_external_catalogs: true
I20251027 21:07:52.356986 10031 tablet_bootstrap.cc:492] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:52.357280 10040 tablet_replica.cc:333] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:52.357506 10040 raft_consensus.cc:2243] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.357684 10040 raft_consensus.cc:2272] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.358042 10043 ts_tablet_manager.cc:1929] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.358225 10040 ts_tablet_manager.cc:1916] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.358283 10043 log.cc:1199] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/d8ef795c88da49b58082ba1264d83378
W20251027 21:07:52.357267 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet adaa4f66bbab491f8030d439d85cc7d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet adaa4f66bbab491f8030d439d85cc7d1 already in progress: creating tablet
W20251027 21:07:52.359804 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet a9bf3a093be8498c9dd5511b53436b16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9bf3a093be8498c9dd5511b53436b16 already in progress: creating tablet
W20251027 21:07:52.360128 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 77797eea3e9c4f8099dc9d58f116200b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77797eea3e9c4f8099dc9d58f116200b already in progress: creating tablet
I20251027 21:07:52.360903 10031 tablet_bootstrap.cc:654] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.362767 10032 tablet_bootstrap.cc:492] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:52.363343 10032 ts_tablet_manager.cc:1403] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20251027 21:07:52.363559 10178 raft_consensus.cc:493] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:52.363705 10178 raft_consensus.cc:515] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.364367 10178 leader_election.cc:290] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:52.365060 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8fcfa6b85e7b46d78a84947e9f11ae46" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:52.365653 9859 leader_election.cc:343] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:52.366108 10032 raft_consensus.cc:359] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.366530 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8fcfa6b85e7b46d78a84947e9f11ae46" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:52.367058 10032 raft_consensus.cc:385] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251027 21:07:52.367127 9860 leader_election.cc:343] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:52.367252 9860 leader_election.cc:304] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:52.367156 10032 raft_consensus.cc:740] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.367694 10043 ts_tablet_manager.cc:1950] T d8ef795c88da49b58082ba1264d83378 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:52.367616 10032 consensus_queue.cc:260] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.368203 10178 raft_consensus.cc:2749] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:52.369486 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet d8ef795c88da49b58082ba1264d83378 (table default.Table_df95aebbdcf94d8289b235722b7d2c8c [id=98143d42942742568f2c3a1d0fbcc100]) successfully deleted
I20251027 21:07:52.369892 10043 tablet_replica.cc:333] T 36e168cc44274dae9682b0c1329d4558 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:52.370220 10031 tablet_bootstrap.cc:492] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:52.370386 10031 ts_tablet_manager.cc:1403] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.004s
I20251027 21:07:52.370119 10043 raft_consensus.cc:2243] T 36e168cc44274dae9682b0c1329d4558 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.370644 10043 raft_consensus.cc:2272] T 36e168cc44274dae9682b0c1329d4558 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.371208 10043 ts_tablet_manager.cc:1916] T 36e168cc44274dae9682b0c1329d4558 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.371186 10031 raft_consensus.cc:359] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.371376 10031 raft_consensus.cc:385] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.371474 10031 raft_consensus.cc:740] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.371702 10031 consensus_queue.cc:260] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.372205 10031 ts_tablet_manager.cc:1434] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:07:52.373703 10031 tablet_bootstrap.cc:492] T 81e914ecc3c74b869e9fc3ad102887ec P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:52.377919 10031 tablet_bootstrap.cc:654] T 81e914ecc3c74b869e9fc3ad102887ec P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.378284 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 3103c8e5a7c744be90be097179904ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.381556 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_60b6845211be4b7798d0690248d2b718" } new_table_name: "default.Table_c4cd97ce361b428799f703eec191abfc" modify_external_catalogs: true
I20251027 21:07:52.382589 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet efc0019840244d14bb95ae581793e74b with delete_type TABLET_DATA_DELETED (replaced by tablet d2203ee62f7e404c95c5f86b5ae85feb) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.382823 10041 tablet_replica.cc:333] T efc0019840244d14bb95ae581793e74b P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:52.383013 10041 raft_consensus.cc:2243] T efc0019840244d14bb95ae581793e74b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.383162 10041 raft_consensus.cc:2272] T efc0019840244d14bb95ae581793e74b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.383523 9306 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 1e5f5b57f5d948f3b2352451ccaacb09 (table default.Table_f0792045d49041578f5ff6459c99f77e [id=7f83f7c0f9524147bb778a06dd233a8e]). Expected version 1 got 0
I20251027 21:07:52.383955 10041 ts_tablet_manager.cc:1916] T efc0019840244d14bb95ae581793e74b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.384994 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 49b3d23798b740008c98cd0754ae33f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:52.387115 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a58cd07e1bb4ffdbf39e7e16a46022a consensus state has no leader
I20251027 21:07:52.387326 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4469fd75838845e9b60fd07ddc845f5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.387789 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4469fd75838845e9b60fd07ddc845f5c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.388459 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b247829a11349b6835a2f18906e20a9 with delete_type TABLET_DATA_DELETED (replaced by tablet fde0980aa64c4496861a662f7ed3ef75) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.389753 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 533f231e3ff64531ae2331409a961425
I20251027 21:07:52.391778 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 533f231e3ff64531ae2331409a961425 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.392675 10041 ts_tablet_manager.cc:1929] T efc0019840244d14bb95ae581793e74b P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.392877 10041 log.cc:1199] T efc0019840244d14bb95ae581793e74b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/efc0019840244d14bb95ae581793e74b
I20251027 21:07:52.393337 10041 ts_tablet_manager.cc:1950] T efc0019840244d14bb95ae581793e74b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:52.393358 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet 533f231e3ff64531ae2331409a961425 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.394304 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 533f231e3ff64531ae2331409a961425 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:52.394500 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04c579bb008448e9baa841135b6dbc61 consensus state has no leader
I20251027 21:07:52.394716 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet efc0019840244d14bb95ae581793e74b (table default.Table_89407f0c58ec4ee7995b892701a6377f [id=f773d65a07a4473ab8b87cd82ac50bcf]) successfully deleted
I20251027 21:07:52.394891 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 6bf9c2a78aac42528d67846dc58adc40
I20251027 21:07:52.395196 10041 tablet_replica.cc:333] T 49b3d23798b740008c98cd0754ae33f4 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:52.395344 10312 raft_consensus.cc:493] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:52.395519 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bf9c2a78aac42528d67846dc58adc40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.395496 10312 raft_consensus.cc:515] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.396263 10312 leader_election.cc:290] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:52.396916 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bf9c2a78aac42528d67846dc58adc40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.397126 10041 raft_consensus.cc:2243] T 49b3d23798b740008c98cd0754ae33f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.397351 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet 94a00ff844f0415fa419613d70a318e6 with delete_type TABLET_DATA_DELETED (replaced by tablet efc0019840244d14bb95ae581793e74b) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:52.397475 9568 leader_election.cc:343] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 9c4369a10bf846fb94c9b71a70f21600
I20251027 21:07:52.397606 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet 61cd8ee8e1f8459eb7af5dd48c018e02 with delete_type TABLET_DATA_DELETED (replaced by tablet 49d9e5c5960a4534a6603875006e6266) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.397835 10041 raft_consensus.cc:2272] T 49b3d23798b740008c98cd0754ae33f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.397830 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.397866 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bf9c2a78aac42528d67846dc58adc40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.398559 10041 ts_tablet_manager.cc:1916] T 49b3d23798b740008c98cd0754ae33f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:52.399497 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2532bf7190064443a898e47e843fd9ac consensus state has no leader
I20251027 21:07:52.405313 10043 ts_tablet_manager.cc:1929] T 36e168cc44274dae9682b0c1329d4558 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.406034 10043 log.cc:1199] T 36e168cc44274dae9682b0c1329d4558 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/36e168cc44274dae9682b0c1329d4558
I20251027 21:07:52.406678 10043 ts_tablet_manager.cc:1950] T 36e168cc44274dae9682b0c1329d4558 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:52.407603 10040 ts_tablet_manager.cc:1929] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.407739 10040 log.cc:1199] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/44783760defd433a972c9cd83933249c
I20251027 21:07:52.408098 10040 ts_tablet_manager.cc:1950] T 44783760defd433a972c9cd83933249c P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:52.408797 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 36e168cc44274dae9682b0c1329d4558 (table default.Table_795e30871da94371958578747d1c898b [id=9a111e9480b84a55b5226f0940d999c4]) successfully deleted
I20251027 21:07:52.409500 10043 tablet_replica.cc:333] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:52.409428 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 44783760defd433a972c9cd83933249c (table default.Table_44e3f093c8d7476a98074b07828da3ab [id=30fbbddfaed646e3b2d80f788f774741]) successfully deleted
I20251027 21:07:52.409750 10040 tablet_replica.cc:333] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:52.409926 10040 raft_consensus.cc:2243] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.409986 10043 raft_consensus.cc:2243] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251027 21:07:52.409957 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 2b247829a11349b6835a2f18906e20a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b247829a11349b6835a2f18906e20a9 already in progress: creating tablet
W20251027 21:07:52.410276 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 648f506ec87c4fb8a3f6d4591873dbb8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 648f506ec87c4fb8a3f6d4591873dbb8 already in progress: creating tablet
I20251027 21:07:52.410504 10043 raft_consensus.cc:2272] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251027 21:07:52.410756 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet a26ded6012174348b7295fc7c0e05e6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a26ded6012174348b7295fc7c0e05e6e already in progress: creating tablet
W20251027 21:07:52.411049 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet dc589852710342f7bd017193e4075303 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc589852710342f7bd017193e4075303 already in progress: creating tablet
I20251027 21:07:52.411226 10043 ts_tablet_manager.cc:1916] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:52.411326 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet e9edb62a8e4b4dfea6a31a2202545a6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9edb62a8e4b4dfea6a31a2202545a6e already in progress: creating tablet
I20251027 21:07:52.413679 10029 tablet_bootstrap.cc:492] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:52.413820 10029 ts_tablet_manager.cc:1403] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.261s user 0.001s sys 0.013s
I20251027 21:07:52.414570 10029 raft_consensus.cc:359] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.414737 10029 raft_consensus.cc:385] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.414840 10029 raft_consensus.cc:740] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.415031 10029 consensus_queue.cc:260] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.415515 10029 ts_tablet_manager.cc:1434] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:52.415814 10029 tablet_bootstrap.cc:492] T 2d659e4d73ea4ff3975a4b58a794090a P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:52.416328 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_019b51c3ff544dae94c53821dc97ab9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.416862 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_019b51c3ff544dae94c53821dc97ab9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.417438 10029 tablet_bootstrap.cc:654] T 2d659e4d73ea4ff3975a4b58a794090a P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.421566 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_918877c62b6c4722a5dcf9c7ca26dd76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:52.421674 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_03200469a2e740baab131dd8bcbe9da7" } new_table_name: "default.Table_62ebc53a16ee4e8ab2022fc3fdbb95b1" modify_external_catalogs: true
I20251027 21:07:52.422314 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:52.422111 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_918877c62b6c4722a5dcf9c7ca26dd76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.423355 9889 tablet_service.cc:1505] Processing CreateTablet for tablet e6b58b1f4f63461aa96e416cae879be8 (DEFAULT_TABLE table=default.Table_f3c2de263e8849caaef87b570fc69459 [id=ef50ef2238804c98ad19c5b1c0b881c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.423735 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6b58b1f4f63461aa96e416cae879be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.427335 10041 ts_tablet_manager.cc:1929] T 49b3d23798b740008c98cd0754ae33f4 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.427443 10041 log.cc:1199] T 49b3d23798b740008c98cd0754ae33f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/49b3d23798b740008c98cd0754ae33f4
I20251027 21:07:52.427806 10041 ts_tablet_manager.cc:1950] T 49b3d23798b740008c98cd0754ae33f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
W20251027 21:07:52.428390 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dd9e6656d8df43afbb7f70706bc14585 has no consensus state
I20251027 21:07:52.428751 10041 tablet_replica.cc:333] T 2b247829a11349b6835a2f18906e20a9 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:52.428932 10041 raft_consensus.cc:2243] T 2b247829a11349b6835a2f18906e20a9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.429095 10041 raft_consensus.cc:2272] T 2b247829a11349b6835a2f18906e20a9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.429314 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 49b3d23798b740008c98cd0754ae33f4 (table default.Table_5d5f327d728f4250841a9f4f10a2b533 [id=33dbf5fe986c415cb54fbf1b574651ac]) successfully deleted
I20251027 21:07:52.430841 10041 ts_tablet_manager.cc:1916] T 2b247829a11349b6835a2f18906e20a9 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.432040 10043 ts_tablet_manager.cc:1929] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.432233 10043 log.cc:1199] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/b1bdb2be5032415187c2826c14e75f07
I20251027 21:07:52.432627 10043 ts_tablet_manager.cc:1950] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:52.433465 9604 tablet_service.cc:1505] Processing CreateTablet for tablet 96a680e8f16343b1908829a67e960b60 (DEFAULT_TABLE table=default.Table_a46825cc888a4a0898d6db70b02e5ab6 [id=8579d3ff46d64c75b1da213fdee9cc1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.433913 9610 tablet_service.cc:1505] Processing CreateTablet for tablet e6b58b1f4f63461aa96e416cae879be8 (DEFAULT_TABLE table=default.Table_f3c2de263e8849caaef87b570fc69459 [id=ef50ef2238804c98ad19c5b1c0b881c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.434535 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6b58b1f4f63461aa96e416cae879be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.434721 10043 tablet_replica.cc:333] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:52.434942 10043 raft_consensus.cc:2243] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.435129 10043 raft_consensus.cc:2272] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.435659 10043 ts_tablet_manager.cc:1916] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.438546 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96a680e8f16343b1908829a67e960b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.442327 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet b1bdb2be5032415187c2826c14e75f07 (table default.Table_3e0f707f8e2844bfaf937150b747a935 [id=d7dcec1e03494fd5a219a97ea8b1ab1d]) successfully deleted
W20251027 21:07:52.442585 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 132a01405a63447ab27b2137b803224f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 132a01405a63447ab27b2137b803224f already in progress: creating tablet
W20251027 21:07:52.442867 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 26872be132fb46ee9e79a9c2e62374ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26872be132fb46ee9e79a9c2e62374ef already in progress: creating tablet
W20251027 21:07:52.443105 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 9b04349b15f3494bb0933db56b7b24bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b04349b15f3494bb0933db56b7b24bc already in progress: creating tablet
I20251027 21:07:52.443851 10043 ts_tablet_manager.cc:1929] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.444075 10043 log.cc:1199] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/61cd8ee8e1f8459eb7af5dd48c018e02
I20251027 21:07:52.444638 10043 ts_tablet_manager.cc:1950] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:52.445739 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 61cd8ee8e1f8459eb7af5dd48c018e02 (table default.Table_52c53bafe66b4c66b99e5b26e4df7ef5 [id=27dc835f316f41e091be9b2b1f560bbf]) successfully deleted
W20251027 21:07:52.446218 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet abc7458fcfb94985acf76d77c0292b08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abc7458fcfb94985acf76d77c0292b08 already in progress: creating tablet
W20251027 21:07:52.447012 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet a339c0dee0994788b8644912f74c1b82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a339c0dee0994788b8644912f74c1b82 already in progress: creating tablet
I20251027 21:07:52.447552 10043 tablet_replica.cc:333] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:52.447851 10043 raft_consensus.cc:2243] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251027 21:07:52.447855 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet a287439ea6c9469da5fb56d84ee68048 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a287439ea6c9469da5fb56d84ee68048 already in progress: creating tablet
I20251027 21:07:52.448114 10043 raft_consensus.cc:2272] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.438024 10041 ts_tablet_manager.cc:1929] T 2b247829a11349b6835a2f18906e20a9 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.448684 10041 log.cc:1199] T 2b247829a11349b6835a2f18906e20a9 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/2b247829a11349b6835a2f18906e20a9
I20251027 21:07:52.448947 10043 ts_tablet_manager.cc:1916] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.449060 10041 ts_tablet_manager.cc:1950] T 2b247829a11349b6835a2f18906e20a9 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:52.450100 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 96a680e8f16343b1908829a67e960b60 (DEFAULT_TABLE table=default.Table_a46825cc888a4a0898d6db70b02e5ab6 [id=8579d3ff46d64c75b1da213fdee9cc1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.450534 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96a680e8f16343b1908829a67e960b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.454422 9897 tablet_service.cc:1505] Processing CreateTablet for tablet a033551a591146039b017b4193b221ca (DEFAULT_TABLE table=default.Table_bef3b1992e374f98a2e2ccd86f02ae75 [id=84d2047a211e479e89130346cedd9fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.454739 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a033551a591146039b017b4193b221ca. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:52.456386 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 222383us
I20251027 21:07:52.456486 9310 master_service.cc:954] ReplaceTablet: received request to replace tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:52.461318 10043 ts_tablet_manager.cc:1929] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.461520 10043 log.cc:1199] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/cad3a3e910ac42399cf371ad82c89ff7
I20251027 21:07:52.461943 10043 ts_tablet_manager.cc:1950] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:52.462601 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet cad3a3e910ac42399cf371ad82c89ff7 (table default.Table_b84197018ff44b40b7219e21846339f2 [id=0286a1b3137649eeaa5339cbdf97aa03]) successfully deleted
I20251027 21:07:52.463012 10043 tablet_replica.cc:333] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:52.463236 10043 raft_consensus.cc:2243] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.463464 10043 raft_consensus.cc:2272] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.465112 10043 ts_tablet_manager.cc:1916] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.465158 10029 tablet_bootstrap.cc:492] T 2d659e4d73ea4ff3975a4b58a794090a P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:52.465418 10029 ts_tablet_manager.cc:1403] T 2d659e4d73ea4ff3975a4b58a794090a P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.005s
I20251027 21:07:52.466137 10032 ts_tablet_manager.cc:1434] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.102s user 0.015s sys 0.018s
I20251027 21:07:52.466735 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 2b247829a11349b6835a2f18906e20a9 (table default.Table_096b7cf08dbf49279d8d5c11b3b687ab [id=20b788f5a2d14b31a58e1b29fa954951]) successfully deleted
W20251027 21:07:52.466960 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 533f231e3ff64531ae2331409a961425 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 533f231e3ff64531ae2331409a961425 already in progress: creating tablet
I20251027 21:07:52.467073 10029 raft_consensus.cc:359] T 2d659e4d73ea4ff3975a4b58a794090a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.467286 10029 raft_consensus.cc:385] T 2d659e4d73ea4ff3975a4b58a794090a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.467384 10029 raft_consensus.cc:740] T 2d659e4d73ea4ff3975a4b58a794090a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.410120 10040 raft_consensus.cc:2272] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.467628 10029 consensus_queue.cc:260] T 2d659e4d73ea4ff3975a4b58a794090a P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.468093 10029 ts_tablet_manager.cc:1434] T 2d659e4d73ea4ff3975a4b58a794090a P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:07:52.468106 10040 ts_tablet_manager.cc:1916] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.466172 9610 tablet_service.cc:1505] Processing CreateTablet for tablet a033551a591146039b017b4193b221ca (DEFAULT_TABLE table=default.Table_bef3b1992e374f98a2e2ccd86f02ae75 [id=84d2047a211e479e89130346cedd9fdf]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:52.468252 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 6bf9c2a78aac42528d67846dc58adc40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bf9c2a78aac42528d67846dc58adc40 already in progress: creating tablet
I20251027 21:07:52.468484 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a033551a591146039b017b4193b221ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.468485 10032 tablet_bootstrap.cc:492] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:52.470022 10032 tablet_bootstrap.cc:654] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.450532 9753 tablet_service.cc:1505] Processing CreateTablet for tablet a033551a591146039b017b4193b221ca (DEFAULT_TABLE table=default.Table_bef3b1992e374f98a2e2ccd86f02ae75 [id=84d2047a211e479e89130346cedd9fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.458390 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5d8837ff3c44a66a1f933d80540ba0c" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:52.470472 9771 raft_consensus.cc:2468] T e5d8837ff3c44a66a1f933d80540ba0c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:52.470578 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a033551a591146039b017b4193b221ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.458842 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5d8837ff3c44a66a1f933d80540ba0c" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:52.470867 9773 raft_consensus.cc:3060] T e5d8837ff3c44a66a1f933d80540ba0c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:52.473775 9773 raft_consensus.cc:2468] T e5d8837ff3c44a66a1f933d80540ba0c P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:52.459298 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c4369a10bf846fb94c9b71a70f21600" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:52.474298 9772 raft_consensus.cc:2468] T 9c4369a10bf846fb94c9b71a70f21600 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:52.464251 10031 tablet_bootstrap.cc:492] T 81e914ecc3c74b869e9fc3ad102887ec P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:52.430507 9752 tablet_service.cc:1505] Processing CreateTablet for tablet e6b58b1f4f63461aa96e416cae879be8 (DEFAULT_TABLE table=default.Table_f3c2de263e8849caaef87b570fc69459 [id=ef50ef2238804c98ad19c5b1c0b881c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.474593 10031 ts_tablet_manager.cc:1403] T 81e914ecc3c74b869e9fc3ad102887ec P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.101s user 0.006s sys 0.002s
I20251027 21:07:52.474716 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6b58b1f4f63461aa96e416cae879be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.475414 10031 raft_consensus.cc:359] T 81e914ecc3c74b869e9fc3ad102887ec P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.475586 10031 raft_consensus.cc:385] T 81e914ecc3c74b869e9fc3ad102887ec P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.475658 10031 raft_consensus.cc:740] T 81e914ecc3c74b869e9fc3ad102887ec P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.475883 10031 consensus_queue.cc:260] T 81e914ecc3c74b869e9fc3ad102887ec P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.476411 10031 ts_tablet_manager.cc:1434] T 81e914ecc3c74b869e9fc3ad102887ec P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:52.476697 10031 tablet_bootstrap.cc:492] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:52.433200 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 96a680e8f16343b1908829a67e960b60 (DEFAULT_TABLE table=default.Table_a46825cc888a4a0898d6db70b02e5ab6 [id=8579d3ff46d64c75b1da213fdee9cc1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.478483 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96a680e8f16343b1908829a67e960b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.479038 10029 tablet_bootstrap.cc:492] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:52.481208 10029 tablet_bootstrap.cc:654] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.482218 10032 tablet_bootstrap.cc:492] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:52.482393 10032 ts_tablet_manager.cc:1403] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.004s
I20251027 21:07:52.483166 10032 raft_consensus.cc:359] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.483330 10032 raft_consensus.cc:385] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.483405 10032 raft_consensus.cc:740] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.456990 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cad3a3e910ac42399cf371ad82c89ff7" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:52.484022 9774 raft_consensus.cc:3060] T cad3a3e910ac42399cf371ad82c89ff7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:52.485343 10032 consensus_queue.cc:260] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.457839 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1bdb2be5032415187c2826c14e75f07" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:52.486373 9775 raft_consensus.cc:3060] T b1bdb2be5032415187c2826c14e75f07 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:52.486861 10032 ts_tablet_manager.cc:1434] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251027 21:07:52.487107 10032 tablet_bootstrap.cc:492] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:52.488674 10032 tablet_bootstrap.cc:654] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.488811 9775 raft_consensus.cc:2468] T b1bdb2be5032415187c2826c14e75f07 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:52.490805 10029 tablet_bootstrap.cc:492] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:52.491034 10029 ts_tablet_manager.cc:1403] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.003s
I20251027 21:07:52.491623 9774 raft_consensus.cc:2468] T cad3a3e910ac42399cf371ad82c89ff7 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:52.492622 10029 raft_consensus.cc:359] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.493575 10029 raft_consensus.cc:385] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.493750 10032 tablet_bootstrap.cc:492] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:52.493729 10029 raft_consensus.cc:740] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.493456 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d659e4d73ea4ff3975a4b58a794090a with delete_type TABLET_DATA_DELETED (replaced by tablet 6bf9c2a78aac42528d67846dc58adc40) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.493891 9566 leader_election.cc:304] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
W20251027 21:07:52.494117 9566 raft_consensus.cc:2686] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251027 21:07:52.494315 10032 ts_tablet_manager.cc:1403] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20251027 21:07:52.494313 10029 consensus_queue.cc:260] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.494439 9566 leader_election.cc:304] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:07:52.494886 9970 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:07:52.495525 10031 tablet_bootstrap.cc:654] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.495716 10032 raft_consensus.cc:359] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.496541 10032 raft_consensus.cc:385] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.496788 10032 raft_consensus.cc:740] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.497159 10032 consensus_queue.cc:260] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.498116 10032 ts_tablet_manager.cc:1434] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20251027 21:07:52.498369 9566 raft_consensus.cc:2686] T b1bdb2be5032415187c2826c14e75f07 P 51902b6b61ff4d5584ff21896e847e14: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251027 21:07:52.498852 10032 tablet_bootstrap.cc:492] T 779b7f63b0fa4f9ab47cf5799b3e085d P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:52.498994 9566 leader_election.cc:304] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
W20251027 21:07:52.499200 9566 raft_consensus.cc:2686] T cad3a3e910ac42399cf371ad82c89ff7 P 51902b6b61ff4d5584ff21896e847e14: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251027 21:07:52.500144 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet 81e914ecc3c74b869e9fc3ad102887ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.501827 10032 tablet_bootstrap.cc:654] T 779b7f63b0fa4f9ab47cf5799b3e085d P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.502384 10029 ts_tablet_manager.cc:1434] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20251027 21:07:52.503878 10043 ts_tablet_manager.cc:1929] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.506177 10043 log.cc:1199] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/9c4369a10bf846fb94c9b71a70f21600
I20251027 21:07:52.508695 10043 ts_tablet_manager.cc:1950] T 9c4369a10bf846fb94c9b71a70f21600 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:52.509728 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 9c4369a10bf846fb94c9b71a70f21600 (table default.Table_ec86de5add82433b81e1060e76d559e3 [id=731981b3bb2c4078a597da73199e31d8]) successfully deleted
I20251027 21:07:52.509891 10032 tablet_bootstrap.cc:492] T 779b7f63b0fa4f9ab47cf5799b3e085d P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:52.506014 10029 tablet_bootstrap.cc:492] T 33e41145c79b4465b8d564741174b8e0 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:52.510413 10032 ts_tablet_manager.cc:1403] T 779b7f63b0fa4f9ab47cf5799b3e085d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20251027 21:07:52.511350 10032 raft_consensus.cc:359] T 779b7f63b0fa4f9ab47cf5799b3e085d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.511626 10029 tablet_bootstrap.cc:654] T 33e41145c79b4465b8d564741174b8e0 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.512490 10032 raft_consensus.cc:385] T 779b7f63b0fa4f9ab47cf5799b3e085d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.512809 10032 raft_consensus.cc:740] T 779b7f63b0fa4f9ab47cf5799b3e085d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
W20251027 21:07:52.513262 9298 catalog_manager.cc:4952] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 5c34c8f1433b40d491ae05e7c6e60ba1 because the tablet was not found. No further retry: Not found: Tablet not found: 5c34c8f1433b40d491ae05e7c6e60ba1
I20251027 21:07:52.513258 10032 consensus_queue.cc:260] T 779b7f63b0fa4f9ab47cf5799b3e085d P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.513929 10043 tablet_replica.cc:333] T 94a00ff844f0415fa419613d70a318e6 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:52.513900 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_dbda62167b92488788bb65bb4b75b73a" } new_table_name: "default.Table_e74355d0f90e4c50abfb229be0c398c8" modify_external_catalogs: true
I20251027 21:07:52.514232 10043 raft_consensus.cc:2243] T 94a00ff844f0415fa419613d70a318e6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.514521 10032 ts_tablet_manager.cc:1434] T 779b7f63b0fa4f9ab47cf5799b3e085d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251027 21:07:52.514542 10043 raft_consensus.cc:2272] T 94a00ff844f0415fa419613d70a318e6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.515085 10032 tablet_bootstrap.cc:492] T a9cc64ad85334daa9a09538e24ebd8a5 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:52.515151 10043 ts_tablet_manager.cc:1916] T 94a00ff844f0415fa419613d70a318e6 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.515426 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3f8272b52c50446086e4ed9f69f4ae82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.515954 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f8272b52c50446086e4ed9f69f4ae82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.519341 10032 tablet_bootstrap.cc:654] T a9cc64ad85334daa9a09538e24ebd8a5 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.519922 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3f5ab2fc1a934004816ae27b651671ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.520412 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f5ab2fc1a934004816ae27b651671ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.521389 10029 tablet_bootstrap.cc:492] T 33e41145c79b4465b8d564741174b8e0 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:52.521622 10029 ts_tablet_manager.cc:1403] T 33e41145c79b4465b8d564741174b8e0 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
W20251027 21:07:52.521989 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8856c5dc8f144358445165324bfb477 consensus state has no leader
I20251027 21:07:52.522642 10029 raft_consensus.cc:359] T 33e41145c79b4465b8d564741174b8e0 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.522934 10029 raft_consensus.cc:385] T 33e41145c79b4465b8d564741174b8e0 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.523054 10029 raft_consensus.cc:740] T 33e41145c79b4465b8d564741174b8e0 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.523479 10029 consensus_queue.cc:260] T 33e41145c79b4465b8d564741174b8e0 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.524111 10029 ts_tablet_manager.cc:1434] T 33e41145c79b4465b8d564741174b8e0 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:52.524926 10029 tablet_bootstrap.cc:492] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:52.527292 10029 tablet_bootstrap.cc:654] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.527989 10032 tablet_bootstrap.cc:492] T a9cc64ad85334daa9a09538e24ebd8a5 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:52.528635 10032 ts_tablet_manager.cc:1403] T a9cc64ad85334daa9a09538e24ebd8a5 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.002s
I20251027 21:07:52.528709 9682 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:07:52.529497 10032 raft_consensus.cc:359] T a9cc64ad85334daa9a09538e24ebd8a5 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.531009 10032 raft_consensus.cc:385] T a9cc64ad85334daa9a09538e24ebd8a5 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.531770 10031 tablet_bootstrap.cc:492] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:52.531791 10032 raft_consensus.cc:740] T a9cc64ad85334daa9a09538e24ebd8a5 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.531957 10031 ts_tablet_manager.cc:1403] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.006s
I20251027 21:07:52.532263 10032 consensus_queue.cc:260] T a9cc64ad85334daa9a09538e24ebd8a5 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.532833 10031 raft_consensus.cc:359] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.533084 10031 raft_consensus.cc:385] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.533183 10032 ts_tablet_manager.cc:1434] T a9cc64ad85334daa9a09538e24ebd8a5 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251027 21:07:52.533213 10031 raft_consensus.cc:740] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.533500 10031 consensus_queue.cc:260] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.533828 10032 tablet_bootstrap.cc:492] T 77797eea3e9c4f8099dc9d58f116200b P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:52.535699 10032 tablet_bootstrap.cc:654] T 77797eea3e9c4f8099dc9d58f116200b P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.537406 10043 ts_tablet_manager.cc:1929] T 94a00ff844f0415fa419613d70a318e6 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.537550 10031 ts_tablet_manager.cc:1434] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251027 21:07:52.537581 10043 log.cc:1199] T 94a00ff844f0415fa419613d70a318e6 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/94a00ff844f0415fa419613d70a318e6
I20251027 21:07:52.537907 10043 ts_tablet_manager.cc:1950] T 94a00ff844f0415fa419613d70a318e6 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:52.538784 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d601b22c11c34ebd8559009cd144affe" } modify_external_catalogs: true
I20251027 21:07:52.538978 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d601b22c11c34ebd8559009cd144affe" } modify_external_catalogs: true
I20251027 21:07:52.541599 10371 raft_consensus.cc:493] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:52.541725 10371 raft_consensus.cc:515] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.542196 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 94a00ff844f0415fa419613d70a318e6 (table default.Table_89407f0c58ec4ee7995b892701a6377f [id=f773d65a07a4473ab8b87cd82ac50bcf]) successfully deleted
I20251027 21:07:52.542498 10371 leader_election.cc:290] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
W20251027 21:07:52.542642 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet efc0019840244d14bb95ae581793e74b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efc0019840244d14bb95ae581793e74b already in progress: creating tablet
I20251027 21:07:52.540122 10031 tablet_bootstrap.cc:492] T 601c74459d3a4a41b812fdcdf8b0a62c P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:52.543443 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f7ad06a15c04efbb701634e14904351" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:52.543501 9298 catalog_manager.cc:4952] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 5c34c8f1433b40d491ae05e7c6e60ba1 because the tablet was not found. No further retry: Not found: Tablet not found: 5c34c8f1433b40d491ae05e7c6e60ba1
W20251027 21:07:52.543968 9860 leader_election.cc:343] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:52.544185 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d659e4d73ea4ff3975a4b58a794090a with delete_type TABLET_DATA_DELETED (replaced by tablet 6bf9c2a78aac42528d67846dc58adc40) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:52.544293 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 01e9db9eaf8e41c6a2ee13b9ed127421 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01e9db9eaf8e41c6a2ee13b9ed127421 already in progress: creating tablet
I20251027 21:07:52.544984 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 29693cf06bc24930870dc60457126519 (DEFAULT_TABLE table=default.Table_2994a62669894e3baf66db0e580ece68 [id=d46947a29ddf4d94b5a9da22d9e5383e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.545089 10043 tablet_replica.cc:333] T 33e41145c79b4465b8d564741174b8e0 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:52.545300 10043 raft_consensus.cc:2243] T 33e41145c79b4465b8d564741174b8e0 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.545485 10043 raft_consensus.cc:2272] T 33e41145c79b4465b8d564741174b8e0 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.546288 10032 tablet_bootstrap.cc:492] T 77797eea3e9c4f8099dc9d58f116200b P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:52.546464 10032 ts_tablet_manager.cc:1403] T 77797eea3e9c4f8099dc9d58f116200b P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.002s
I20251027 21:07:52.546590 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet f22f013b25074d4db9c7f13caf5b9b42
I20251027 21:07:52.546604 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f7ad06a15c04efbb701634e14904351" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:52.547035 9859 leader_election.cc:343] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:52.547179 9859 leader_election.cc:304] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:52.547405 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet f22f013b25074d4db9c7f13caf5b9b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.548077 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet f22f013b25074d4db9c7f13caf5b9b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.548518 10043 ts_tablet_manager.cc:1916] T 33e41145c79b4465b8d564741174b8e0 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.549160 9306 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 1e5f5b57f5d948f3b2352451ccaacb09 (table default.Table_f0792045d49041578f5ff6459c99f77e [id=7f83f7c0f9524147bb778a06dd233a8e]). Expected version 1 got 0
I20251027 21:07:52.549456 10031 tablet_bootstrap.cc:654] T 601c74459d3a4a41b812fdcdf8b0a62c P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.551662 10029 tablet_bootstrap.cc:492] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:52.551868 10029 ts_tablet_manager.cc:1403] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.000s
I20251027 21:07:52.552774 10029 raft_consensus.cc:359] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.553020 10029 raft_consensus.cc:385] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.553191 10029 raft_consensus.cc:740] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.553462 10029 consensus_queue.cc:260] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.553994 10029 ts_tablet_manager.cc:1434] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:52.554895 10029 tablet_bootstrap.cc:492] T 10a44aecfade42969454126b990eca24 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:52.554966 10371 raft_consensus.cc:2749] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:52.555176 10032 raft_consensus.cc:359] T 77797eea3e9c4f8099dc9d58f116200b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.555305 10040 ts_tablet_manager.cc:1929] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.555356 10032 raft_consensus.cc:385] T 77797eea3e9c4f8099dc9d58f116200b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.555419 10040 log.cc:1199] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/0c1a6a7616f24f69b83ec8a282746e92
I20251027 21:07:52.555785 10040 ts_tablet_manager.cc:1950] T 0c1a6a7616f24f69b83ec8a282746e92 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:52.557485 10029 tablet_bootstrap.cc:654] T 10a44aecfade42969454126b990eca24 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.562757 10043 ts_tablet_manager.cc:1929] T 33e41145c79b4465b8d564741174b8e0 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.562918 10043 log.cc:1199] T 33e41145c79b4465b8d564741174b8e0 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/33e41145c79b4465b8d564741174b8e0
I20251027 21:07:52.563397 10043 ts_tablet_manager.cc:1950] T 33e41145c79b4465b8d564741174b8e0 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:52.564486 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet f22f013b25074d4db9c7f13caf5b9b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.565047 10371 raft_consensus.cc:493] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:52.565202 10371 raft_consensus.cc:515] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.565862 10371 leader_election.cc:290] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
W20251027 21:07:52.566248 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet f22f013b25074d4db9c7f13caf5b9b42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f22f013b25074d4db9c7f13caf5b9b42 already in progress: creating tablet
I20251027 21:07:52.566320 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c49775486ff24686b0f45c47e2757aba" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:52.566463 10032 raft_consensus.cc:740] T 77797eea3e9c4f8099dc9d58f116200b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.566771 10032 consensus_queue.cc:260] T 77797eea3e9c4f8099dc9d58f116200b P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:52.566932 9859 leader_election.cc:343] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:52.567317 10032 ts_tablet_manager.cc:1434] T 77797eea3e9c4f8099dc9d58f116200b P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20251027 21:07:52.567627 10032 tablet_bootstrap.cc:492] T 533f231e3ff64531ae2331409a961425 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:52.568583 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet c49775486ff24686b0f45c47e2757aba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.568773 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c49775486ff24686b0f45c47e2757aba" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:52.568843 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet a9cc64ad85334daa9a09538e24ebd8a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.569208 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet 779b7f63b0fa4f9ab47cf5799b3e085d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:52.569408 9860 leader_election.cc:343] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:52.569485 10029 tablet_bootstrap.cc:492] T 10a44aecfade42969454126b990eca24 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:52.569543 9860 leader_election.cc:304] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:52.569633 10029 ts_tablet_manager.cc:1403] T 10a44aecfade42969454126b990eca24 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.002s
I20251027 21:07:52.569792 10371 raft_consensus.cc:2749] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:52.570592 10029 raft_consensus.cc:359] T 10a44aecfade42969454126b990eca24 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.570848 10029 raft_consensus.cc:385] T 10a44aecfade42969454126b990eca24 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.570968 10029 raft_consensus.cc:740] T 10a44aecfade42969454126b990eca24 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.571237 10029 consensus_queue.cc:260] T 10a44aecfade42969454126b990eca24 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.571365 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 33e41145c79b4465b8d564741174b8e0 (table default.Table_e4d0e9336a834d6d8ab29ac0b1de7ace [id=af5b5b94fcef4677affe7e7afaa7f023]) successfully deleted
I20251027 21:07:52.571833 10041 tablet_replica.cc:333] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:52.572583 10041 raft_consensus.cc:2243] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.572763 10041 raft_consensus.cc:2272] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.573606 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet 77797eea3e9c4f8099dc9d58f116200b with delete_type TABLET_DATA_DELETED (replaced by tablet 631ff72613734913818a3cad24f20dbc) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.573915 10041 ts_tablet_manager.cc:1916] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:52.574126 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 3103c8e5a7c744be90be097179904ba3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3103c8e5a7c744be90be097179904ba3 already in progress: creating tablet
I20251027 21:07:52.574226 10029 ts_tablet_manager.cc:1434] T 10a44aecfade42969454126b990eca24 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251027 21:07:52.575047 10029 tablet_bootstrap.cc:492] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
W20251027 21:07:52.575979 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 533f231e3ff64531ae2331409a961425 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 533f231e3ff64531ae2331409a961425 already in progress: creating tablet
I20251027 21:07:52.546315 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29693cf06bc24930870dc60457126519. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:52.576973 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 6bf9c2a78aac42528d67846dc58adc40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bf9c2a78aac42528d67846dc58adc40 already in progress: creating tablet
W20251027 21:07:52.577505 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 132a01405a63447ab27b2137b803224f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 132a01405a63447ab27b2137b803224f already in progress: creating tablet
I20251027 21:07:52.577669 10040 tablet_replica.cc:333] T 69b1fb42428748239d8939d98428d8fe P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:52.577868 10040 raft_consensus.cc:2243] T 69b1fb42428748239d8939d98428d8fe P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.578022 10040 raft_consensus.cc:2272] T 69b1fb42428748239d8939d98428d8fe P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:07:52.578013 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 26872be132fb46ee9e79a9c2e62374ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26872be132fb46ee9e79a9c2e62374ef already in progress: creating tablet
I20251027 21:07:52.578680 10040 ts_tablet_manager.cc:1916] T 69b1fb42428748239d8939d98428d8fe P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.579098 10032 tablet_bootstrap.cc:654] T 533f231e3ff64531ae2331409a961425 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.579926 9310 master_service.cc:954] ReplaceTablet: received request to replace tablet 65fbcaa5447c48818fabfccac22c4048 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:52.585577 10359 raft_consensus.cc:493] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:52.585721 10359 raft_consensus.cc:515] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.586664 10359 leader_election.cc:290] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:52.586920 10041 ts_tablet_manager.cc:1929] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.587023 10041 log.cc:1199] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/c49775486ff24686b0f45c47e2757aba
I20251027 21:07:52.587386 10041 ts_tablet_manager.cc:1950] T c49775486ff24686b0f45c47e2757aba P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:52.587450 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d659e4d73ea4ff3975a4b58a794090a" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:52.587632 9631 raft_consensus.cc:2468] T 2d659e4d73ea4ff3975a4b58a794090a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:07:52.588299 9710 leader_election.cc:304] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:07:52.588531 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet c49775486ff24686b0f45c47e2757aba (table default.Table_b534fdb9e3084d6f9656dfdbb0e01750 [id=dcefc291ce454343964e99be8876837c]) successfully deleted
W20251027 21:07:52.588757 9711 leader_election.cc:343] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 2d659e4d73ea4ff3975a4b58a794090a
I20251027 21:07:52.588819 10041 tablet_replica.cc:333] T a9cc64ad85334daa9a09538e24ebd8a5 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:52.589177 10359 raft_consensus.cc:2804] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:52.589300 10359 raft_consensus.cc:493] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:52.589406 10359 raft_consensus.cc:3060] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:52.589893 10358 raft_consensus.cc:493] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:52.590020 10358 raft_consensus.cc:515] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.591126 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd57440d3b2a4f37a8db7d1fa4bc4dbe" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:52.591323 9774 raft_consensus.cc:2468] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:52.591710 9566 leader_election.cc:304] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:07:52.591972 10312 raft_consensus.cc:2804] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:52.592080 10312 raft_consensus.cc:493] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:52.592185 10312 raft_consensus.cc:3060] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:52.592507 10359 raft_consensus.cc:515] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.594022 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d659e4d73ea4ff3975a4b58a794090a" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:07:52.594245 9631 raft_consensus.cc:3060] T 2d659e4d73ea4ff3975a4b58a794090a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:52.595449 10312 raft_consensus.cc:515] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.595657 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_18e48b818fc040bfa4272f08e203dc3e" } new_table_name: "default.Table_9c6b1e43fc5f4f4389620f1ee244db12" modify_external_catalogs: true
I20251027 21:07:52.596959 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_92dfa85c272446cb910801b5f7050bcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.597080 9568 leader_election.cc:343] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: cd57440d3b2a4f37a8db7d1fa4bc4dbe
W20251027 21:07:52.597463 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92dfa85c272446cb910801b5f7050bcf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.597488 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd57440d3b2a4f37a8db7d1fa4bc4dbe" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:52.597644 9774 raft_consensus.cc:3060] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:52.597960 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2e2c99ff6d264bd2bc52dbb5a6a725b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:52.598198 10312 leader_election.cc:290] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:07:52.598452 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e2c99ff6d264bd2bc52dbb5a6a725b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.598340 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_faf6fa3adb644cdf984ac3704b878293" } modify_external_catalogs: true
I20251027 21:07:52.598414 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7a3ebf1ecf35428495d2d7bb5540530a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.598842 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a3ebf1ecf35428495d2d7bb5540530a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.599500 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4d2a5a0a0ac24d4bb907b89a6bee1aaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.599861 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d2a5a0a0ac24d4bb907b89a6bee1aaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.600232 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_faf6fa3adb644cdf984ac3704b878293" } modify_external_catalogs: true
I20251027 21:07:52.600364 9774 raft_consensus.cc:2468] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:52.600771 9566 leader_election.cc:304] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:07:52.600891 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_fa13b0f459f9443e965078c8867a629c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:52.601120 10312 raft_consensus.cc:2804] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
W20251027 21:07:52.601315 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa13b0f459f9443e965078c8867a629c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.601845 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_fc695aa989384c1ba355a09381428926" } modify_external_catalogs: true
I20251027 21:07:52.601974 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_fc695aa989384c1ba355a09381428926" } modify_external_catalogs: true
I20251027 21:07:52.603091 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_44ef0d8b29e34b8fb63bf6a7532a47fb" } new_table_name: "default.Table_d395c65b39c04ef2a3ce8490e8487f47" modify_external_catalogs: true
W20251027 21:07:52.603281 9568 leader_election.cc:343] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: cd57440d3b2a4f37a8db7d1fa4bc4dbe
I20251027 21:07:52.603583 10358 leader_election.cc:290] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:52.605062 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 81e914ecc3c74b869e9fc3ad102887ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.606959 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet 970fbf01a4c3406dafe1ea4cd005ea23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.607848 9631 raft_consensus.cc:2468] T 2d659e4d73ea4ff3975a4b58a794090a P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:07:52.608423 9710 leader_election.cc:304] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:07:52.608686 10141 raft_consensus.cc:2804] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
W20251027 21:07:52.613384 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ccaf5c2f8e544842880ba8ba27300ae6 consensus state has no leader
I20251027 21:07:52.613739 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_294e3c4092274dc1a146d3f604bc2179" } modify_external_catalogs: true
I20251027 21:07:52.613893 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_294e3c4092274dc1a146d3f604bc2179" } modify_external_catalogs: true
I20251027 21:07:52.615581 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a061b11818fc4a2db61e51a5972d69db" } modify_external_catalogs: true
I20251027 21:07:52.615720 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a061b11818fc4a2db61e51a5972d69db" } modify_external_catalogs: true
I20251027 21:07:52.616381 10043 tablet_replica.cc:333] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:52.616922 10359 leader_election.cc:290] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:07:52.617336 9711 leader_election.cc:343] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 2d659e4d73ea4ff3975a4b58a794090a
I20251027 21:07:52.618652 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9e88d5f78e27447c8072ce0563686ac8" } new_table_name: "default.Table_fc2da109d1934a3fbe14ae51b20e771d" modify_external_catalogs: true
I20251027 21:07:52.619534 10029 tablet_bootstrap.cc:654] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.621682 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 29693cf06bc24930870dc60457126519 (DEFAULT_TABLE table=default.Table_2994a62669894e3baf66db0e580ece68 [id=d46947a29ddf4d94b5a9da22d9e5383e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.622611 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29693cf06bc24930870dc60457126519. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.622797 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4658a756ae8149cf9e248c65a102bdb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:52.622071 9897 tablet_service.cc:1505] Processing CreateTablet for tablet a4d26f73b8534f08bcaa386d4de75c01 (DEFAULT_TABLE table=default.Table_a6cf2819ed9241cea09738477423c84a [id=f5ef094809e24355a7e021408e2ddd20]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:52.623174 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4658a756ae8149cf9e248c65a102bdb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.623220 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4d26f73b8534f08bcaa386d4de75c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.625672 10032 tablet_bootstrap.cc:492] T 533f231e3ff64531ae2331409a961425 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:52.625833 10032 ts_tablet_manager.cc:1403] T 533f231e3ff64531ae2331409a961425 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.000s
I20251027 21:07:52.588989 10041 raft_consensus.cc:2243] T a9cc64ad85334daa9a09538e24ebd8a5 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.626574 10041 raft_consensus.cc:2272] T a9cc64ad85334daa9a09538e24ebd8a5 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.626641 10032 raft_consensus.cc:359] T 533f231e3ff64531ae2331409a961425 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.626896 10032 raft_consensus.cc:385] T 533f231e3ff64531ae2331409a961425 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.626971 10032 raft_consensus.cc:740] T 533f231e3ff64531ae2331409a961425 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.627701 9749 tablet_service.cc:1505] Processing CreateTablet for tablet a4d26f73b8534f08bcaa386d4de75c01 (DEFAULT_TABLE table=default.Table_a6cf2819ed9241cea09738477423c84a [id=f5ef094809e24355a7e021408e2ddd20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.628027 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4d26f73b8534f08bcaa386d4de75c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.628803 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet fe24af9838954a9bb680474800642e62
W20251027 21:07:52.629544 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7425b6b0184e4d0b871777b25d904ac4 has no consensus state
I20251027 21:07:52.630621 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet fe24af9838954a9bb680474800642e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.630767 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet fe24af9838954a9bb680474800642e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.631217 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_38e99066d2014865a10ad4f614cf0339" } new_table_name: "default.Table_796614d1dc4148699dfa07316ea8febf" modify_external_catalogs: true
I20251027 21:07:52.633111 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 631ff72613734913818a3cad24f20dbc (DEFAULT_TABLE table=default.Table_ed23972576b74b5fbc7167cf7b541bfc [id=e7b7bdb2ea7f468c98bfe88f900a5be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.633687 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet fe24af9838954a9bb680474800642e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.630034 10032 consensus_queue.cc:260] T 533f231e3ff64531ae2331409a961425 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.635129 10032 ts_tablet_manager.cc:1434] T 533f231e3ff64531ae2331409a961425 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251027 21:07:52.635389 9310 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet e0385c664abb428bb01ecc9942c0d214
I20251027 21:07:52.636269 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 631ff72613734913818a3cad24f20dbc (DEFAULT_TABLE table=default.Table_ed23972576b74b5fbc7167cf7b541bfc [id=e7b7bdb2ea7f468c98bfe88f900a5be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.636626 9604 tablet_service.cc:1505] Processing CreateTablet for tablet a4d26f73b8534f08bcaa386d4de75c01 (DEFAULT_TABLE table=default.Table_a6cf2819ed9241cea09738477423c84a [id=f5ef094809e24355a7e021408e2ddd20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.636901 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 631ff72613734913818a3cad24f20dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.637866 10029 tablet_bootstrap.cc:492] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:52.638059 10029 ts_tablet_manager.cc:1403] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.002s
I20251027 21:07:52.638943 10029 raft_consensus.cc:359] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.639182 10029 raft_consensus.cc:385] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.639328 10029 raft_consensus.cc:740] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.639552 10029 consensus_queue.cc:260] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.640208 10029 ts_tablet_manager.cc:1434] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:52.640497 10029 tablet_bootstrap.cc:492] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:52.642268 10041 ts_tablet_manager.cc:1916] T a9cc64ad85334daa9a09538e24ebd8a5 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.642351 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4d26f73b8534f08bcaa386d4de75c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.643376 10032 tablet_bootstrap.cc:492] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:52.645661 10032 tablet_bootstrap.cc:654] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.645879 10029 tablet_bootstrap.cc:654] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.634243 10141 raft_consensus.cc:697] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:07:52.646562 10141 consensus_queue.cc:237] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.648765 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet 533f231e3ff64531ae2331409a961425 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.651208 10388 raft_consensus.cc:493] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:52.651407 10388 raft_consensus.cc:515] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.638075 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 29693cf06bc24930870dc60457126519 (DEFAULT_TABLE table=default.Table_2994a62669894e3baf66db0e580ece68 [id=d46947a29ddf4d94b5a9da22d9e5383e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.651744 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 631ff72613734913818a3cad24f20dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.653916 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 9c2d713d72fd4b5ca34cb3e828c75761
I20251027 21:07:52.654670 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 288b88e13fdf4427aae53f64b87733bf
I20251027 21:07:52.654897 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c2d713d72fd4b5ca34cb3e828c75761 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:52.655781 9710 leader_election.cc:343] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 61cd8ee8e1f8459eb7af5dd48c018e02
I20251027 21:07:52.655992 10388 leader_election.cc:290] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:52.657316 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c2d713d72fd4b5ca34cb3e828c75761 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.657627 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 288b88e13fdf4427aae53f64b87733bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.658035 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 631ff72613734913818a3cad24f20dbc (DEFAULT_TABLE table=default.Table_ed23972576b74b5fbc7167cf7b541bfc [id=e7b7bdb2ea7f468c98bfe88f900a5be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.658495 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 631ff72613734913818a3cad24f20dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.658452 9890 tablet_service.cc:1505] Processing CreateTablet for tablet aabaa34787cf4d179822269b7f323314 (DEFAULT_TABLE table=default.Table_0f7c274ead214bb68565a54bc2da4f63 [id=a0a10aefbf4c4ab083982e9e46bba513]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.658777 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aabaa34787cf4d179822269b7f323314. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.659040 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 5c34c8f1433b40d491ae05e7c6e60ba1 (DEFAULT_TABLE table=default.Table_5877315061df40eb80ba9d966632471a [id=c70fb6448778479fa17e811b21830ea5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.659448 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c34c8f1433b40d491ae05e7c6e60ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.659775 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 288b88e13fdf4427aae53f64b87733bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.660152 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 5c34c8f1433b40d491ae05e7c6e60ba1 (DEFAULT_TABLE table=default.Table_5877315061df40eb80ba9d966632471a [id=c70fb6448778479fa17e811b21830ea5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.660225 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_ab9ae94ee52942d588d846e58da2e533"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:52.660501 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c34c8f1433b40d491ae05e7c6e60ba1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:52.660725 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab9ae94ee52942d588d846e58da2e533 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.662917 10031 tablet_bootstrap.cc:492] T 601c74459d3a4a41b812fdcdf8b0a62c P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:52.663084 10031 ts_tablet_manager.cc:1403] T 601c74459d3a4a41b812fdcdf8b0a62c P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.123s user 0.003s sys 0.001s
I20251027 21:07:52.663720 10029 tablet_bootstrap.cc:492] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:52.663857 10029 ts_tablet_manager.cc:1403] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.003s
I20251027 21:07:52.663913 10031 raft_consensus.cc:359] T 601c74459d3a4a41b812fdcdf8b0a62c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.664117 10031 raft_consensus.cc:385] T 601c74459d3a4a41b812fdcdf8b0a62c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.664192 10031 raft_consensus.cc:740] T 601c74459d3a4a41b812fdcdf8b0a62c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.664420 10031 consensus_queue.cc:260] T 601c74459d3a4a41b812fdcdf8b0a62c P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.664621 10029 raft_consensus.cc:359] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.664814 10029 raft_consensus.cc:385] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.664872 10031 ts_tablet_manager.cc:1434] T 601c74459d3a4a41b812fdcdf8b0a62c P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:52.664918 10029 raft_consensus.cc:740] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.665162 10031 tablet_bootstrap.cc:492] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:52.665133 10029 consensus_queue.cc:260] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:52.665971 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c3b9a461567a48fcb4af514ecdf618b4 consensus state has no leader
I20251027 21:07:52.666283 10029 ts_tablet_manager.cc:1434] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251027 21:07:52.666642 10029 tablet_bootstrap.cc:492] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:52.668084 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_aeaed62246864354b04616e1e4461644"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:52.668444 10029 tablet_bootstrap.cc:654] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:52.668706 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aeaed62246864354b04616e1e4461644 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:52.671535 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9ed4f9c61ad54e1292281b30bcd19533 has no consensus state
I20251027 21:07:52.673321 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet a287439ea6c9469da5fb56d84ee68048 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:52.674630 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c2d713d72fd4b5ca34cb3e828c75761 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.674875 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 288b88e13fdf4427aae53f64b87733bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.675032 9741 tablet_service.cc:1505] Processing CreateTablet for tablet 5c34c8f1433b40d491ae05e7c6e60ba1 (DEFAULT_TABLE table=default.Table_5877315061df40eb80ba9d966632471a [id=c70fb6448778479fa17e811b21830ea5]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:52.675268 9711 leader_election.cc:343] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 61cd8ee8e1f8459eb7af5dd48c018e02
I20251027 21:07:52.675428 9711 leader_election.cc:304] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:52.675944 10359 raft_consensus.cc:2749] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:52.680469 10312 raft_consensus.cc:697] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:07:52.680898 10312 consensus_queue.cc:237] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.684528 9749 tablet_service.cc:1505] Processing CreateTablet for tablet aabaa34787cf4d179822269b7f323314 (DEFAULT_TABLE table=default.Table_0f7c274ead214bb68565a54bc2da4f63 [id=a0a10aefbf4c4ab083982e9e46bba513]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.684863 10359 raft_consensus.cc:493] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:52.684991 10359 raft_consensus.cc:515] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
W20251027 21:07:52.686323 9710 leader_election.cc:343] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 94a00ff844f0415fa419613d70a318e6
I20251027 21:07:52.686576 10359 leader_election.cc:290] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
W20251027 21:07:52.687927 9711 leader_election.cc:343] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 94a00ff844f0415fa419613d70a318e6
I20251027 21:07:52.688077 9711 leader_election.cc:304] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:52.688905 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29693cf06bc24930870dc60457126519. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.689191 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c34c8f1433b40d491ae05e7c6e60ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.689380 10359 raft_consensus.cc:2749] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:52.694372 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aabaa34787cf4d179822269b7f323314. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.696713 10031 tablet_bootstrap.cc:654] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.702045 9753 tablet_service.cc:1505] Processing CreateTablet for tablet f2710f265ae9485dafec389168328067 (DEFAULT_TABLE table=default.Table_cf6134c7114f4ad5b61f1bda6ffa5015 [id=27d1cff8fc7940c9abb741c8c7d87e24]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.702430 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2710f265ae9485dafec389168328067. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.703243 9741 tablet_service.cc:1552] Processing DeleteTablet for tablet 601c74459d3a4a41b812fdcdf8b0a62c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.704360 9604 tablet_service.cc:1505] Processing CreateTablet for tablet f2710f265ae9485dafec389168328067 (DEFAULT_TABLE table=default.Table_cf6134c7114f4ad5b61f1bda6ffa5015 [id=27d1cff8fc7940c9abb741c8c7d87e24]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.704753 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2710f265ae9485dafec389168328067. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.706847 9614 tablet_service.cc:1505] Processing CreateTablet for tablet d672118ecd6f4094855adf106e5f7115 (DEFAULT_TABLE table=default.Table_6d0035eeace14307bbb048e1f584b0c5 [id=887ea33c2fbe4573a1d7169dfd226567]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.707180 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d672118ecd6f4094855adf106e5f7115. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.708459 10043 raft_consensus.cc:2243] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:07:52.709321 10043 pending_rounds.cc:70] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14: Trying to abort 1 pending ops.
I20251027 21:07:52.709432 10043 pending_rounds.cc:77] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510620904484864 op_type: NO_OP noop_request { }
I20251027 21:07:52.709539 10043 raft_consensus.cc:2889] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:07:52.709645 10043 raft_consensus.cc:2272] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.709975 9820 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:07:52.710727 9594 tablet_service.cc:1505] Processing CreateTablet for tablet fde0980aa64c4496861a662f7ed3ef75 (DEFAULT_TABLE table=default.Table_096b7cf08dbf49279d8d5c11b3b687ab [id=20b788f5a2d14b31a58e1b29fa954951]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.711030 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fde0980aa64c4496861a662f7ed3ef75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.711373 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 9fa39697360c46138ef4cd21e8c7a950 (DEFAULT_TABLE table=default.Table_30780e9f9901401ba8240a70a20ad769 [id=434ed8c436c149ef9c7eb86ad252291d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.711716 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fa39697360c46138ef4cd21e8c7a950. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.717056 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet 10a44aecfade42969454126b990eca24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.719709 9311 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 8fcfa6b85e7b46d78a84947e9f11ae46 (table default.Table_4723d51b934b4f6a90c790ee4bc7ea86 [id=e68b63cf420d49289a5bd1db81e1cbc1]). Expected version 2 got 0
I20251027 21:07:52.720701 9897 tablet_service.cc:1505] Processing CreateTablet for tablet f2710f265ae9485dafec389168328067 (DEFAULT_TABLE table=default.Table_cf6134c7114f4ad5b61f1bda6ffa5015 [id=27d1cff8fc7940c9abb741c8c7d87e24]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.721323 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2710f265ae9485dafec389168328067. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.723443 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_bef3b1992e374f98a2e2ccd86f02ae75" } new_table_name: "default.Table_7a01697cdee04a929068e3297b598128" modify_external_catalogs: true
I20251027 21:07:52.723886 9891 tablet_service.cc:1505] Processing CreateTablet for tablet d672118ecd6f4094855adf106e5f7115 (DEFAULT_TABLE table=default.Table_6d0035eeace14307bbb048e1f584b0c5 [id=887ea33c2fbe4573a1d7169dfd226567]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.724259 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d672118ecd6f4094855adf106e5f7115. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.724220 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet 601c74459d3a4a41b812fdcdf8b0a62c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.724983 9600 tablet_service.cc:1505] Processing CreateTablet for tablet aabaa34787cf4d179822269b7f323314 (DEFAULT_TABLE table=default.Table_0f7c274ead214bb68565a54bc2da4f63 [id=a0a10aefbf4c4ab083982e9e46bba513]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.725384 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aabaa34787cf4d179822269b7f323314. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.729179 9889 tablet_service.cc:1505] Processing CreateTablet for tablet fde0980aa64c4496861a662f7ed3ef75 (DEFAULT_TABLE table=default.Table_096b7cf08dbf49279d8d5c11b3b687ab [id=20b788f5a2d14b31a58e1b29fa954951]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.729485 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fde0980aa64c4496861a662f7ed3ef75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.730065 10041 ts_tablet_manager.cc:1929] T a9cc64ad85334daa9a09538e24ebd8a5 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.735040 10043 ts_tablet_manager.cc:1916] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.739777 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_066fc1d1c0814ddc965fe16fbc04f2d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.740350 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_066fc1d1c0814ddc965fe16fbc04f2d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.741298 9741 tablet_service.cc:1505] Processing CreateTablet for tablet fde0980aa64c4496861a662f7ed3ef75 (DEFAULT_TABLE table=default.Table_096b7cf08dbf49279d8d5c11b3b687ab [id=20b788f5a2d14b31a58e1b29fa954951]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.741672 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fde0980aa64c4496861a662f7ed3ef75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.741616 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f99fba6d68834ef4b426a23a2ca11c9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.742035 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f99fba6d68834ef4b426a23a2ca11c9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:52.744026 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a033551a591146039b017b4193b221ca consensus state has no leader
I20251027 21:07:52.746168 10043 ts_tablet_manager.cc:1929] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:07:52.746413 10043 log.cc:1199] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/cd57440d3b2a4f37a8db7d1fa4bc4dbe
I20251027 21:07:52.747320 10043 ts_tablet_manager.cc:1950] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:52.748481 10043 tablet_replica.cc:333] T 2d659e4d73ea4ff3975a4b58a794090a P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:52.748663 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe (table default.Table_4c5433970bd64291a2b1289a84f35590 [id=a1b2d46db3764c7f97712a0c5f891881]) successfully deleted
I20251027 21:07:52.749775 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3f8272b52c50446086e4ed9f69f4ae82" } modify_external_catalogs: true
I20251027 21:07:52.749948 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3f8272b52c50446086e4ed9f69f4ae82" } modify_external_catalogs: true
I20251027 21:07:52.757685 9307 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 6d5a639b8ccc43bcb16d3c885e1d3725
I20251027 21:07:52.760313 10040 ts_tablet_manager.cc:1929] T 69b1fb42428748239d8939d98428d8fe P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.760459 10040 log.cc:1199] T 69b1fb42428748239d8939d98428d8fe P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/69b1fb42428748239d8939d98428d8fe
I20251027 21:07:52.760831 10040 ts_tablet_manager.cc:1950] T 69b1fb42428748239d8939d98428d8fe P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:52.761942 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 69b1fb42428748239d8939d98428d8fe (table default.Table_5d5f327d728f4250841a9f4f10a2b533 [id=33dbf5fe986c415cb54fbf1b574651ac]) successfully deleted
I20251027 21:07:52.762442 10031 tablet_bootstrap.cc:492] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:52.765368 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 1d64a13667194c12864fa388eb88a380 (DEFAULT_TABLE table=default.Table_d11118d727fb4579a6465bd6deebaa43 [id=9df9c59afc184f9bb7494b154852d943]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.764253 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 1d64a13667194c12864fa388eb88a380 (DEFAULT_TABLE table=default.Table_d11118d727fb4579a6465bd6deebaa43 [id=9df9c59afc184f9bb7494b154852d943]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.764379 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 83f5fbf5757a49df93ab949b7657c61f (DEFAULT_TABLE table=default.Table_762d79ee19da4feb90eca3f718458fc7 [id=2719caa93b0945c59e6ad4d6f1e9e717]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.764864 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 1d64a13667194c12864fa388eb88a380 (DEFAULT_TABLE table=default.Table_d11118d727fb4579a6465bd6deebaa43 [id=9df9c59afc184f9bb7494b154852d943]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.766319 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83f5fbf5757a49df93ab949b7657c61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.766343 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d64a13667194c12864fa388eb88a380. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.766590 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 83f5fbf5757a49df93ab949b7657c61f (DEFAULT_TABLE table=default.Table_762d79ee19da4feb90eca3f718458fc7 [id=2719caa93b0945c59e6ad4d6f1e9e717]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.766732 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d64a13667194c12864fa388eb88a380. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.766944 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83f5fbf5757a49df93ab949b7657c61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.769796 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 9c2d713d72fd4b5ca34cb3e828c75761 (DEFAULT_TABLE table=default.Table_294e3c4092274dc1a146d3f604bc2179 [id=bd8f38a55b9f42b99e372c133a148007]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.770128 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c2d713d72fd4b5ca34cb3e828c75761. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.771497 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 9c2d713d72fd4b5ca34cb3e828c75761 (DEFAULT_TABLE table=default.Table_294e3c4092274dc1a146d3f604bc2179 [id=bd8f38a55b9f42b99e372c133a148007]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.772053 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c2d713d72fd4b5ca34cb3e828c75761. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.772140 10031 ts_tablet_manager.cc:1403] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.107s user 0.005s sys 0.005s
I20251027 21:07:52.774097 9900 tablet_service.cc:1505] Processing CreateTablet for tablet 7b8156d9ab2e4dfba383ddc72137aab4 (DEFAULT_TABLE table=default.Table_d5b5337ae438439d984f34a4e0af6c93 [id=8932a6e05aae478ab0a22e213896851f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.774492 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b8156d9ab2e4dfba383ddc72137aab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.765791 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d64a13667194c12864fa388eb88a380. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.770630 10029 tablet_bootstrap.cc:492] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:52.773468 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 9c2d713d72fd4b5ca34cb3e828c75761 (DEFAULT_TABLE table=default.Table_294e3c4092274dc1a146d3f604bc2179 [id=bd8f38a55b9f42b99e372c133a148007]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.775503 10029 ts_tablet_manager.cc:1403] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.109s user 0.040s sys 0.031s
I20251027 21:07:52.774132 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 9861a0efcba64141aae450663c5d83de (DEFAULT_TABLE table=default.Table_a8906ade8f93405bb76d1ca9b7a0afa6 [id=a0c9cfaafd6c46dfb9a999f25f8977fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.775868 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9861a0efcba64141aae450663c5d83de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.776324 10029 raft_consensus.cc:359] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.776506 10029 raft_consensus.cc:385] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.776599 10029 raft_consensus.cc:740] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.776876 10029 consensus_queue.cc:260] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.777347 10029 ts_tablet_manager.cc:1434] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 21:07:52.777647 10029 tablet_bootstrap.cc:492] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:52.778337 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c2d713d72fd4b5ca34cb3e828c75761. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.778415 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 9861a0efcba64141aae450663c5d83de (DEFAULT_TABLE table=default.Table_a8906ade8f93405bb76d1ca9b7a0afa6 [id=a0c9cfaafd6c46dfb9a999f25f8977fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.778769 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9861a0efcba64141aae450663c5d83de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.774590 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 83f5fbf5757a49df93ab949b7657c61f (DEFAULT_TABLE table=default.Table_762d79ee19da4feb90eca3f718458fc7 [id=2719caa93b0945c59e6ad4d6f1e9e717]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.775193 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 7b8156d9ab2e4dfba383ddc72137aab4 (DEFAULT_TABLE table=default.Table_d5b5337ae438439d984f34a4e0af6c93 [id=8932a6e05aae478ab0a22e213896851f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.784741 10029 tablet_bootstrap.cc:654] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.784837 9743 tablet_service.cc:1505] Processing CreateTablet for tablet 9861a0efcba64141aae450663c5d83de (DEFAULT_TABLE table=default.Table_a8906ade8f93405bb76d1ca9b7a0afa6 [id=a0c9cfaafd6c46dfb9a999f25f8977fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.785152 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet f6249fc5c6174e8e81b9d02d598d5ebd with delete_type TABLET_DATA_DELETED (replaced by tablet 7d8825b80cec4ba68da077ca3b8d89db) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.785224 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9861a0efcba64141aae450663c5d83de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.786337 10371 raft_consensus.cc:493] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:52.786540 10371 raft_consensus.cc:515] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.787358 10371 leader_election.cc:290] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:52.793355 10032 tablet_bootstrap.cc:492] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:52.793937 10032 ts_tablet_manager.cc:1403] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.151s user 0.008s sys 0.002s
I20251027 21:07:52.795437 10032 raft_consensus.cc:359] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.795744 10032 raft_consensus.cc:385] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.795884 10032 raft_consensus.cc:740] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.796182 10032 consensus_queue.cc:260] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.796931 10032 ts_tablet_manager.cc:1434] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:07:52.797308 10032 tablet_bootstrap.cc:492] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:52.798566 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet e841c54a94d649c19b639f5021f150cd from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:52.800621 10032 tablet_bootstrap.cc:654] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.766001 10043 raft_consensus.cc:2243] T 2d659e4d73ea4ff3975a4b58a794090a P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.805503 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b8156d9ab2e4dfba383ddc72137aab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.807310 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e5f5b57f5d948f3b2352451ccaacb09" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:52.807595 10312 raft_consensus.cc:493] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:52.808636 10312 raft_consensus.cc:515] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.809259 10358 raft_consensus.cc:493] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:52.810047 10312 leader_election.cc:290] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:52.810034 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet 24ec2e0ba1994ab59c53977583bd2058 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:52.810601 10043 raft_consensus.cc:2272] T 2d659e4d73ea4ff3975a4b58a794090a P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.811527 10040 tablet_replica.cc:333] T b1bdb2be5032415187c2826c14e75f07 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:52.812554 10040 raft_consensus.cc:2243] T b1bdb2be5032415187c2826c14e75f07 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.811897 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e5f5b57f5d948f3b2352451ccaacb09" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:52.812227 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 7b8156d9ab2e4dfba383ddc72137aab4 (DEFAULT_TABLE table=default.Table_d5b5337ae438439d984f34a4e0af6c93 [id=8932a6e05aae478ab0a22e213896851f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.812292 9749 tablet_service.cc:1505] Processing CreateTablet for tablet d672118ecd6f4094855adf106e5f7115 (DEFAULT_TABLE table=default.Table_6d0035eeace14307bbb048e1f584b0c5 [id=887ea33c2fbe4573a1d7169dfd226567]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.811085 10358 raft_consensus.cc:515] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.805579 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 9fa39697360c46138ef4cd21e8c7a950 (DEFAULT_TABLE table=default.Table_30780e9f9901401ba8240a70a20ad769 [id=434ed8c436c149ef9c7eb86ad252291d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.800684 10041 log.cc:1199] T a9cc64ad85334daa9a09538e24ebd8a5 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/a9cc64ad85334daa9a09538e24ebd8a5
W20251027 21:07:52.807917 9860 leader_election.cc:343] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:52.817211 10043 ts_tablet_manager.cc:1916] T 2d659e4d73ea4ff3975a4b58a794090a P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:52.817826 9859 leader_election.cc:343] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:52.817945 9859 leader_election.cc:304] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:52.818229 10371 raft_consensus.cc:2749] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:52.818923 10041 ts_tablet_manager.cc:1950] T a9cc64ad85334daa9a09538e24ebd8a5 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:52.821453 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_cf6134c7114f4ad5b61f1bda6ffa5015" } new_table_name: "default.Table_8840cffd9b454f6ba433d8bc28a3dd81" modify_external_catalogs: true
I20251027 21:07:52.821774 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7392db44fc434fd1a795922799aaf639"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.823750 9568 leader_election.cc:343] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 970fbf01a4c3406dafe1ea4cd005ea23
I20251027 21:07:52.824188 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7153e01438c54500a2e518e4c5ea51cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:52.824311 10358 leader_election.cc:290] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:07:52.826901 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2710f265ae9485dafec389168328067 consensus state has no leader
I20251027 21:07:52.828466 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_de808e6797d54833a87f262e64dcd576"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:52.829329 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d4fb67aef18b41048fd4f61f7f1f2ebd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:52.825227 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet a9cc64ad85334daa9a09538e24ebd8a5 (table default.Table_03046ffdd6ca47e8aa4cde13574dc119 [id=8a893ef2396748f08f737fddfb76fd65]) successfully deleted
W20251027 21:07:52.831396 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7153e01438c54500a2e518e4c5ea51cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.829083 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83f5fbf5757a49df93ab949b7657c61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.832288 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fa39697360c46138ef4cd21e8c7a950. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:52.832608 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de808e6797d54833a87f262e64dcd576 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:52.833891 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7392db44fc434fd1a795922799aaf639 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:52.829169 9568 leader_election.cc:343] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 81e914ecc3c74b869e9fc3ad102887ec
I20251027 21:07:52.835842 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_274a6d6e3de14343aac38fb44fe222a2" } new_table_name: "default.Table_318d41ccfb2c4d2c99bafcea875143a1" modify_external_catalogs: true
I20251027 21:07:52.835988 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7954e74d079b416cac73f9a77979c8b3" } modify_external_catalogs: true
I20251027 21:07:52.836191 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7954e74d079b416cac73f9a77979c8b3" } modify_external_catalogs: true
I20251027 21:07:52.836715 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c3c17096bae64f378a84096c3b6f5ab5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.837170 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3c17096bae64f378a84096c3b6f5ab5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:52.839180 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4fb67aef18b41048fd4f61f7f1f2ebd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:52.844434 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7f7ad06a15c04efbb701634e14904351 consensus state has no leader
I20251027 21:07:52.845844 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_660eb77e69884fe5bd5e65b47ea91dde" } modify_external_catalogs: true
I20251027 21:07:52.846010 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_660eb77e69884fe5bd5e65b47ea91dde" } modify_external_catalogs: true
I20251027 21:07:52.848497 10041 tablet_replica.cc:333] T 779b7f63b0fa4f9ab47cf5799b3e085d P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:52.848760 10041 raft_consensus.cc:2243] T 779b7f63b0fa4f9ab47cf5799b3e085d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.848991 10041 raft_consensus.cc:2272] T 779b7f63b0fa4f9ab47cf5799b3e085d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.850260 10041 ts_tablet_manager.cc:1916] T 779b7f63b0fa4f9ab47cf5799b3e085d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.857197 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_cfb36d57b28d49f88f50a05650a40b49" } modify_external_catalogs: true
I20251027 21:07:52.857407 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_cfb36d57b28d49f88f50a05650a40b49" } modify_external_catalogs: true
I20251027 21:07:52.858897 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ab9ae94ee52942d588d846e58da2e533" } modify_external_catalogs: true
I20251027 21:07:52.859061 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ab9ae94ee52942d588d846e58da2e533" } modify_external_catalogs: true
I20251027 21:07:52.859555 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b8156d9ab2e4dfba383ddc72137aab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.859818 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d672118ecd6f4094855adf106e5f7115. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.860227 10388 raft_consensus.cc:493] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:52.861382 10040 raft_consensus.cc:2272] T b1bdb2be5032415187c2826c14e75f07 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.861835 10388 raft_consensus.cc:515] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.862370 10043 ts_tablet_manager.cc:1929] T 2d659e4d73ea4ff3975a4b58a794090a P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.862516 10029 tablet_bootstrap.cc:492] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:52.862530 10388 leader_election.cc:290] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:52.862689 10029 ts_tablet_manager.cc:1403] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.085s user 0.006s sys 0.002s
I20251027 21:07:52.863780 10029 raft_consensus.cc:359] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.864063 10029 raft_consensus.cc:385] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.864151 10029 raft_consensus.cc:740] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.864439 10029 consensus_queue.cc:260] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.866117 10029 ts_tablet_manager.cc:1434] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:07:52.866716 10029 tablet_bootstrap.cc:492] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:52.867412 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c73837b0d9514a8980a8a70c61ae8019" } new_table_name: "default.Table_88e8bdcff395438e9684379b653b965e" modify_external_catalogs: true
I20251027 21:07:52.862524 10043 log.cc:1199] T 2d659e4d73ea4ff3975a4b58a794090a P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/2d659e4d73ea4ff3975a4b58a794090a
I20251027 21:07:52.869714 10043 ts_tablet_manager.cc:1950] T 2d659e4d73ea4ff3975a4b58a794090a P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:52.870393 9682 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:07:52.871692 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 2d659e4d73ea4ff3975a4b58a794090a (table default.Table_e9fd9bee6a2a40d7ad92d2d31b03afce [id=af6cb904f8cd4a929be7a978f9bc2001]) successfully deleted
I20251027 21:07:52.872035 10043 tablet_replica.cc:333] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:52.872220 10043 raft_consensus.cc:2243] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.872403 10043 raft_consensus.cc:2272] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.873714 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_757ff3ea762e447ba76041b257234644" } new_table_name: "default.Table_f08509866aae44609f26ce5e30963f6c" modify_external_catalogs: true
I20251027 21:07:52.874320 10029 tablet_bootstrap.cc:654] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:52.875564 9711 leader_election.cc:343] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 35e5b7bbab7f4ee9bbf23d913c0aee76
W20251027 21:07:52.878926 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 84b9de82d55447bba2a897edb683e5c5 consensus state has no leader
I20251027 21:07:52.879317 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6845351c1f1949a9932e39a805cc4033"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.879755 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6845351c1f1949a9932e39a805cc4033 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.880473 10032 tablet_bootstrap.cc:492] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:52.880645 10032 ts_tablet_manager.cc:1403] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.002s
I20251027 21:07:52.881409 10032 raft_consensus.cc:359] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.881659 10032 raft_consensus.cc:385] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.881762 10032 raft_consensus.cc:740] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.882042 10032 consensus_queue.cc:260] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.882702 10032 ts_tablet_manager.cc:1434] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:52.883060 10032 tablet_bootstrap.cc:492] T 9248f458baca409aa3505cc0db321b34 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:52.884608 10032 tablet_bootstrap.cc:654] T 9248f458baca409aa3505cc0db321b34 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:52.885051 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a4aa1fe05114c1287ca580c1f411d3f consensus state has no leader
I20251027 21:07:52.885380 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_1b463a3e59984a129fc2824928138b26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.885838 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b463a3e59984a129fc2824928138b26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.857264 10041 ts_tablet_manager.cc:1929] T 779b7f63b0fa4f9ab47cf5799b3e085d P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.890210 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_92242f48713b48e5969d30e05bed6686"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:52.890717 10041 log.cc:1199] T 779b7f63b0fa4f9ab47cf5799b3e085d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/779b7f63b0fa4f9ab47cf5799b3e085d
W20251027 21:07:52.890739 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92242f48713b48e5969d30e05bed6686 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.891088 10041 ts_tablet_manager.cc:1950] T 779b7f63b0fa4f9ab47cf5799b3e085d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:52.892021 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "970fbf01a4c3406dafe1ea4cd005ea23" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:52.893167 10029 tablet_bootstrap.cc:492] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:52.893388 10029 ts_tablet_manager.cc:1403] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.002s
I20251027 21:07:52.893350 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 9fa39697360c46138ef4cd21e8c7a950 (DEFAULT_TABLE table=default.Table_30780e9f9901401ba8240a70a20ad769 [id=434ed8c436c149ef9c7eb86ad252291d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.893716 10040 ts_tablet_manager.cc:1916] T b1bdb2be5032415187c2826c14e75f07 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:52.894181 9710 leader_election.cc:343] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 35e5b7bbab7f4ee9bbf23d913c0aee76
I20251027 21:07:52.894270 10029 raft_consensus.cc:359] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.894445 10029 raft_consensus.cc:385] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.894474 9710 leader_election.cc:304] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:52.894522 10029 raft_consensus.cc:740] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.895013 10388 raft_consensus.cc:2749] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:52.894754 10029 consensus_queue.cc:260] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:52.895315 9566 leader_election.cc:343] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:52.895471 9566 leader_election.cc:304] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:52.895942 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81e914ecc3c74b869e9fc3ad102887ec" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:52.896189 9775 raft_consensus.cc:2468] T 81e914ecc3c74b869e9fc3ad102887ec P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:52.896873 10358 raft_consensus.cc:2749] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:52.897413 10031 raft_consensus.cc:359] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.897583 10031 raft_consensus.cc:385] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.897831 10029 ts_tablet_manager.cc:1434] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251027 21:07:52.898134 9566 leader_election.cc:304] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:07:52.898264 10029 tablet_bootstrap.cc:492] T 019a892bfdd54e62bc3560a341d5c2da P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
W20251027 21:07:52.898429 9566 raft_consensus.cc:2686] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251027 21:07:52.899886 10031 raft_consensus.cc:740] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.900134 10031 consensus_queue.cc:260] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.902493 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fa39697360c46138ef4cd21e8c7a950. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.903100 10029 tablet_bootstrap.cc:654] T 019a892bfdd54e62bc3560a341d5c2da P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.903802 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 779b7f63b0fa4f9ab47cf5799b3e085d (table default.Table_d98a9c0b61d649f2945cb6c199f70a64 [id=6c62d93feb804d6aa1d442f8458b7ef8]) successfully deleted
I20251027 21:07:52.905022 10041 tablet_replica.cc:333] T 77797eea3e9c4f8099dc9d58f116200b P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:52.905377 10041 raft_consensus.cc:2243] T 77797eea3e9c4f8099dc9d58f116200b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:52.905633 10041 raft_consensus.cc:2272] T 77797eea3e9c4f8099dc9d58f116200b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:52.907773 10041 ts_tablet_manager.cc:1916] T 77797eea3e9c4f8099dc9d58f116200b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.911304 10032 tablet_bootstrap.cc:492] T 9248f458baca409aa3505cc0db321b34 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:52.911482 10032 ts_tablet_manager.cc:1403] T 9248f458baca409aa3505cc0db321b34 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20251027 21:07:52.912353 10032 raft_consensus.cc:359] T 9248f458baca409aa3505cc0db321b34 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.912613 10032 raft_consensus.cc:385] T 9248f458baca409aa3505cc0db321b34 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:52.912734 10032 raft_consensus.cc:740] T 9248f458baca409aa3505cc0db321b34 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:52.913056 10032 consensus_queue.cc:260] T 9248f458baca409aa3505cc0db321b34 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:52.913714 10032 ts_tablet_manager.cc:1434] T 9248f458baca409aa3505cc0db321b34 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:52.914027 10032 tablet_bootstrap.cc:492] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:52.919626 10031 ts_tablet_manager.cc:1434] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.128s user 0.002s sys 0.000s
I20251027 21:07:52.927922 10043 ts_tablet_manager.cc:1916] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:52.932320 10031 tablet_bootstrap.cc:492] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:52.940091 10032 tablet_bootstrap.cc:654] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.946223 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ac768571ea3e43e4af7551ee98801793" } new_table_name: "default.Table_5cdd921f5bdf47bab7ec19ac00c6292f" modify_external_catalogs: true
I20251027 21:07:52.947431 10031 tablet_bootstrap.cc:654] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:52.948134 10371 raft_consensus.cc:493] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:52.948292 10371 raft_consensus.cc:515] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.948958 10371 leader_election.cc:290] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:52.949532 10358 raft_consensus.cc:493] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:52.949707 10358 raft_consensus.cc:515] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.950415 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_65b14aa010624a2d84b3d758fc00e92a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:52.950618 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d13d00353b6740b5bed5a153936d2059" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:52.950798 9631 raft_consensus.cc:2468] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 0.
W20251027 21:07:52.950924 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65b14aa010624a2d84b3d758fc00e92a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.951495 9860 leader_election.cc:304] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:07:52.951995 10358 leader_election.cc:290] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:52.952486 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 24ec2e0ba1994ab59c53977583bd2058
I20251027 21:07:52.952694 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d13d00353b6740b5bed5a153936d2059" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:52.952870 9924 raft_consensus.cc:2468] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:52.953042 9311 catalog_manager.cc:6719] ReplaceTablet: tablet 24ec2e0ba1994ab59c53977583bd2058 deleted and replaced by tablet 815e12edcd3847ff8485ba768291a824
I20251027 21:07:52.953312 9568 leader_election.cc:304] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:07:52.953799 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ec2e0ba1994ab59c53977583bd2058 with delete_type TABLET_DATA_DELETED (replaced by tablet 815e12edcd3847ff8485ba768291a824) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.954797 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ec2e0ba1994ab59c53977583bd2058 with delete_type TABLET_DATA_DELETED (replaced by tablet 815e12edcd3847ff8485ba768291a824) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.955078 10371 raft_consensus.cc:2804] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:52.955189 10371 raft_consensus.cc:493] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:52.955278 10371 raft_consensus.cc:3060] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:07:52.956115 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e841c54a94d649c19b639f5021f150cd consensus state has no leader
I20251027 21:07:52.957559 10358 raft_consensus.cc:2804] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:52.957655 10358 raft_consensus.cc:493] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:52.957758 10358 raft_consensus.cc:3060] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:52.958807 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_beb6404fe3a046f6bab89fa4753dfcfc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:52.959460 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_beb6404fe3a046f6bab89fa4753dfcfc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:52.960759 10358 raft_consensus.cc:515] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.962218 10371 raft_consensus.cc:515] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.963176 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d13d00353b6740b5bed5a153936d2059" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:07:52.963423 9631 raft_consensus.cc:2393] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fbd567f929a4f6d8e4d78feef7ed678 in current term 1: Already voted for candidate 51902b6b61ff4d5584ff21896e847e14 in this term.
I20251027 21:07:52.964157 10371 leader_election.cc:290] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:52.964967 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d13d00353b6740b5bed5a153936d2059" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:07:52.965339 9924 raft_consensus.cc:2393] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 51902b6b61ff4d5584ff21896e847e14 in current term 1: Already voted for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in this term.
I20251027 21:07:52.965989 10358 leader_election.cc:290] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:52.966055 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f08509866aae44609f26ce5e30963f6c" } modify_external_catalogs: true
I20251027 21:07:52.966293 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f08509866aae44609f26ce5e30963f6c" } modify_external_catalogs: true
I20251027 21:07:52.971199 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 30ce703650c94c06b93bce976cb5d1fd
I20251027 21:07:52.972368 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 30ce703650c94c06b93bce976cb5d1fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.972817 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 94f2dbf47a5c4458bee54b26f272156e
I20251027 21:07:52.973721 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 94f2dbf47a5c4458bee54b26f272156e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.974512 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 30ce703650c94c06b93bce976cb5d1fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.974612 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet 94f2dbf47a5c4458bee54b26f272156e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.975684 10358 raft_consensus.cc:493] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:52.975891 10358 raft_consensus.cc:515] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.977510 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 9248f458baca409aa3505cc0db321b34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.977619 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d13d00353b6740b5bed5a153936d2059" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:52.977641 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d13d00353b6740b5bed5a153936d2059" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:52.977793 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet e07b8aa0a41241d79254c4dae8738e79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.977975 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d13d00353b6740b5bed5a153936d2059" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:52.978129 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 2a4aa1fe05114c1287ca580c1f411d3f
W20251027 21:07:52.978288 9566 leader_election.cc:343] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:52.978420 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "601c74459d3a4a41b812fdcdf8b0a62c" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:52.978593 9773 raft_consensus.cc:2468] T 601c74459d3a4a41b812fdcdf8b0a62c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:52.979025 9566 leader_election.cc:304] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:52.979321 10312 raft_consensus.cc:2749] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251027 21:07:52.979434 9566 leader_election.cc:304] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:07:52.979594 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d13d00353b6740b5bed5a153936d2059" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:52.980495 10312 raft_consensus.cc:2804] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:52.980633 10312 raft_consensus.cc:493] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:52.980726 10312 raft_consensus.cc:3060] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:52.983820 10312 raft_consensus.cc:515] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:52.983963 10043 ts_tablet_manager.cc:1929] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:52.984109 10043 log.cc:1199] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/81e914ecc3c74b869e9fc3ad102887ec
I20251027 21:07:52.984496 10043 ts_tablet_manager.cc:1950] T 81e914ecc3c74b869e9fc3ad102887ec P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:52.985054 10312 leader_election.cc:290] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:07:52.985502 9859 leader_election.cc:343] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:52.985599 9737 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a4aa1fe05114c1287ca580c1f411d3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:52.985615 9568 leader_election.cc:343] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 601c74459d3a4a41b812fdcdf8b0a62c
I20251027 21:07:52.985790 9746 tablet_service.cc:1552] Processing DeleteTablet for tablet 94f2dbf47a5c4458bee54b26f272156e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:52.985857 9859 leader_election.cc:343] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:52.985896 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a4aa1fe05114c1287ca580c1f411d3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.985967 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 30ce703650c94c06b93bce976cb5d1fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.985971 9859 leader_election.cc:304] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:52.986459 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "601c74459d3a4a41b812fdcdf8b0a62c" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:52.987272 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a4aa1fe05114c1287ca580c1f411d3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.987489 9523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 30ce703650c94c06b93bce976cb5d1fd
W20251027 21:07:52.987708 9568 leader_election.cc:343] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 601c74459d3a4a41b812fdcdf8b0a62c
I20251027 21:07:52.988250 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 30ce703650c94c06b93bce976cb5d1fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:52.988431 9771 raft_consensus.cc:3060] T 601c74459d3a4a41b812fdcdf8b0a62c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:52.989097 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 81e914ecc3c74b869e9fc3ad102887ec (table default.Table_a7f999b292d04332b831f454d4aa6743 [id=a6803bcbb3c2427992f31518cc1de040]) successfully deleted
I20251027 21:07:52.989807 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet 30ce703650c94c06b93bce976cb5d1fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:52.989813 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 30ce703650c94c06b93bce976cb5d1fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:52.990562 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 (DEFAULT_TABLE table=default.Table_f99fba6d68834ef4b426a23a2ca11c9e [id=e6e865b72bbc4d01867f19308bdd343b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.990981 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.991542 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 (DEFAULT_TABLE table=default.Table_f99fba6d68834ef4b426a23a2ca11c9e [id=e6e865b72bbc4d01867f19308bdd343b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.992974 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 420ccfaaa7f24ad0aea9fdfea24fc5dd (DEFAULT_TABLE table=default.Table_066fc1d1c0814ddc965fe16fbc04f2d8 [id=733378bb17ab410e808d9310ad24c404]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.993360 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 420ccfaaa7f24ad0aea9fdfea24fc5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:52.996783 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 420ccfaaa7f24ad0aea9fdfea24fc5dd (DEFAULT_TABLE table=default.Table_066fc1d1c0814ddc965fe16fbc04f2d8 [id=733378bb17ab410e808d9310ad24c404]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:52.996898 10371 raft_consensus.cc:2749] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251027 21:07:52.997233 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e841c54a94d649c19b639f5021f150cd
I20251027 21:07:52.997687 9307 catalog_manager.cc:6719] ReplaceTablet: tablet e841c54a94d649c19b639f5021f150cd deleted and replaced by tablet 666842b8e54c4561989fd9401210464b
I20251027 21:07:52.996778 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 (DEFAULT_TABLE table=default.Table_f99fba6d68834ef4b426a23a2ca11c9e [id=e6e865b72bbc4d01867f19308bdd343b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.001503 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet e841c54a94d649c19b639f5021f150cd with delete_type TABLET_DATA_DELETED (replaced by tablet 666842b8e54c4561989fd9401210464b) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.001945 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet e841c54a94d649c19b639f5021f150cd with delete_type TABLET_DATA_DELETED (replaced by tablet 666842b8e54c4561989fd9401210464b) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.002590 10358 leader_election.cc:290] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:53.003217 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet f34512059f0c47fc9af7db2645f86b40
I20251027 21:07:53.003962 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet f6789dc2f7cb4a3c826ac507d5c9a239
I20251027 21:07:53.004366 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet f34512059f0c47fc9af7db2645f86b40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.005846 10041 ts_tablet_manager.cc:1929] T 77797eea3e9c4f8099dc9d58f116200b P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:53.005971 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet f6789dc2f7cb4a3c826ac507d5c9a239 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.006700 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ec2e0ba1994ab59c53977583bd2058 with delete_type TABLET_DATA_DELETED (replaced by tablet 815e12edcd3847ff8485ba768291a824) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.007463 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.008173 9771 raft_consensus.cc:2468] T 601c74459d3a4a41b812fdcdf8b0a62c P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:53.006172 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet f6789dc2f7cb4a3c826ac507d5c9a239 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.006361 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet f34512059f0c47fc9af7db2645f86b40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.008666 9566 leader_election.cc:304] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:07:53.008890 10040 ts_tablet_manager.cc:1929] T b1bdb2be5032415187c2826c14e75f07 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:53.009644 9746 tablet_service.cc:1552] Processing DeleteTablet for tablet f34512059f0c47fc9af7db2645f86b40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.009682 10358 raft_consensus.cc:2804] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:53.009930 10358 raft_consensus.cc:697] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:07:53.010018 9737 tablet_service.cc:1552] Processing DeleteTablet for tablet f6789dc2f7cb4a3c826ac507d5c9a239 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.010784 10358 consensus_queue.cc:237] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.010987 9743 tablet_service.cc:1505] Processing CreateTablet for tablet 420ccfaaa7f24ad0aea9fdfea24fc5dd (DEFAULT_TABLE table=default.Table_066fc1d1c0814ddc965fe16fbc04f2d8 [id=733378bb17ab410e808d9310ad24c404]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.011207 10029 tablet_bootstrap.cc:492] T 019a892bfdd54e62bc3560a341d5c2da P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:53.011348 10029 ts_tablet_manager.cc:1403] T 019a892bfdd54e62bc3560a341d5c2da P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.113s user 0.018s sys 0.012s
I20251027 21:07:53.012209 10029 raft_consensus.cc:359] T 019a892bfdd54e62bc3560a341d5c2da P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.012406 10029 raft_consensus.cc:385] T 019a892bfdd54e62bc3560a341d5c2da P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:53.012437 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet e841c54a94d649c19b639f5021f150cd with delete_type TABLET_DATA_DELETED (replaced by tablet 666842b8e54c4561989fd9401210464b) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.012467 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 420ccfaaa7f24ad0aea9fdfea24fc5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.012486 10029 raft_consensus.cc:740] T 019a892bfdd54e62bc3560a341d5c2da P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:53.012215 10040 log.cc:1199] T b1bdb2be5032415187c2826c14e75f07 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/b1bdb2be5032415187c2826c14e75f07
I20251027 21:07:53.012741 10029 consensus_queue.cc:260] T 019a892bfdd54e62bc3560a341d5c2da P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.012959 9602 tablet_service.cc:1505] Processing CreateTablet for tablet a2d26784bbb24ae3893103802c41affb (DEFAULT_TABLE table=default.Table_aeaed62246864354b04616e1e4461644 [id=54ab773e30384ae98a0b0539284c85b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.013216 10040 ts_tablet_manager.cc:1950] T b1bdb2be5032415187c2826c14e75f07 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:53.013258 10029 ts_tablet_manager.cc:1434] T 019a892bfdd54e62bc3560a341d5c2da P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:53.013612 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 286) took 1107 ms. Trace:
I20251027 21:07:53.013747 10040 rpcz_store.cc:276] 1027 21:07:51.906377 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:51.907308 (+ 931us) service_pool.cc:225] Handling call
1027 21:07:52.806231 (+898923us) ts_tablet_manager.cc:1151] Deleting tablet b1bdb2be5032415187c2826c14e75f07
1027 21:07:52.806243 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:52.917858 (+111615us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:52.983388 (+ 65530us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:53.013564 (+ 30176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":898609}
I20251027 21:07:53.014159 9890 tablet_service.cc:1505] Processing CreateTablet for tablet a2d26784bbb24ae3893103802c41affb (DEFAULT_TABLE table=default.Table_aeaed62246864354b04616e1e4461644 [id=54ab773e30384ae98a0b0539284c85b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.014258 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet b1bdb2be5032415187c2826c14e75f07 (table default.Table_3e0f707f8e2844bfaf937150b747a935 [id=d7dcec1e03494fd5a219a97ea8b1ab1d]) successfully deleted
I20251027 21:07:53.014603 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d26784bbb24ae3893103802c41affb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.014803 10032 tablet_bootstrap.cc:492] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:53.015056 10032 ts_tablet_manager.cc:1403] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.101s user 0.004s sys 0.000s
I20251027 21:07:53.015806 10032 raft_consensus.cc:359] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:53.015982 10032 raft_consensus.cc:385] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:53.016070 10032 raft_consensus.cc:740] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:53.016345 10032 consensus_queue.cc:260] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:53.018090 9600 tablet_service.cc:1505] Processing CreateTablet for tablet f6789dc2f7cb4a3c826ac507d5c9a239 (DEFAULT_TABLE table=default.Table_ab9ae94ee52942d588d846e58da2e533 [id=19f6e48bb56848afb9b5982c825b093e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.018872 10388 raft_consensus.cc:493] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:53.019179 9891 tablet_service.cc:1505] Processing CreateTablet for tablet f6789dc2f7cb4a3c826ac507d5c9a239 (DEFAULT_TABLE table=default.Table_ab9ae94ee52942d588d846e58da2e533 [id=19f6e48bb56848afb9b5982c825b093e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.019340 10388 raft_consensus.cc:515] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:53.019920 10388 leader_election.cc:290] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:53.020066 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 7292bc6c22ca4ab5b8f53a69520833ba (DEFAULT_TABLE table=default.Table_4658a756ae8149cf9e248c65a102bdb6 [id=0a31c72c314e4e5ab0655d9b853c941c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.022589 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet cfce0e5c837743ea969374244163be3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.022729 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 7292bc6c22ca4ab5b8f53a69520833ba (DEFAULT_TABLE table=default.Table_4658a756ae8149cf9e248c65a102bdb6 [id=0a31c72c314e4e5ab0655d9b853c941c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.016906 10032 ts_tablet_manager.cc:1434] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:53.023145 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6789dc2f7cb4a3c826ac507d5c9a239. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.024888 9749 tablet_service.cc:1505] Processing CreateTablet for tablet f6789dc2f7cb4a3c826ac507d5c9a239 (DEFAULT_TABLE table=default.Table_ab9ae94ee52942d588d846e58da2e533 [id=19f6e48bb56848afb9b5982c825b093e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.025426 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6789dc2f7cb4a3c826ac507d5c9a239. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.026433 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 07a9c3b7ef1548ec9d5ae9f20545f263 (DEFAULT_TABLE table=default.Table_2e2c99ff6d264bd2bc52dbb5a6a725b7 [id=4a103f97fc964eaba06cd48d4ba854a6]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:53.026536 9710 leader_election.cc:343] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 36e168cc44274dae9682b0c1329d4558
I20251027 21:07:53.027108 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 287) took 1013 ms. Trace:
I20251027 21:07:53.032909 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_beb6404fe3a046f6bab89fa4753dfcfc" } new_table_name: "default.Table_516d1b7b54d14032a3579f92eea905ea" modify_external_catalogs: true
I20251027 21:07:53.039505 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet 32f80614e30d41128f9fb2e0ea165223 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:53.041460 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet 019a892bfdd54e62bc3560a341d5c2da with delete_type TABLET_DATA_DELETED (replaced by tablet ccaf5c2f8e544842880ba8ba27300ae6) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.041610 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet d89c3c588a1747d7875ebf45924fb890 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.042358 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a9c3b7ef1548ec9d5ae9f20545f263. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:53.047336 9711 leader_election.cc:343] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 36e168cc44274dae9682b0c1329d4558
I20251027 21:07:53.047469 9711 leader_election.cc:304] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:53.047982 10040 rpcz_store.cc:276] 1027 21:07:52.005667 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.005720 (+ 53us) service_pool.cc:225] Handling call
1027 21:07:53.014057 (+1008337us) ts_tablet_manager.cc:1151] Deleting tablet 9b04349b15f3494bb0933db56b7b24bc
1027 21:07:53.014064 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:53.019192 (+ 5128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1008102}
I20251027 21:07:53.049912 10031 tablet_bootstrap.cc:492] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:53.053767 10029 tablet_bootstrap.cc:492] T efc0019840244d14bb95ae581793e74b P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:53.055862 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 420ccfaaa7f24ad0aea9fdfea24fc5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.056888 10041 log.cc:1199] T 77797eea3e9c4f8099dc9d58f116200b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/77797eea3e9c4f8099dc9d58f116200b
W20251027 21:07:53.057561 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet f22f013b25074d4db9c7f13caf5b9b42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f22f013b25074d4db9c7f13caf5b9b42 already in progress: creating tablet
W20251027 21:07:53.057749 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e10ae34eaaf3476594813de8e96c7e77 has no consensus state
W20251027 21:07:53.059425 9297 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet dd9e6656d8df43afbb7f70706bc14585 (table default.Table_62ebc53a16ee4e8ab2022fc3fdbb95b1 [id=3cdf432ec43d42e8bc2d253a92eac8d3], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet dd9e6656d8df43afbb7f70706bc14585 consensus state has no leader
I20251027 21:07:53.060143 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2041587a20dc42f4bef58ce06f47958f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.060782 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2041587a20dc42f4bef58ce06f47958f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.062786 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 32f80614e30d41128f9fb2e0ea165223
I20251027 21:07:53.063192 9303 catalog_manager.cc:6719] ReplaceTablet: tablet 32f80614e30d41128f9fb2e0ea165223 deleted and replaced by tablet a7d8d060916941a5b7e944e8b08a2882
I20251027 21:07:53.063885 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet 32f80614e30d41128f9fb2e0ea165223 with delete_type TABLET_DATA_DELETED (replaced by tablet a7d8d060916941a5b7e944e8b08a2882) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.066500 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.066991 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d26784bbb24ae3893103802c41affb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.067436 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7292bc6c22ca4ab5b8f53a69520833ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.067553 10041 ts_tablet_manager.cc:1950] T 77797eea3e9c4f8099dc9d58f116200b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:53.067870 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6789dc2f7cb4a3c826ac507d5c9a239. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.068637 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 77797eea3e9c4f8099dc9d58f116200b (table default.Table_ed23972576b74b5fbc7167cf7b541bfc [id=e7b7bdb2ea7f468c98bfe88f900a5be8]) successfully deleted
I20251027 21:07:53.069826 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 07a9c3b7ef1548ec9d5ae9f20545f263 (DEFAULT_TABLE table=default.Table_2e2c99ff6d264bd2bc52dbb5a6a725b7 [id=4a103f97fc964eaba06cd48d4ba854a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.070245 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a9c3b7ef1548ec9d5ae9f20545f263. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.072597 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0c6076a969244f39ac709114449efb98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.073124 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c6076a969244f39ac709114449efb98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.074266 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_cd4bceadd9e4409c9ed9dc8365e4ed8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.074723 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd4bceadd9e4409c9ed9dc8365e4ed8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.075947 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4081f3b9db1e4af2a1a8faa1d337461e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.076403 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4081f3b9db1e4af2a1a8faa1d337461e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.065599 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_73d87eb46d0c436db67f3b5c9ccbb8db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.077667 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73d87eb46d0c436db67f3b5c9ccbb8db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.079667 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_caf93d0298314165abb645155ffd1a41" } new_table_name: "default.Table_f343a39b880b4c7a827bff89fd3fe411" modify_external_catalogs: true
I20251027 21:07:53.083456 10388 raft_consensus.cc:2749] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:53.083698 9750 tablet_service.cc:1505] Processing CreateTablet for tablet a2d26784bbb24ae3893103802c41affb (DEFAULT_TABLE table=default.Table_aeaed62246864354b04616e1e4461644 [id=54ab773e30384ae98a0b0539284c85b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.084118 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d26784bbb24ae3893103802c41affb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.084184 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 288) took 1044 ms. Trace:
I20251027 21:07:53.084347 10040 rpcz_store.cc:276] 1027 21:07:52.039597 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.039651 (+ 54us) service_pool.cc:225] Handling call
1027 21:07:53.083993 (+1044342us) ts_tablet_manager.cc:1151] Deleting tablet abc7458fcfb94985acf76d77c0292b08
1027 21:07:53.084005 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:53.084134 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1044156}
I20251027 21:07:53.084738 10040 tablet_replica.cc:333] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:53.085008 10040 raft_consensus.cc:2243] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:53.085196 10040 raft_consensus.cc:2272] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:53.085810 10040 ts_tablet_manager.cc:1916] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:53.088196 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0a8b47a441114631917879e9fe99d612" } new_table_name: "default.Table_c2e56e5918cb4007b681c71977325233" modify_external_catalogs: true
I20251027 21:07:53.088415 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 882c58a45d46402eabe96591a7c1de71 (DEFAULT_TABLE table=default.Table_92dfa85c272446cb910801b5f7050bcf [id=f05fb772c1b44876aba41f0517250a14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.088768 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882c58a45d46402eabe96591a7c1de71. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:53.089275 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 9b04349b15f3494bb0933db56b7b24bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b04349b15f3494bb0933db56b7b24bc already in progress: creating tablet
W20251027 21:07:53.089553 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet abc7458fcfb94985acf76d77c0292b08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abc7458fcfb94985acf76d77c0292b08 already in progress: creating tablet
I20251027 21:07:53.089824 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0d0de750eda84895820447f59e8ab3d3" } new_table_name: "default.Table_1d766c40686340d59cc4680fbf8a2dbb" modify_external_catalogs: true
I20251027 21:07:53.090421 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 882c58a45d46402eabe96591a7c1de71 (DEFAULT_TABLE table=default.Table_92dfa85c272446cb910801b5f7050bcf [id=f05fb772c1b44876aba41f0517250a14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.091089 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882c58a45d46402eabe96591a7c1de71. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:53.099438 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet fe24af9838954a9bb680474800642e62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe24af9838954a9bb680474800642e62 already in progress: creating tablet
I20251027 21:07:53.101094 10358 raft_consensus.cc:493] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:53.101258 10358 raft_consensus.cc:515] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.102546 10358 leader_election.cc:290] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:07:53.102965 9568 leader_election.cc:343] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: f6249fc5c6174e8e81b9d02d598d5ebd
I20251027 21:07:53.103731 10041 tablet_replica.cc:333] T 533f231e3ff64531ae2331409a961425 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:53.103971 10041 raft_consensus.cc:2243] T 533f231e3ff64531ae2331409a961425 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:53.104954 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 588e2f58f07b4a32a78dffd5b92ceb12 (DEFAULT_TABLE table=default.Table_7a3ebf1ecf35428495d2d7bb5540530a [id=c4838f30ee9f4aec96e6c73b4ce15c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.105306 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 588e2f58f07b4a32a78dffd5b92ceb12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.108072 10029 tablet_bootstrap.cc:654] T efc0019840244d14bb95ae581793e74b P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:53.108441 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c6f57d0c3113494cb54754d67cfefa71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:53.108922 9737 tablet_service.cc:1552] Processing DeleteTablet for tablet 32f80614e30d41128f9fb2e0ea165223 with delete_type TABLET_DATA_DELETED (replaced by tablet a7d8d060916941a5b7e944e8b08a2882) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.109057 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 882c58a45d46402eabe96591a7c1de71 (DEFAULT_TABLE table=default.Table_92dfa85c272446cb910801b5f7050bcf [id=f05fb772c1b44876aba41f0517250a14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.109294 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 07a9c3b7ef1548ec9d5ae9f20545f263 (DEFAULT_TABLE table=default.Table_2e2c99ff6d264bd2bc52dbb5a6a725b7 [id=4a103f97fc964eaba06cd48d4ba854a6]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:53.109548 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8fc2d7d6601d4cc2849064f4cdddc90d consensus state has no leader
W20251027 21:07:53.110061 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6f57d0c3113494cb54754d67cfefa71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.110536 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_15d3288990844c4cbd5cfd221a5a7334"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.111097 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15d3288990844c4cbd5cfd221a5a7334 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.111811 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9fa8f9725a2042b2aabd775502550393"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.112227 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9fa8f9725a2042b2aabd775502550393 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.111477 10371 raft_consensus.cc:493] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:53.114369 10371 raft_consensus.cc:515] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:53.114774 10032 tablet_bootstrap.cc:492] T 1776590d8387494da5dfbd08db17f4b1 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:53.115077 10371 leader_election.cc:290] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
W20251027 21:07:53.115693 10371 raft_consensus.cc:670] T 533f231e3ff64531ae2331409a961425 P 0fbd567f929a4f6d8e4d78feef7ed678: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopping
I20251027 21:07:53.116286 10041 raft_consensus.cc:2272] T 533f231e3ff64531ae2331409a961425 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:53.116743 10032 tablet_bootstrap.cc:654] T 1776590d8387494da5dfbd08db17f4b1 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:53.117978 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7292bc6c22ca4ab5b8f53a69520833ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.119400 10041 ts_tablet_manager.cc:1916] T 533f231e3ff64531ae2331409a961425 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:53.120328 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 588e2f58f07b4a32a78dffd5b92ceb12 (DEFAULT_TABLE table=default.Table_7a3ebf1ecf35428495d2d7bb5540530a [id=c4838f30ee9f4aec96e6c73b4ce15c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.122840 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 588e2f58f07b4a32a78dffd5b92ceb12. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:53.128265 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 69da799c886841a19002d3c7c5084e73 consensus state has no leader
I20251027 21:07:53.131201 10388 raft_consensus.cc:493] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:53.132115 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfce0e5c837743ea969374244163be3e" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:53.132344 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet 32f80614e30d41128f9fb2e0ea165223 with delete_type TABLET_DATA_DELETED (replaced by tablet a7d8d060916941a5b7e944e8b08a2882) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:53.134406 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92a2393cff8044028c86c8cd2bccd930 consensus state has no leader
I20251027 21:07:53.137439 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 7292bc6c22ca4ab5b8f53a69520833ba (DEFAULT_TABLE table=default.Table_4658a756ae8149cf9e248c65a102bdb6 [id=0a31c72c314e4e5ab0655d9b853c941c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.138839 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882c58a45d46402eabe96591a7c1de71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.138904 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfce0e5c837743ea969374244163be3e" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:53.141074 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f6249fc5c6174e8e81b9d02d598d5ebd" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:53.141325 10359 raft_consensus.cc:493] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251027 21:07:53.132576 9860 leader_election.cc:343] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:53.142040 10359 raft_consensus.cc:515] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:07:53.142092 9859 leader_election.cc:343] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:53.142587 9859 leader_election.cc:304] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:53.142892 10371 raft_consensus.cc:2749] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:53.144025 10388 raft_consensus.cc:515] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:53.144730 10388 leader_election.cc:290] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:53.145337 10359 leader_election.cc:290] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:53.141767 10031 ts_tablet_manager.cc:1403] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.118s user 0.004s sys 0.007s
I20251027 21:07:53.152462 9970 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
W20251027 21:07:53.154727 9566 leader_election.cc:343] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:53.157303 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 588e2f58f07b4a32a78dffd5b92ceb12 (DEFAULT_TABLE table=default.Table_7a3ebf1ecf35428495d2d7bb5540530a [id=c4838f30ee9f4aec96e6c73b4ce15c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.157922 9308 master_service.cc:954] ReplaceTablet: received request to replace tablet 583fba7c750e4ac7b768da447002c211 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:53.158005 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_516d1b7b54d14032a3579f92eea905ea" } new_table_name: "default.Table_107ad52a34d44600aa08dbe6983ab081" modify_external_catalogs: true
W20251027 21:07:53.158159 9711 leader_election.cc:343] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 33e41145c79b4465b8d564741174b8e0
W20251027 21:07:53.158388 9711 leader_election.cc:343] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: d8ef795c88da49b58082ba1264d83378
I20251027 21:07:53.158857 10031 raft_consensus.cc:359] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.159118 10031 raft_consensus.cc:385] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:53.159281 10031 raft_consensus.cc:740] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:53.159794 10031 consensus_queue.cc:260] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.160774 10031 ts_tablet_manager.cc:1434] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251027 21:07:53.160665 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_048e52e3f32d46a1a3d1619888efc5ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.161206 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_048e52e3f32d46a1a3d1619888efc5ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.162570 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a9c3b7ef1548ec9d5ae9f20545f263. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:53.165470 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e10ae34eaaf3476594813de8e96c7e77 has no consensus state
I20251027 21:07:53.169600 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4a10fa72e7c84bf68191010a19693570"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.170289 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a10fa72e7c84bf68191010a19693570 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.171106 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 588e2f58f07b4a32a78dffd5b92ceb12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.174669 10043 tablet_replica.cc:333] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:53.176348 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7a3ebf1ecf35428495d2d7bb5540530a" } modify_external_catalogs: true
I20251027 21:07:53.176501 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7a3ebf1ecf35428495d2d7bb5540530a" } modify_external_catalogs: true
I20251027 21:07:53.177789 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_dfa3f51576fa4c689b2682b0e277df60" } modify_external_catalogs: true
I20251027 21:07:53.178001 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_dfa3f51576fa4c689b2682b0e277df60" } modify_external_catalogs: true
I20251027 21:07:53.179726 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d276ca9d745f41a58deb7846600ddcc0" } modify_external_catalogs: true
I20251027 21:07:53.179957 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d276ca9d745f41a58deb7846600ddcc0" } modify_external_catalogs: true
I20251027 21:07:53.186690 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 588e2f58f07b4a32a78dffd5b92ceb12
I20251027 21:07:53.187769 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet 588e2f58f07b4a32a78dffd5b92ceb12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.188333 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7292bc6c22ca4ab5b8f53a69520833ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.191149 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet bbb4116d1edd492893adfbf5fb3ddeef
I20251027 21:07:53.191679 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet f4187b2f66f24055b1df170f1062cc88
I20251027 21:07:53.191817 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet bbb4116d1edd492893adfbf5fb3ddeef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.193127 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet f4187b2f66f24055b1df170f1062cc88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.194566 10031 tablet_bootstrap.cc:492] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:53.194825 9743 tablet_service.cc:1552] Processing DeleteTablet for tablet f6249fc5c6174e8e81b9d02d598d5ebd with delete_type TABLET_DATA_DELETED (replaced by tablet 7d8825b80cec4ba68da077ca3b8d89db) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.195604 9741 tablet_service.cc:1552] Processing DeleteTablet for tablet 588e2f58f07b4a32a78dffd5b92ceb12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.195811 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet f4187b2f66f24055b1df170f1062cc88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.196837 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet bbb4116d1edd492893adfbf5fb3ddeef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.199851 9566 leader_election.cc:304] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
W20251027 21:07:53.200230 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 58504.5us
I20251027 21:07:53.202050 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 588e2f58f07b4a32a78dffd5b92ceb12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.202272 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet f4187b2f66f24055b1df170f1062cc88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.203477 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 90f96c21da9944f1ade05d35787d47e2 (DEFAULT_TABLE table=default.Table_3f5ab2fc1a934004816ae27b651671ed [id=ec9aa8523bf44f2f9486b14f4cf7639c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.203905 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90f96c21da9944f1ade05d35787d47e2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:53.204102 9710 leader_election.cc:343] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: d8ef795c88da49b58082ba1264d83378
I20251027 21:07:53.204790 9710 leader_election.cc:304] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
W20251027 21:07:53.205196 9710 leader_election.cc:343] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 33e41145c79b4465b8d564741174b8e0
I20251027 21:07:53.205825 9710 leader_election.cc:304] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:53.206283 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet bbb4116d1edd492893adfbf5fb3ddeef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.207361 10388 raft_consensus.cc:2749] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:53.207347 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d11118d727fb4579a6465bd6deebaa43" } modify_external_catalogs: true
I20251027 21:07:53.207538 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d11118d727fb4579a6465bd6deebaa43" } modify_external_catalogs: true
I20251027 21:07:53.207918 10359 raft_consensus.cc:2749] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:53.212170 10043 raft_consensus.cc:2243] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:53.212369 10043 raft_consensus.cc:2272] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:53.212888 10043 ts_tablet_manager.cc:1916] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:53.218587 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_41078339d0834d069db2e8ae485bcc89" } modify_external_catalogs: true
I20251027 21:07:53.218777 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_41078339d0834d069db2e8ae485bcc89" } modify_external_catalogs: true
I20251027 21:07:53.219373 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet 343eaeca24dc4816ae032c7e674e9412 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:53.220218 10358 raft_consensus.cc:2749] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:53.220461 10359 raft_consensus.cc:493] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:53.221195 10359 raft_consensus.cc:515] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.221829 10359 leader_election.cc:290] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:53.222404 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f6249fc5c6174e8e81b9d02d598d5ebd" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:53.222610 9631 raft_consensus.cc:2468] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:07:53.225123 10043 ts_tablet_manager.cc:1929] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:53.225262 10043 log.cc:1199] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/970fbf01a4c3406dafe1ea4cd005ea23
I20251027 21:07:53.227572 9710 leader_election.cc:304] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:07:53.227634 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 1d64a13667194c12864fa388eb88a380
I20251027 21:07:53.227833 10043 ts_tablet_manager.cc:1950] T 970fbf01a4c3406dafe1ea4cd005ea23 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:53.228451 10359 raft_consensus.cc:2804] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:53.228560 10359 raft_consensus.cc:493] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:53.228617 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d64a13667194c12864fa388eb88a380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.228789 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d64a13667194c12864fa388eb88a380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.230971 10359 raft_consensus.cc:3060] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:53.231233 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet b719dc40d7e348608cdfc78401c19824
I20251027 21:07:53.232036 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet b719dc40d7e348608cdfc78401c19824 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.233273 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet b719dc40d7e348608cdfc78401c19824 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.234091 10029 tablet_bootstrap.cc:492] T efc0019840244d14bb95ae581793e74b P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:53.234314 10029 ts_tablet_manager.cc:1403] T efc0019840244d14bb95ae581793e74b P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.181s user 0.014s sys 0.006s
I20251027 21:07:53.234432 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet b719dc40d7e348608cdfc78401c19824 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.235116 10029 raft_consensus.cc:359] T efc0019840244d14bb95ae581793e74b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.235298 10029 raft_consensus.cc:385] T efc0019840244d14bb95ae581793e74b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:53.235389 10029 raft_consensus.cc:740] T efc0019840244d14bb95ae581793e74b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:53.236045 10359 raft_consensus.cc:515] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.236158 10029 consensus_queue.cc:260] T efc0019840244d14bb95ae581793e74b P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.237696 10032 tablet_bootstrap.cc:492] T 1776590d8387494da5dfbd08db17f4b1 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:53.237893 10032 ts_tablet_manager.cc:1403] T 1776590d8387494da5dfbd08db17f4b1 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.123s user 0.002s sys 0.002s
I20251027 21:07:53.238049 10395 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a4b8b31d47da48aaa386a8e64a5749c2)
I20251027 21:07:53.238242 10340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4b8b31d47da48aaa386a8e64a5749c2. Latest consensus state: current_term: 3 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:53.238512 10340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 21:07:53.238368 10395 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:53.239156 10032 raft_consensus.cc:359] T 1776590d8387494da5dfbd08db17f4b1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:53.239405 10032 raft_consensus.cc:385] T 1776590d8387494da5dfbd08db17f4b1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:53.239535 10032 raft_consensus.cc:740] T 1776590d8387494da5dfbd08db17f4b1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:53.239781 10032 consensus_queue.cc:260] T 1776590d8387494da5dfbd08db17f4b1 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:53.244653 10395 leader_election.cc:290] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a4b8b31d47da48aaa386a8e64a5749c2 (127.9.14.126:34057), 18b15927ce5744ad8ba81dda52725c11 (127.9.14.125:46313)
I20251027 21:07:53.252540 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 90f96c21da9944f1ade05d35787d47e2 (DEFAULT_TABLE table=default.Table_3f5ab2fc1a934004816ae27b651671ed [id=ec9aa8523bf44f2f9486b14f4cf7639c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.253018 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90f96c21da9944f1ade05d35787d47e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.254482 10359 leader_election.cc:290] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:53.255048 10266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53488 (request call id 0) took 1767 ms. Trace:
I20251027 21:07:53.259174 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f6249fc5c6174e8e81b9d02d598d5ebd" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:07:53.259383 9631 raft_consensus.cc:3060] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:53.263883 9631 raft_consensus.cc:2468] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:07:53.264416 9710 leader_election.cc:304] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:07:53.266271 10359 raft_consensus.cc:2804] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:53.266997 10359 raft_consensus.cc:697] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:07:53.267213 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet efc0019840244d14bb95ae581793e74b with delete_type TABLET_DATA_DELETED (replaced by tablet d2203ee62f7e404c95c5f86b5ae85feb) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.267321 10359 consensus_queue.cc:237] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.268187 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 970fbf01a4c3406dafe1ea4cd005ea23 (table default.Table_dddd93d5fb3c4727b64ded3ff6cc3382 [id=741c9ce04c9d47d9a016bce1f4403ec7]) successfully deleted
W20251027 21:07:53.270470 9711 leader_election.cc:343] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: f6249fc5c6174e8e81b9d02d598d5ebd
I20251027 21:07:53.273255 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet 1776590d8387494da5dfbd08db17f4b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 7aa8c538b6c849a28a28ffe706306891) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.274293 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 90f96c21da9944f1ade05d35787d47e2 (DEFAULT_TABLE table=default.Table_3f5ab2fc1a934004816ae27b651671ed [id=ec9aa8523bf44f2f9486b14f4cf7639c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:53.275017 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet fe24af9838954a9bb680474800642e62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe24af9838954a9bb680474800642e62 already in progress: creating tablet
W20251027 21:07:53.275890 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 9c2d713d72fd4b5ca34cb3e828c75761 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c2d713d72fd4b5ca34cb3e828c75761 already in progress: creating tablet
W20251027 21:07:53.277045 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 288b88e13fdf4427aae53f64b87733bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 288b88e13fdf4427aae53f64b87733bf already in progress: creating tablet
I20251027 21:07:53.278108 10358 raft_consensus.cc:493] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:53.278294 10358 raft_consensus.cc:515] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.278993 10358 leader_election.cc:290] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:07:53.279907 9568 leader_election.cc:343] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: d89c3c588a1747d7875ebf45924fb890
I20251027 21:07:53.280403 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d89c3c588a1747d7875ebf45924fb890" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:53.280967 9566 leader_election.cc:343] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:53.281081 9566 leader_election.cc:304] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:53.281540 10358 raft_consensus.cc:2749] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:53.282825 9898 tablet_service.cc:1505] Processing CreateTablet for tablet b3abc5f78c4e41e29128e2d790319c75 (DEFAULT_TABLE table=default.Table_019b51c3ff544dae94c53821dc97ab9f [id=b9930c5b513749ac905895ebca813733]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.283078 9607 tablet_service.cc:1505] Processing CreateTablet for tablet b3abc5f78c4e41e29128e2d790319c75 (DEFAULT_TABLE table=default.Table_019b51c3ff544dae94c53821dc97ab9f [id=b9930c5b513749ac905895ebca813733]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.283228 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3abc5f78c4e41e29128e2d790319c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.283463 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3abc5f78c4e41e29128e2d790319c75. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:53.287945 9711 leader_election.cc:343] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: f6249fc5c6174e8e81b9d02d598d5ebd
I20251027 21:07:53.289541 10031 tablet_bootstrap.cc:654] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:53.289830 10043 tablet_replica.cc:333] T 10a44aecfade42969454126b990eca24 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:53.291178 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90f96c21da9944f1ade05d35787d47e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.291415 10043 raft_consensus.cc:2243] T 10a44aecfade42969454126b990eca24 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:53.294802 9737 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d64a13667194c12864fa388eb88a380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.295245 9750 tablet_service.cc:1505] Processing CreateTablet for tablet b3abc5f78c4e41e29128e2d790319c75 (DEFAULT_TABLE table=default.Table_019b51c3ff544dae94c53821dc97ab9f [id=b9930c5b513749ac905895ebca813733]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.295608 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3abc5f78c4e41e29128e2d790319c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.298502 10032 ts_tablet_manager.cc:1434] T 1776590d8387494da5dfbd08db17f4b1 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.060s user 0.002s sys 0.000s
I20251027 21:07:53.298768 10043 raft_consensus.cc:2272] T 10a44aecfade42969454126b990eca24 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:53.302829 10043 ts_tablet_manager.cc:1916] T 10a44aecfade42969454126b990eca24 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:53.319381 10040 ts_tablet_manager.cc:1929] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:53.319546 10040 log.cc:1199] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/35e5b7bbab7f4ee9bbf23d913c0aee76
I20251027 21:07:53.319864 10040 ts_tablet_manager.cc:1950] T 35e5b7bbab7f4ee9bbf23d913c0aee76 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:53.320210 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 289) took 1271 ms. Trace:
I20251027 21:07:53.320310 10040 rpcz_store.cc:276] 1027 21:07:52.048952 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.049007 (+ 55us) service_pool.cc:225] Handling call
1027 21:07:53.084664 (+1035657us) ts_tablet_manager.cc:1151] Deleting tablet 35e5b7bbab7f4ee9bbf23d913c0aee76
1027 21:07:53.084672 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:53.155311 (+ 70639us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:53.319275 (+163964us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:53.320164 (+ 889us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1035445}
I20251027 21:07:53.320833 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 35e5b7bbab7f4ee9bbf23d913c0aee76 (table default.Table_e68b56d3ddd94f308243ec6a03ea9e6e [id=0a94e88b69094ffe8be3367470be86da]) successfully deleted
I20251027 21:07:53.322677 10411 raft_consensus.cc:493] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:53.324400 10411 raft_consensus.cc:515] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:53.325188 10411 leader_election.cc:290] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:53.325901 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb1686fc7c3a4996aa4462884d30c41f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:53.327873 9859 leader_election.cc:343] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:53.330539 10032 tablet_bootstrap.cc:492] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:53.334416 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 7425b6b0184e4d0b871777b25d904ac4 (DEFAULT_TABLE table=default.Table_d395c65b39c04ef2a3ce8490e8487f47 [id=479548b373ef42b4ad8683a9dcfb3ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.334878 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7425b6b0184e4d0b871777b25d904ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.340199 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2e2c99ff6d264bd2bc52dbb5a6a725b7" } new_table_name: "default.Table_acf5bf3b68d24276bfd6df1b4f2015fd" modify_external_catalogs: true
I20251027 21:07:53.340350 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3f462ba7e3ab4705a1daa311661c4f3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.340974 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f462ba7e3ab4705a1daa311661c4f3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.340317 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a65e330a0b6d42efb7ae68d62db38755"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.342432 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a65e330a0b6d42efb7ae68d62db38755 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.342809 10414 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20251027 21:07:53.344350 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_15d3288990844c4cbd5cfd221a5a7334" } new_table_name: "default.Table_d09f1941d15f4517ac7561a66dd9b5e9" modify_external_catalogs: true
I20251027 21:07:53.344401 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3c1a4867db0849ceb59d5900eb57eb73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.344880 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c1a4867db0849ceb59d5900eb57eb73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.345970 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3dd9e38840f54ce2a2b0c5642eac1425"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.346429 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3dd9e38840f54ce2a2b0c5642eac1425 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.347141 10032 tablet_bootstrap.cc:654] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:53.348438 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_12986d58c67145d69cc562041489fbc4" } new_table_name: "default.Table_1376974f614548189208cb9cf0fd5353" modify_external_catalogs: true
I20251027 21:07:53.349478 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_dac5ed8bed8045668aa636907042bf0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.349860 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dac5ed8bed8045668aa636907042bf0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.350351 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c6c89ff2d21c4e4db430817f6398887e" } new_table_name: "default.Table_87ac731a20bf4f6bbce8d82c1ad3bd97" modify_external_catalogs: true
I20251027 21:07:53.351406 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6ef774fe4a464edc8b6ac91a3011fd41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.351820 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ef774fe4a464edc8b6ac91a3011fd41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.354843 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 290) took 1293 ms. Trace:
I20251027 21:07:53.354959 10040 rpcz_store.cc:276] 1027 21:07:52.061017 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.061121 (+ 104us) service_pool.cc:225] Handling call
1027 21:07:53.354693 (+1293572us) ts_tablet_manager.cc:1151] Deleting tablet a339c0dee0994788b8644912f74c1b82
1027 21:07:53.354703 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:53.354794 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1293339}
I20251027 21:07:53.355353 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 291) took 1293 ms. Trace:
I20251027 21:07:53.355434 10040 rpcz_store.cc:276] 1027 21:07:52.062219 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.062269 (+ 50us) service_pool.cc:225] Handling call
1027 21:07:53.355208 (+1292939us) ts_tablet_manager.cc:1151] Deleting tablet a287439ea6c9469da5fb56d84ee68048
1027 21:07:53.355216 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:53.355306 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1292756}
W20251027 21:07:53.355881 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07a9c3b7ef1548ec9d5ae9f20545f263 consensus state has no leader
W20251027 21:07:53.356407 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet a339c0dee0994788b8644912f74c1b82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a339c0dee0994788b8644912f74c1b82 already in progress: creating tablet
W20251027 21:07:53.356686 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet a287439ea6c9469da5fb56d84ee68048 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a287439ea6c9469da5fb56d84ee68048 already in progress: creating tablet
I20251027 21:07:53.358078 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 67cb71ce43244d8ab9e71ead6bb315c9 (DEFAULT_TABLE table=default.Table_35709e995c574069b4df49b470f87b26 [id=3eb54d245ec64ee48022c541c2e49b17]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.359318 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67cb71ce43244d8ab9e71ead6bb315c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.359867 10254 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "77b0513f17c0472088901a1ce8b08218" instance_seqno: 1761599266252874) as {username='slave'} at 127.9.14.66:55231; Asking this server to re-register.
I20251027 21:07:53.359344 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb1686fc7c3a4996aa4462884d30c41f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:53.360998 9860 leader_election.cc:343] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:53.361164 9860 leader_election.cc:304] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:53.361241 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 67cb71ce43244d8ab9e71ead6bb315c9 (DEFAULT_TABLE table=default.Table_35709e995c574069b4df49b470f87b26 [id=3eb54d245ec64ee48022c541c2e49b17]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.361613 10411 raft_consensus.cc:2749] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:53.364465 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 67cb71ce43244d8ab9e71ead6bb315c9 (DEFAULT_TABLE table=default.Table_35709e995c574069b4df49b470f87b26 [id=3eb54d245ec64ee48022c541c2e49b17]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:53.364945 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9be6b996df8145b9b0df1a2d2dc0ef03 has no consensus state
I20251027 21:07:53.364850 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2d1949f2a17c47748742fda3abdaaada"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.365362 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d1949f2a17c47748742fda3abdaaada in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.365571 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6f3e1c9642e3433eab3f8e5a409470c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.365989 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f3e1c9642e3433eab3f8e5a409470c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.366328 10254 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" instance_seqno: 1761599265362526) as {username='slave'} at 127.9.14.65:56933; Asking this server to re-register.
I20251027 21:07:53.368428 10041 ts_tablet_manager.cc:1929] T 533f231e3ff64531ae2331409a961425 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:53.369529 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0f7c274ead214bb68565a54bc2da4f63" } new_table_name: "default.Table_0eecf2c874144e7594c825f01fcc5770" modify_external_catalogs: true
I20251027 21:07:53.368613 10041 log.cc:1199] T 533f231e3ff64531ae2331409a961425 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/533f231e3ff64531ae2331409a961425
I20251027 21:07:53.371556 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4863860f9b4c419fb46dca7a45e5b3ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:53.372308 9820 heartbeater.cc:461] Registering TS with master...
W20251027 21:07:53.375038 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4863860f9b4c419fb46dca7a45e5b3ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.376780 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 7425b6b0184e4d0b871777b25d904ac4 (DEFAULT_TABLE table=default.Table_d395c65b39c04ef2a3ce8490e8487f47 [id=479548b373ef42b4ad8683a9dcfb3ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.378118 9737 tablet_service.cc:1505] Processing CreateTablet for tablet cb95618d9fd54a4f8121b43e717d5616 (DEFAULT_TABLE table=default.Table_f2775123ffc8464eaf6d5e7d4ee692b0 [id=3298547a152045fdb51d360f4af22108]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:53.378738 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fde4fd91943498d8888d1783005e492 consensus state has no leader
W20251027 21:07:53.379323 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 78884c2be75f45f881f5424d3b3f4e08 consensus state has no leader
I20251027 21:07:53.379902 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67cb71ce43244d8ab9e71ead6bb315c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.382369 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb95618d9fd54a4f8121b43e717d5616. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.383801 10254 ts_manager.cc:194] Registered new tserver with Master: 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:53.385413 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_110dd5476e984ca299e60f37066a8058"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.385869 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_110dd5476e984ca299e60f37066a8058 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:53.387099 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aabaa34787cf4d179822269b7f323314 consensus state has no leader
I20251027 21:07:53.387246 9682 heartbeater.cc:461] Registering TS with master...
I20251027 21:07:53.387976 9605 tablet_service.cc:1505] Processing CreateTablet for tablet cb95618d9fd54a4f8121b43e717d5616 (DEFAULT_TABLE table=default.Table_f2775123ffc8464eaf6d5e7d4ee692b0 [id=3298547a152045fdb51d360f4af22108]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.388343 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb95618d9fd54a4f8121b43e717d5616. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.391076 10029 ts_tablet_manager.cc:1434] T efc0019840244d14bb95ae581793e74b P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.125s user 0.001s sys 0.001s
I20251027 21:07:53.391049 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5cdd921f5bdf47bab7ec19ac00c6292f" } modify_external_catalogs: true
I20251027 21:07:53.391235 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5cdd921f5bdf47bab7ec19ac00c6292f" } modify_external_catalogs: true
I20251027 21:07:53.390370 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 7425b6b0184e4d0b871777b25d904ac4 (DEFAULT_TABLE table=default.Table_d395c65b39c04ef2a3ce8490e8487f47 [id=479548b373ef42b4ad8683a9dcfb3ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.392839 10254 ts_manager.cc:194] Registered new tserver with Master: 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:53.393653 10029 tablet_bootstrap.cc:492] T 2b247829a11349b6835a2f18906e20a9 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:53.393707 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4bd57498972a413ca2bb7de2cd588e8b" } modify_external_catalogs: true
I20251027 21:07:53.393878 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4bd57498972a413ca2bb7de2cd588e8b" } modify_external_catalogs: true
I20251027 21:07:53.393989 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ac66ea1435fe49518c073dfc9b2368c8" } modify_external_catalogs: true
I20251027 21:07:53.394184 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ac66ea1435fe49518c073dfc9b2368c8" } modify_external_catalogs: true
I20251027 21:07:53.397516 9742 tablet_service.cc:1505] Processing CreateTablet for tablet f4187b2f66f24055b1df170f1062cc88 (DEFAULT_TABLE table=default.Table_d276ca9d745f41a58deb7846600ddcc0 [id=a23a34526a1b46db9ce948189242dc02]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.397902 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4187b2f66f24055b1df170f1062cc88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.399358 9614 tablet_service.cc:1505] Processing CreateTablet for tablet f4187b2f66f24055b1df170f1062cc88 (DEFAULT_TABLE table=default.Table_d276ca9d745f41a58deb7846600ddcc0 [id=a23a34526a1b46db9ce948189242dc02]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.378171 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 292) took 1254 ms. Trace:
I20251027 21:07:53.402524 10040 rpcz_store.cc:276] 1027 21:07:52.123737 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.123785 (+ 48us) service_pool.cc:225] Handling call
1027 21:07:53.355621 (+1231836us) ts_tablet_manager.cc:1151] Deleting tablet 5c34c8f1433b40d491ae05e7c6e60ba1
1027 21:07:53.376994 (+ 21373us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:53.378096 (+ 1102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1231651}
W20251027 21:07:53.403019 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 5c34c8f1433b40d491ae05e7c6e60ba1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c34c8f1433b40d491ae05e7c6e60ba1 already in progress: creating tablet
I20251027 21:07:53.403386 10040 tablet_replica.cc:333] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:53.403599 10040 raft_consensus.cc:2243] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:53.403811 10040 raft_consensus.cc:2272] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:53.404409 10040 ts_tablet_manager.cc:1916] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:53.410871 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7425b6b0184e4d0b871777b25d904ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.411221 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4187b2f66f24055b1df170f1062cc88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.412604 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 9ed4f9c61ad54e1292281b30bcd19533 (DEFAULT_TABLE table=default.Table_796614d1dc4148699dfa07316ea8febf [id=5f4093d9346248dba86e86cf2fbacd92]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.412966 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ed4f9c61ad54e1292281b30bcd19533. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.417341 10029 tablet_bootstrap.cc:654] T 2b247829a11349b6835a2f18906e20a9 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:53.423795 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 9ed4f9c61ad54e1292281b30bcd19533 (DEFAULT_TABLE table=default.Table_796614d1dc4148699dfa07316ea8febf [id=5f4093d9346248dba86e86cf2fbacd92]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.424170 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ed4f9c61ad54e1292281b30bcd19533. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.373318 9897 tablet_service.cc:1505] Processing CreateTablet for tablet cb95618d9fd54a4f8121b43e717d5616 (DEFAULT_TABLE table=default.Table_f2775123ffc8464eaf6d5e7d4ee692b0 [id=3298547a152045fdb51d360f4af22108]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.390617 10411 raft_consensus.cc:493] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:53.425184 10411 raft_consensus.cc:515] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.425499 10041 ts_tablet_manager.cc:1950] T 533f231e3ff64531ae2331409a961425 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:53.425885 10411 leader_election.cc:290] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:53.426466 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e07b8aa0a41241d79254c4dae8738e79" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:53.396158 9891 tablet_service.cc:1505] Processing CreateTablet for tablet f4187b2f66f24055b1df170f1062cc88 (DEFAULT_TABLE table=default.Table_d276ca9d745f41a58deb7846600ddcc0 [id=a23a34526a1b46db9ce948189242dc02]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.411531 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 9ed4f9c61ad54e1292281b30bcd19533 (DEFAULT_TABLE table=default.Table_796614d1dc4148699dfa07316ea8febf [id=5f4093d9346248dba86e86cf2fbacd92]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.426841 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e07b8aa0a41241d79254c4dae8738e79" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:53.427356 9859 leader_election.cc:343] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:07:53.427498 9860 leader_election.cc:343] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:53.427605 9860 leader_election.cc:304] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:53.427826 10411 raft_consensus.cc:2749] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:53.429186 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 30ce703650c94c06b93bce976cb5d1fd (DEFAULT_TABLE table=default.Table_cfb36d57b28d49f88f50a05650a40b49 [id=b82c052176d94b1292aa47473d08a68c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.429730 10031 tablet_bootstrap.cc:492] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:53.429782 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 30ce703650c94c06b93bce976cb5d1fd (DEFAULT_TABLE table=default.Table_cfb36d57b28d49f88f50a05650a40b49 [id=b82c052176d94b1292aa47473d08a68c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.429922 10031 ts_tablet_manager.cc:1403] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.263s user 0.001s sys 0.005s
I20251027 21:07:53.430238 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30ce703650c94c06b93bce976cb5d1fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.430799 10031 raft_consensus.cc:359] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.430977 10031 raft_consensus.cc:385] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:53.431073 10031 raft_consensus.cc:740] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:53.431290 10031 consensus_queue.cc:260] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.432322 10031 ts_tablet_manager.cc:1434] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:53.433797 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4723d51b934b4f6a90c790ee4bc7ea86" } modify_external_catalogs: true
W20251027 21:07:53.433969 9710 consensus_peers.cc:597] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 -> Peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Couldn't send request to peer 51902b6b61ff4d5584ff21896e847e14. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d659e4d73ea4ff3975a4b58a794090a. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:07:53.433980 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4723d51b934b4f6a90c790ee4bc7ea86" } modify_external_catalogs: true
I20251027 21:07:53.434661 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_fe308a2645c14e6c8efc7ce330b5add7" } modify_external_catalogs: true
I20251027 21:07:53.434868 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_fe308a2645c14e6c8efc7ce330b5add7" } modify_external_catalogs: true
I20251027 21:07:53.439841 9741 tablet_service.cc:1552] Processing DeleteTablet for tablet 970fbf01a4c3406dafe1ea4cd005ea23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.442461 10032 tablet_bootstrap.cc:492] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:53.442675 10032 ts_tablet_manager.cc:1403] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.112s user 0.003s sys 0.001s
I20251027 21:07:53.444885 10032 raft_consensus.cc:359] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.445158 10032 raft_consensus.cc:385] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:53.445281 10032 raft_consensus.cc:740] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:53.445580 10032 consensus_queue.cc:260] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.446575 10032 ts_tablet_manager.cc:1434] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251027 21:07:53.446766 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet 76e4fb2bea7f4a4696fe9f6bb892bccc from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:53.448928 9307 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 666842b8e54c4561989fd9401210464b
I20251027 21:07:53.451012 10254 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" instance_seqno: 1761599266734167) as {username='slave'} at 127.9.14.67:36727; Asking this server to re-register.
I20251027 21:07:53.452674 9970 heartbeater.cc:461] Registering TS with master...
I20251027 21:07:53.452986 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 986bf148b3e9482fa2fe53bbac54750d (DEFAULT_TABLE table=default.Table_e5aa7dc0a7da4c1cb2887c10710fbdf1 [id=0004aa305b9848a59125a57650d0590b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.454241 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67cb71ce43244d8ab9e71ead6bb315c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.454331 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d8825b80cec4ba68da077ca3b8d89db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.454497 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 4debc3f13f3a42939b6b83f6523aab7c
I20251027 21:07:53.454641 10358 consensus_queue.cc:1048] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:07:53.455502 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 4debc3f13f3a42939b6b83f6523aab7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.456099 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet 4debc3f13f3a42939b6b83f6523aab7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.456977 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 986bf148b3e9482fa2fe53bbac54750d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.458426 10254 ts_manager.cc:194] Registered new tserver with Master: 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:53.460711 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet da26c83976df40328cf9b11734fcfa44
I20251027 21:07:53.461444 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 8fcfa6b85e7b46d78a84947e9f11ae46
I20251027 21:07:53.461452 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet da26c83976df40328cf9b11734fcfa44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.461899 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet da26c83976df40328cf9b11734fcfa44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.462277 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 8fcfa6b85e7b46d78a84947e9f11ae46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.462373 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet 8fcfa6b85e7b46d78a84947e9f11ae46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.464192 9743 tablet_service.cc:1552] Processing DeleteTablet for tablet 8fcfa6b85e7b46d78a84947e9f11ae46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.464193 9741 tablet_service.cc:1552] Processing DeleteTablet for tablet da26c83976df40328cf9b11734fcfa44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.474385 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 533f231e3ff64531ae2331409a961425 (table default.Table_913666abca6044f9aacb48e76118aac5 [id=b45a015e747e4f0db54899a28ab97a7c]) successfully deleted
I20251027 21:07:53.475045 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ed4f9c61ad54e1292281b30bcd19533. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.477012 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 30ce703650c94c06b93bce976cb5d1fd (DEFAULT_TABLE table=default.Table_cfb36d57b28d49f88f50a05650a40b49 [id=b82c052176d94b1292aa47473d08a68c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.477370 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30ce703650c94c06b93bce976cb5d1fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.479487 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30ce703650c94c06b93bce976cb5d1fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.482591 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb95618d9fd54a4f8121b43e717d5616. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.255168 10266 rpcz_store.cc:276] 1027 21:07:51.487770 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:51.487822 (+ 52us) service_pool.cc:225] Handling call
1027 21:07:52.128140 (+640318us) raft_consensus.cc:1471] Updating replica for 311 ops
1027 21:07:52.186606 (+ 58466us) raft_consensus.cc:1534] Early marking committed up to index 38
1027 21:07:52.435175 (+248569us) raft_consensus.cc:1539] Triggering prepare for 311 ops
1027 21:07:52.435529 (+ 354us) write_op.cc:270] Start()
1027 21:07:52.435577 (+ 48us) write_op.cc:276] Timestamp: P: 1761599267118742 usec, L: 0
1027 21:07:52.436416 (+ 839us) write_op.cc:270] Start()
1027 21:07:52.436457 (+ 41us) write_op.cc:276] Timestamp: P: 1761599267119407 usec, L: 0
1027 21:07:52.437156 (+ 699us) write_op.cc:270] Start()
1027 21:07:52.437179 (+ 23us) write_op.cc:276] Timestamp: P: 1761599267120872 usec, L: 0
1027 21:07:52.437619 (+ 440us) write_op.cc:270] Start()
1027 21:07:52.437640 (+ 21us) write_op.cc:276] Timestamp: P: 1761599267122639 usec, L: 0
1027 21:07:52.438461 (+ 821us) write_op.cc:270] Start()
1027 21:07:52.438488 (+ 27us) write_op.cc:276] Timestamp: P: 1761599267157928 usec, L: 0
1027 21:07:52.438965 (+ 477us) write_op.cc:270] Start()
1027 21:07:52.438993 (+ 28us) write_op.cc:276] Timestamp: P: 1761599267158709 usec, L: 0
1027 21:07:52.439487 (+ 494us) write_op.cc:270] Start()
1027 21:07:52.439547 (+ 60us) write_op.cc:276] Timestamp: P: 1761599267187285 usec, L: 0
1027 21:07:52.440040 (+ 493us) write_op.cc:270] Start()
1027 21:07:52.440063 (+ 23us) write_op.cc:276] Timestamp: P: 1761599267188538 usec, L: 0
1027 21:07:52.440556 (+ 493us) write_op.cc:270] Start()
1027 21:07:52.440580 (+ 24us) write_op.cc:276] Timestamp: P: 1761599267190759 usec, L: 0
1027 21:07:52.441067 (+ 487us) write_op.cc:270] Start()
1027 21:07:52.441089 (+ 22us) write_op.cc:276] Timestamp: P: 1761599267191836 usec, L: 0
1027 21:07:52.441620 (+ 531us) write_op.cc:270] Start()
1027 21:07:52.441642 (+ 22us) write_op.cc:276] Timestamp: P: 1761599267201375 usec, L: 0
1027 21:07:52.441819 (+ 177us) write_op.cc:270] Start()
1027 21:07:52.441841 (+ 22us) write_op.cc:276] Timestamp: P: 1761599267211228 usec, L: 0
1027 21:07:52.441992 (+ 151us) write_op.cc:270] Start()
1027 21:07:52.442015 (+ 23us) write_op.cc:276] Timestamp: P: 1761599267212457 usec, L: 0
1027 21:07:52.442194 (+ 179us) write_op.cc:270] Start()
1027 21:07:52.442233 (+ 39us) write_op.cc:276] Timestamp: P: 1761599267222852 usec, L: 0
1027 21:07:52.442392 (+ 159us) write_op.cc:270] Start()
1027 21:07:52.442416 (+ 24us) write_op.cc:276] Timestamp: P: 1761599267305970 usec, L: 0
1027 21:07:52.442545 (+ 129us) write_op.cc:270] Start()
1027 21:07:52.442574 (+ 29us) write_op.cc:276] Timestamp: P: 1761599267307338 usec, L: 0
1027 21:07:52.442732 (+ 158us) write_op.cc:270] Start()
1027 21:07:52.442752 (+ 20us) write_op.cc:276] Timestamp: P: 1761599267308401 usec, L: 0
1027 21:07:52.442914 (+ 162us) write_op.cc:270] Start()
1027 21:07:52.442935 (+ 21us) write_op.cc:276] Timestamp: P: 1761599267309076 usec, L: 0
1027 21:07:52.443065 (+ 130us) write_op.cc:270] Start()
1027 21:07:52.443089 (+ 24us) write_op.cc:276] Timestamp: P: 1761599267309609 usec, L: 0
1027 21:07:52.443237 (+ 148us) write_op.cc:270] Start()
1027 21:07:52.443262 (+ 25us) write_op.cc:276] Timestamp: P: 1761599267310081 usec, L: 0
1027 21:07:52.443410 (+ 148us) write_op.cc:270] Start()
1027 21:07:52.443430 (+ 20us) write_op.cc:276] Timestamp: P: 1761599267336713 usec, L: 0
1027 21:07:52.443549 (+ 119us) write_op.cc:270] Start()
1027 21:07:52.443567 (+ 18us) write_op.cc:276] Timestamp: P: 1761599267338058 usec, L: 0
1027 21:07:52.443691 (+ 124us) write_op.cc:270] Start()
1027 21:07:52.443708 (+ 17us) write_op.cc:276] Timestamp: P: 1761599267339200 usec, L: 0
1027 21:07:52.443818 (+ 110us) write_op.cc:270] Start()
1027 21:07:52.443837 (+ 19us) write_op.cc:276] Timestamp: P: 1761599267340273 usec, L: 0
1027 21:07:52.444504 (+ 667us) write_op.cc:270] Start()
1027 21:07:52.444531 (+ 27us) write_op.cc:276] Timestamp: P: 1761599267398612 usec, L: 0
1027 21:07:52.444684 (+ 153us) write_op.cc:270] Start()
1027 21:07:52.444703 (+ 19us) write_op.cc:276] Timestamp: P: 1761599267399261 usec, L: 0
1027 21:07:52.444853 (+ 150us) write_op.cc:270] Start()
1027 21:07:52.444874 (+ 21us) write_op.cc:276] Timestamp: P: 1761599267400164 usec, L: 0
1027 21:07:52.444993 (+ 119us) write_op.cc:270] Start()
1027 21:07:52.445010 (+ 17us) write_op.cc:276] Timestamp: P: 1761599267400698 usec, L: 0
1027 21:07:52.445122 (+ 112us) write_op.cc:270] Start()
1027 21:07:52.445140 (+ 18us) write_op.cc:276] Timestamp: P: 1761599267401710 usec, L: 0
1027 21:07:52.445267 (+ 127us) write_op.cc:270] Start()
1027 21:07:52.445286 (+ 19us) write_op.cc:276] Timestamp: P: 1761599267412548 usec, L: 0
1027 21:07:52.445439 (+ 153us) write_op.cc:270] Start()
1027 21:07:52.445458 (+ 19us) write_op.cc:276] Timestamp: P: 1761599267413341 usec, L: 0
1027 21:07:52.445571 (+ 113us) write_op.cc:270] Start()
1027 21:07:52.445588 (+ 17us) write_op.cc:276] Timestamp: P: 1761599267415151 usec, L: 0
1027 21:07:52.445694 (+ 106us) write_op.cc:270] Start()
1027 21:07:52.445712 (+ 18us) write_op.cc:276] Timestamp: P: 1761599267430211 usec, L: 0
1027 21:07:52.445876 (+ 164us) write_op.cc:270] Start()
1027 21:07:52.445899 (+ 23us) write_op.cc:276] Timestamp: P: 1761599267440631 usec, L: 0
1027 21:07:52.446089 (+ 190us) write_op.cc:270] Start()
1027 21:07:52.446112 (+ 23us) write_op.cc:276] Timestamp: P: 1761599267445940 usec, L: 0
1027 21:07:52.446288 (+ 176us) write_op.cc:270] Start()
1027 21:07:52.446330 (+ 42us) write_op.cc:276] Timestamp: P: 1761599267447796 usec, L: 0
1027 21:07:52.446466 (+ 136us) write_op.cc:270] Start()
1027 21:07:52.446485 (+ 19us) write_op.cc:276] Timestamp: P: 1761599267459314 usec, L: 0
1027 21:07:52.446599 (+ 114us) write_op.cc:270] Start()
1027 21:07:52.446620 (+ 21us) write_op.cc:276] Timestamp: P: 1761599267483603 usec, L: 0
1027 21:07:52.446734 (+ 114us) write_op.cc:270] Start()
1027 21:07:52.446753 (+ 19us) write_op.cc:276] Timestamp: P: 1761599267508472 usec, L: 0
1027 21:07:52.446890 (+ 137us) write_op.cc:270] Start()
1027 21:07:52.446911 (+ 21us) write_op.cc:276] Timestamp: P: 1761599267516451 usec, L: 0
1027 21:07:52.447049 (+ 138us) write_op.cc:270] Start()
1027 21:07:52.447085 (+ 36us) write_op.cc:276] Timestamp: P: 1761599267520764 usec, L: 0
1027 21:07:52.447210 (+ 125us) write_op.cc:270] Start()
1027 21:07:52.447230 (+ 20us) write_op.cc:276] Timestamp: P: 1761599267528660 usec, L: 0
1027 21:07:52.447411 (+ 181us) write_op.cc:270] Start()
1027 21:07:52.447435 (+ 24us) write_op.cc:276] Timestamp: P: 1761599267531525 usec, L: 0
1027 21:07:52.447573 (+ 138us) write_op.cc:270] Start()
1027 21:07:52.447597 (+ 24us) write_op.cc:276] Timestamp: P: 1761599267540290 usec, L: 0
1027 21:07:52.447732 (+ 135us) write_op.cc:270] Start()
1027 21:07:52.447751 (+ 19us) write_op.cc:276] Timestamp: P: 1761599267540812 usec, L: 0
1027 21:07:52.582826 (+135075us) write_op.cc:270] Start()
1027 21:07:52.582911 (+ 85us) write_op.cc:276] Timestamp: P: 1761599267541344 usec, L: 0
1027 21:07:52.583064 (+ 153us) write_op.cc:270] Start()
1027 21:07:52.583085 (+ 21us) write_op.cc:276] Timestamp: P: 1761599267568576 usec, L: 0
1027 21:07:52.583234 (+ 149us) write_op.cc:270] Start()
1027 21:07:52.583254 (+ 20us) write_op.cc:276] Timestamp: P: 1761599267573374 usec, L: 0
1027 21:07:52.583372 (+ 118us) write_op.cc:270] Start()
1027 21:07:52.583393 (+ 21us) write_op.cc:276] Timestamp: P: 1761599267592411 usec, L: 0
1027 21:07:52.583654 (+ 261us) write_op.cc:270] Start()
1027 21:07:52.583676 (+ 22us) write_op.cc:276] Timestamp: P: 1761599267600720 usec, L: 0
1027 21:07:52.583805 (+ 129us) write_op.cc:270] Start()
1027 21:07:52.583825 (+ 20us) write_op.cc:276] Timestamp: P: 1761599267660412 usec, L: 0
1027 21:07:52.583962 (+ 137us) write_op.cc:270] Start()
1027 21:07:52.585663 (+ 1701us) write_op.cc:276] Timestamp: P: 1761599267665875 usec, L: 0
1027 21:07:52.585854 (+ 191us) write_op.cc:270] Start()
1027 21:07:52.585879 (+ 25us) write_op.cc:276] Timestamp: P: 1761599267669904 usec, L: 0
1027 21:07:52.586015 (+ 136us) write_op.cc:270] Start()
1027 21:07:52.586036 (+ 21us) write_op.cc:276] Timestamp: P: 1761599267670444 usec, L: 0
1027 21:07:52.586204 (+ 168us) write_op.cc:270] Start()
1027 21:07:52.586225 (+ 21us) write_op.cc:276] Timestamp: P: 1761599267670929 usec, L: 0
1027 21:07:52.594569 (+ 8344us) write_op.cc:270] Start()
1027 21:07:52.594596 (+ 27us) write_op.cc:276] Timestamp: P: 1761599267671429 usec, L: 0
1027 21:07:52.594743 (+ 147us) write_op.cc:270] Start()
1027 21:07:52.594762 (+ 19us) write_op.cc:276] Timestamp: P: 1761599267671866 usec, L: 0
1027 21:07:52.594922 (+ 160us) write_op.cc:270] Start()
1027 21:07:52.594940 (+ 18us) write_op.cc:276] Timestamp: P: 1761599267677665 usec, L: 0
1027 21:07:52.595061 (+ 121us) write_op.cc:270] Start()
1027 21:07:52.595080 (+ 19us) write_op.cc:276] Timestamp: P: 1761599267678104 usec, L: 0
1027 21:07:52.595204 (+ 124us) write_op.cc:270] Start()
1027 21:07:52.595240 (+ 36us) write_op.cc:276] Timestamp: P: 1761599267700206 usec, L: 0
1027 21:07:52.595382 (+ 142us) write_op.cc:270] Start()
1027 21:07:52.599901 (+ 4519us) write_op.cc:276] Timestamp: P: 1761599267700842 usec, L: 0
1027 21:07:52.600073 (+ 172us) write_op.cc:270] Start()
1027 21:07:52.600095 (+ 22us) write_op.cc:276] Timestamp: P: 1761599267701342 usec, L: 0
1027 21:07:52.600238 (+ 143us) write_op.cc:270] Start()
1027 21:07:52.600266 (+ 28us) write_op.cc:276] Timestamp: P: 1761599267711261 usec, L: 0
1027 21:07:52.600411 (+ 145us) write_op.cc:270] Start()
1027 21:07:52.600431 (+ 20us) write_op.cc:276] Timestamp: P: 1761599267746236 usec, L: 0
1027 21:07:52.605964 (+ 5533us) write_op.cc:270] Start()
1027 21:07:52.605989 (+ 25us) write_op.cc:276] Timestamp: P: 1761599267747832 usec, L: 0
1027 21:07:52.606173 (+ 184us) write_op.cc:270] Start()
1027 21:07:52.606198 (+ 25us) write_op.cc:276] Timestamp: P: 1761599267755346 usec, L: 0
1027 21:07:52.606344 (+ 146us) write_op.cc:270] Start()
1027 21:07:52.606365 (+ 21us) write_op.cc:276] Timestamp: P: 1761599267755880 usec, L: 0
1027 21:07:52.606504 (+ 139us) write_op.cc:270] Start()
1027 21:07:52.606523 (+ 19us) write_op.cc:276] Timestamp: P: 1761599267756364 usec, L: 0
1027 21:07:52.606641 (+ 118us) write_op.cc:270] Start()
1027 21:07:52.606659 (+ 18us) write_op.cc:276] Timestamp: P: 1761599267779026 usec, L: 0
1027 21:07:52.606768 (+ 109us) write_op.cc:270] Start()
1027 21:07:52.606787 (+ 19us) write_op.cc:276] Timestamp: P: 1761599267791751 usec, L: 0
1027 21:07:52.606912 (+ 125us) write_op.cc:270] Start()
1027 21:07:52.606939 (+ 27us) write_op.cc:276] Timestamp: P: 1761599267794910 usec, L: 0
1027 21:07:52.607049 (+ 110us) write_op.cc:270] Start()
1027 21:07:52.607065 (+ 16us) write_op.cc:276] Timestamp: P: 1761599267796818 usec, L: 0
1027 21:07:52.607185 (+ 120us) write_op.cc:270] Start()
1027 21:07:52.607202 (+ 17us) write_op.cc:276] Timestamp: P: 1761599267802714 usec, L: 0
1027 21:07:52.607347 (+ 145us) write_op.cc:270] Start()
1027 21:07:52.607364 (+ 17us) write_op.cc:276] Timestamp: P: 1761599267869188 usec, L: 0
1027 21:07:52.607477 (+ 113us) write_op.cc:270] Start()
1027 21:07:52.607495 (+ 18us) write_op.cc:276] Timestamp: P: 1761599267889780 usec, L: 0
1027 21:07:52.607995 (+ 500us) write_op.cc:270] Start()
1027 21:07:52.608016 (+ 21us) write_op.cc:276] Timestamp: P: 1761599267908869 usec, L: 0
1027 21:07:52.608158 (+ 142us) write_op.cc:270] Start()
1027 21:07:52.608184 (+ 26us) write_op.cc:276] Timestamp: P: 1761599267909510 usec, L: 0
1027 21:07:52.608347 (+ 163us) write_op.cc:270] Start()
1027 21:07:52.608370 (+ 23us) write_op.cc:276] Timestamp: P: 1761599267910032 usec, L: 0
1027 21:07:52.608490 (+ 120us) write_op.cc:270] Start()
1027 21:07:52.608519 (+ 29us) write_op.cc:276] Timestamp: P: 1761599267910568 usec, L: 0
1027 21:07:52.608633 (+ 114us) write_op.cc:270] Start()
1027 21:07:52.608647 (+ 14us) write_op.cc:276] Timestamp: P: 1761599267925025 usec, L: 0
1027 21:07:52.608760 (+ 113us) write_op.cc:270] Start()
1027 21:07:52.608774 (+ 14us) write_op.cc:276] Timestamp: P: 1761599267931815 usec, L: 0
1027 21:07:52.608870 (+ 96us) write_op.cc:270] Start()
1027 21:07:52.608887 (+ 17us) write_op.cc:276] Timestamp: P: 1761599267940923 usec, L: 0
1027 21:07:52.608999 (+ 112us) write_op.cc:270] Start()
1027 21:07:52.609022 (+ 23us) write_op.cc:276] Timestamp: P: 1761599267941758 usec, L: 0
1027 21:07:52.609132 (+ 110us) write_op.cc:270] Start()
1027 21:07:52.609155 (+ 23us) write_op.cc:276] Timestamp: P: 1761599267947290 usec, L: 0
1027 21:07:52.609304 (+ 149us) write_op.cc:270] Start()
1027 21:07:52.609335 (+ 31us) write_op.cc:276] Timestamp: P: 1761599267971690 usec, L: 0
1027 21:07:52.609463 (+ 128us) write_op.cc:270] Start()
1027 21:07:52.609482 (+ 19us) write_op.cc:276] Timestamp: P: 1761599267975525 usec, L: 0
1027 21:07:52.609598 (+ 116us) write_op.cc:270] Start()
1027 21:07:52.609616 (+ 18us) write_op.cc:276] Timestamp: P: 1761599267976146 usec, L: 0
1027 21:07:52.609729 (+ 113us) write_op.cc:270] Start()
1027 21:07:52.609744 (+ 15us) write_op.cc:276] Timestamp: P: 1761599267987485 usec, L: 0
1027 21:07:52.609851 (+ 107us) write_op.cc:270] Start()
1027 21:07:52.609871 (+ 20us) write_op.cc:276] Timestamp: P: 1761599267990380 usec, L: 0
1027 21:07:52.610018 (+ 147us) write_op.cc:270] Start()
1027 21:07:52.610037 (+ 19us) write_op.cc:276] Timestamp: P: 1761599267993493 usec, L: 0
1027 21:07:52.610218 (+ 181us) write_op.cc:270] Start()
1027 21:07:52.610239 (+ 21us) write_op.cc:276] Timestamp: P: 1761599268002716 usec, L: 0
1027 21:07:52.610373 (+ 134us) write_op.cc:270] Start()
1027 21:07:52.610393 (+ 20us) write_op.cc:276] Timestamp: P: 1761599268004655 usec, L: 0
1027 21:07:52.610520 (+ 127us) write_op.cc:270] Start()
1027 21:07:52.610541 (+ 21us) write_op.cc:276] Timestamp: P: 1761599268018115 usec, L: 0
1027 21:07:52.610665 (+ 124us) write_op.cc:270] Start()
1027 21:07:52.610682 (+ 17us) write_op.cc:276] Timestamp: P: 1761599268026506 usec, L: 0
1027 21:07:52.611023 (+ 341us) write_op.cc:270] Start()
1027 21:07:52.611050 (+ 27us) write_op.cc:276] Timestamp: P: 1761599268099075 usec, L: 0
1027 21:07:52.611199 (+ 149us) write_op.cc:270] Start()
1027 21:07:52.636603 (+ 25404us) write_op.cc:276] Timestamp: P: 1761599268099782 usec, L: 0
1027 21:07:52.636814 (+ 211us) write_op.cc:270] Start()
1027 21:07:52.636844 (+ 30us) write_op.cc:276] Timestamp: P: 1761599268160164 usec, L: 0
1027 21:07:52.637000 (+ 156us) write_op.cc:270] Start()
1027 21:07:52.637021 (+ 21us) write_op.cc:276] Timestamp: P: 1761599268173213 usec, L: 0
1027 21:07:52.637168 (+ 147us) write_op.cc:270] Start()
1027 21:07:52.637189 (+ 21us) write_op.cc:276] Timestamp: P: 1761599268201840 usec, L: 0
1027 21:07:52.637329 (+ 140us) write_op.cc:270] Start()
1027 21:07:52.637348 (+ 19us) write_op.cc:276] Timestamp: P: 1761599268210987 usec, L: 0
1027 21:07:52.637472 (+ 124us) write_op.cc:270] Start()
1027 21:07:52.637490 (+ 18us) write_op.cc:276] Timestamp: P: 1761599268211807 usec, L: 0
1027 21:07:52.637602 (+ 112us) write_op.cc:270] Start()
1027 21:07:52.637619 (+ 17us) write_op.cc:276] Timestamp: P: 1761599268218298 usec, L: 0
1027 21:07:52.638297 (+ 678us) write_op.cc:270] Start()
1027 21:07:52.638455 (+ 158us) write_op.cc:276] Timestamp: P: 1761599268223788 usec, L: 0
1027 21:07:52.638613 (+ 158us) write_op.cc:270] Start()
1027 21:07:52.638632 (+ 19us) write_op.cc:276] Timestamp: P: 1761599268224894 usec, L: 0
1027 21:07:52.638749 (+ 117us) write_op.cc:270] Start()
1027 21:07:52.638770 (+ 21us) write_op.cc:276] Timestamp: P: 1761599268225703 usec, L: 0
1027 21:07:52.638905 (+ 135us) write_op.cc:270] Start()
1027 21:07:52.638926 (+ 21us) write_op.cc:276] Timestamp: P: 1761599268226583 usec, L: 0
1027 21:07:52.639052 (+ 126us) write_op.cc:270] Start()
1027 21:07:52.639071 (+ 19us) write_op.cc:276] Timestamp: P: 1761599268227945 usec, L: 0
1027 21:07:52.639192 (+ 121us) write_op.cc:270] Start()
1027 21:07:52.639208 (+ 16us) write_op.cc:276] Timestamp: P: 1761599268229076 usec, L: 0
1027 21:07:52.639329 (+ 121us) write_op.cc:270] Start()
1027 21:07:52.639345 (+ 16us) write_op.cc:276] Timestamp: P: 1761599268253644 usec, L: 0
1027 21:07:52.639447 (+ 102us) write_op.cc:270] Start()
1027 21:07:52.639502 (+ 55us) write_op.cc:276] Timestamp: P: 1761599268267420 usec, L: 0
1027 21:07:52.639631 (+ 129us) write_op.cc:270] Start()
1027 21:07:52.639648 (+ 17us) write_op.cc:276] Timestamp: P: 1761599268300544 usec, L: 0
1027 21:07:52.639758 (+ 110us) write_op.cc:270] Start()
1027 21:07:52.639780 (+ 22us) write_op.cc:276] Timestamp: P: 1761599268307179 usec, L: 0
1027 21:07:52.639912 (+ 132us) write_op.cc:270] Start()
1027 21:07:52.639929 (+ 17us) write_op.cc:276] Timestamp: P: 1761599268307675 usec, L: 0
1027 21:07:52.640060 (+ 131us) write_op.cc:270] Start()
1027 21:07:52.640080 (+ 20us) write_op.cc:276] Timestamp: P: 1761599268308186 usec, L: 0
1027 21:07:52.640206 (+ 126us) write_op.cc:270] Start()
1027 21:07:52.640224 (+ 18us) write_op.cc:276] Timestamp: P: 1761599268317456 usec, L: 0
1027 21:07:52.640361 (+ 137us) write_op.cc:270] Start()
1027 21:07:52.640380 (+ 19us) write_op.cc:276] Timestamp: P: 1761599268318031 usec, L: 0
1027 21:07:52.640492 (+ 112us) write_op.cc:270] Start()
1027 21:07:52.640508 (+ 16us) write_op.cc:276] Timestamp: P: 1761599268318612 usec, L: 0
1027 21:07:52.640645 (+ 137us) write_op.cc:270] Start()
1027 21:07:52.640662 (+ 17us) write_op.cc:276] Timestamp: P: 1761599268319216 usec, L: 0
1027 21:07:52.640770 (+ 108us) write_op.cc:270] Start()
1027 21:07:52.640787 (+ 17us) write_op.cc:276] Timestamp: P: 1761599268319776 usec, L: 0
1027 21:07:52.640903 (+ 116us) write_op.cc:270] Start()
1027 21:07:52.640922 (+ 19us) write_op.cc:276] Timestamp: P: 1761599268409767 usec, L: 0
1027 21:07:52.641044 (+ 122us) write_op.cc:270] Start()
1027 21:07:52.641062 (+ 18us) write_op.cc:276] Timestamp: P: 1761599268410354 usec, L: 0
1027 21:07:52.641186 (+ 124us) write_op.cc:270] Start()
1027 21:07:52.641203 (+ 17us) write_op.cc:276] Timestamp: P: 1761599268421092 usec, L: 0
1027 21:07:52.641338 (+ 135us) write_op.cc:270] Start()
1027 21:07:52.641356 (+ 18us) write_op.cc:276] Timestamp: P: 1761599268425878 usec, L: 0
1027 21:07:52.641470 (+ 114us) write_op.cc:270] Start()
1027 21:07:52.641491 (+ 21us) write_op.cc:276] Timestamp: P: 1761599268426468 usec, L: 0
1027 21:07:52.641635 (+ 144us) write_op.cc:270] Start()
1027 21:07:52.641657 (+ 22us) write_op.cc:276] Timestamp: P: 1761599268427021 usec, L: 0
1027 21:07:52.641790 (+ 133us) write_op.cc:270] Start()
1027 21:07:52.641806 (+ 16us) write_op.cc:276] Timestamp: P: 1761599268427497 usec, L: 0
1027 21:07:52.641908 (+ 102us) write_op.cc:270] Start()
1027 21:07:52.641924 (+ 16us) write_op.cc:276] Timestamp: P: 1761599268430849 usec, L: 0
1027 21:07:52.642032 (+ 108us) write_op.cc:270] Start()
1027 21:07:52.642048 (+ 16us) write_op.cc:276] Timestamp: P: 1761599268444940 usec, L: 0
1027 21:07:52.642192 (+ 144us) write_op.cc:270] Start()
1027 21:07:52.642208 (+ 16us) write_op.cc:276] Timestamp: P: 1761599268450630 usec, L: 0
1027 21:07:52.642344 (+ 136us) write_op.cc:270] Start()
1027 21:07:52.642365 (+ 21us) write_op.cc:276] Timestamp: P: 1761599268456592 usec, L: 0
1027 21:07:52.642507 (+ 142us) write_op.cc:270] Start()
1027 21:07:52.642552 (+ 45us) write_op.cc:276] Timestamp: P: 1761599268461018 usec, L: 0
1027 21:07:52.642692 (+ 140us) write_op.cc:270] Start()
1027 21:07:52.642712 (+ 20us) write_op.cc:276] Timestamp: P: 1761599268470939 usec, L: 0
1027 21:07:52.642837 (+ 125us) write_op.cc:270] Start()
1027 21:07:52.642855 (+ 18us) write_op.cc:276] Timestamp: P: 1761599268471512 usec, L: 0
1027 21:07:52.738252 (+ 95397us) write_op.cc:270] Start()
1027 21:07:52.738294 (+ 42us) write_op.cc:276] Timestamp: P: 1761599268555339 usec, L: 0
1027 21:07:52.744093 (+ 5799us) write_op.cc:270] Start()
1027 21:07:52.744133 (+ 40us) write_op.cc:276] Timestamp: P: 1761599268564367 usec, L: 0
1027 21:07:52.744369 (+ 236us) write_op.cc:270] Start()
1027 21:07:52.744391 (+ 22us) write_op.cc:276] Timestamp: P: 1761599268617559 usec, L: 0
1027 21:07:52.744540 (+ 149us) write_op.cc:270] Start()
1027 21:07:52.744565 (+ 25us) write_op.cc:276] Timestamp: P: 1761599268626367 usec, L: 0
1027 21:07:52.752634 (+ 8069us) write_op.cc:270] Start()
1027 21:07:52.752656 (+ 22us) write_op.cc:276] Timestamp: P: 1761599268629114 usec, L: 0
1027 21:07:52.752779 (+ 123us) write_op.cc:270] Start()
1027 21:07:52.752796 (+ 17us) write_op.cc:276] Timestamp: P: 1761599268629717 usec, L: 0
1027 21:07:52.752895 (+ 99us) write_op.cc:270] Start()
1027 21:07:52.752911 (+ 16us) write_op.cc:276] Timestamp: P: 1761599268637001 usec, L: 0
1027 21:07:52.753007 (+ 96us) write_op.cc:270] Start()
1027 21:07:52.753022 (+ 15us) write_op.cc:276] Timestamp: P: 1761599268637602 usec, L: 0
1027 21:07:52.753184 (+ 162us) write_op.cc:270] Start()
1027 21:07:52.753201 (+ 17us) write_op.cc:276] Timestamp: P: 1761599268646532 usec, L: 0
1027 21:07:52.753328 (+ 127us) write_op.cc:270] Start()
1027 21:07:52.753347 (+ 19us) write_op.cc:276] Timestamp: P: 1761599268648243 usec, L: 0
1027 21:07:52.753515 (+ 168us) write_op.cc:270] Start()
1027 21:07:52.753537 (+ 22us) write_op.cc:276] Timestamp: P: 1761599268649260 usec, L: 0
1027 21:07:52.753682 (+ 145us) write_op.cc:270] Start()
1027 21:07:52.753702 (+ 20us) write_op.cc:276] Timestamp: P: 1761599268650177 usec, L: 0
1027 21:07:52.753835 (+ 133us) write_op.cc:270] Start()
1027 21:07:52.753854 (+ 19us) write_op.cc:276] Timestamp: P: 1761599268694072 usec, L: 0
1027 21:07:52.753992 (+ 138us) write_op.cc:270] Start()
1027 21:07:52.754011 (+ 19us) write_op.cc:276] Timestamp: P: 1761599268697891 usec, L: 0
1027 21:07:52.754194 (+ 183us) write_op.cc:270] Start()
1027 21:07:52.754212 (+ 18us) write_op.cc:276] Timestamp: P: 1761599268722418 usec, L: 0
1027 21:07:52.762704 (+ 8492us) write_op.cc:270] Start()
1027 21:07:52.762732 (+ 28us) write_op.cc:276] Timestamp: P: 1761599268736038 usec, L: 0
1027 21:07:52.762922 (+ 190us) write_op.cc:270] Start()
1027 21:07:52.762945 (+ 23us) write_op.cc:276] Timestamp: P: 1761599268835034 usec, L: 0
1027 21:07:52.763090 (+ 145us) write_op.cc:270] Start()
1027 21:07:52.763110 (+ 20us) write_op.cc:276] Timestamp: P: 1761599268835718 usec, L: 0
1027 21:07:52.763229 (+ 119us) write_op.cc:270] Start()
1027 21:07:52.763246 (+ 17us) write_op.cc:276] Timestamp: P: 1761599268839410 usec, L: 0
1027 21:07:52.763357 (+ 111us) write_op.cc:270] Start()
1027 21:07:52.763376 (+ 19us) write_op.cc:276] Timestamp: P: 1761599268871507 usec, L: 0
1027 21:07:52.763543 (+ 167us) write_op.cc:270] Start()
1027 21:07:52.763566 (+ 23us) write_op.cc:276] Timestamp: P: 1761599268872683 usec, L: 0
1027 21:07:52.763713 (+ 147us) write_op.cc:270] Start()
1027 21:07:52.763737 (+ 24us) write_op.cc:276] Timestamp: P: 1761599268874015 usec, L: 0
1027 21:07:52.763869 (+ 132us) write_op.cc:270] Start()
1027 21:07:52.763895 (+ 26us) write_op.cc:276] Timestamp: P: 1761599268877083 usec, L: 0
1027 21:07:52.764022 (+ 127us) write_op.cc:270] Start()
1027 21:07:52.764041 (+ 19us) write_op.cc:276] Timestamp: P: 1761599268910459 usec, L: 0
1027 21:07:52.764211 (+ 170us) write_op.cc:270] Start()
1027 21:07:52.764230 (+ 19us) write_op.cc:276] Timestamp: P: 1761599268931269 usec, L: 0
1027 21:07:52.764367 (+ 137us) write_op.cc:270] Start()
1027 21:07:52.764386 (+ 19us) write_op.cc:276] Timestamp: P: 1761599268933229 usec, L: 0
1027 21:07:52.764532 (+ 146us) write_op.cc:270] Start()
1027 21:07:52.764551 (+ 19us) write_op.cc:276] Timestamp: P: 1761599268957837 usec, L: 0
1027 21:07:52.764671 (+ 120us) write_op.cc:270] Start()
1027 21:07:52.764688 (+ 17us) write_op.cc:276] Timestamp: P: 1761599268958701 usec, L: 0
1027 21:07:52.764800 (+ 112us) write_op.cc:270] Start()
1027 21:07:52.764819 (+ 19us) write_op.cc:276] Timestamp: P: 1761599268959251 usec, L: 0
1027 21:07:52.764955 (+ 136us) write_op.cc:270] Start()
1027 21:07:52.764972 (+ 17us) write_op.cc:276] Timestamp: P: 1761599268982272 usec, L: 0
1027 21:07:52.765088 (+ 116us) write_op.cc:270] Start()
1027 21:07:52.765106 (+ 18us) write_op.cc:276] Timestamp: P: 1761599269001878 usec, L: 0
1027 21:07:52.765235 (+ 129us) write_op.cc:270] Start()
1027 21:07:52.765254 (+ 19us) write_op.cc:276] Timestamp: P: 1761599269007317 usec, L: 0
1027 21:07:52.765386 (+ 132us) write_op.cc:270] Start()
1027 21:07:52.765408 (+ 22us) write_op.cc:276] Timestamp: P: 1761599269018830 usec, L: 0
1027 21:07:52.765559 (+ 151us) write_op.cc:270] Start()
1027 21:07:52.765651 (+ 92us) write_op.cc:276] Timestamp: P: 1761599269021871 usec, L: 0
1027 21:07:52.815005 (+ 49354us) write_op.cc:270] Start()
1027 21:07:52.815081 (+ 76us) write_op.cc:276] Timestamp: P: 1761599269022573 usec, L: 0
1027 21:07:52.815263 (+ 182us) write_op.cc:270] Start()
1027 21:07:52.815288 (+ 25us) write_op.cc:276] Timestamp: P: 1761599269026205 usec, L: 0
1027 21:07:52.815453 (+ 165us) write_op.cc:270] Start()
1027 21:07:52.815477 (+ 24us) write_op.cc:276] Timestamp: P: 1761599269027970 usec, L: 0
1027 21:07:52.815623 (+ 146us) write_op.cc:270] Start()
1027 21:07:52.815647 (+ 24us) write_op.cc:276] Timestamp: P: 1761599269028539 usec, L: 0
1027 21:07:52.815780 (+ 133us) write_op.cc:270] Start()
1027 21:07:52.815802 (+ 22us) write_op.cc:276] Timestamp: P: 1761599269029114 usec, L: 0
1027 21:07:52.815936 (+ 134us) write_op.cc:270] Start()
1027 21:07:52.815958 (+ 22us) write_op.cc:276] Timestamp: P: 1761599269030793 usec, L: 0
1027 21:07:52.816115 (+ 157us) write_op.cc:270] Start()
1027 21:07:52.816152 (+ 37us) write_op.cc:276] Timestamp: P: 1761599269032312 usec, L: 0
1027 21:07:52.816310 (+ 158us) write_op.cc:270] Start()
1027 21:07:52.816334 (+ 24us) write_op.cc:276] Timestamp: P: 1761599269032870 usec, L: 0
1027 21:07:52.816503 (+ 169us) write_op.cc:270] Start()
1027 21:07:52.816524 (+ 21us) write_op.cc:276] Timestamp: P: 1761599269035778 usec, L: 0
1027 21:07:52.816664 (+ 140us) write_op.cc:270] Start()
1027 21:07:52.816686 (+ 22us) write_op.cc:276] Timestamp: P: 1761599269075132 usec, L: 0
1027 21:07:52.850892 (+ 34206us) write_op.cc:270] Start()
1027 21:07:52.850958 (+ 66us) write_op.cc:276] Timestamp: P: 1761599269075750 usec, L: 0
1027 21:07:52.852728 (+ 1770us) write_op.cc:270] Start()
1027 21:07:52.852757 (+ 29us) write_op.cc:276] Timestamp: P: 1761599269076354 usec, L: 0
1027 21:07:52.853014 (+ 257us) write_op.cc:270] Start()
1027 21:07:52.853103 (+ 89us) write_op.cc:276] Timestamp: P: 1761599269080508 usec, L: 0
1027 21:07:52.853317 (+ 214us) write_op.cc:270] Start()
1027 21:07:52.853360 (+ 43us) write_op.cc:276] Timestamp: P: 1761599269101529 usec, L: 0
1027 21:07:52.853605 (+ 245us) write_op.cc:270] Start()
1027 21:07:52.853658 (+ 53us) write_op.cc:276] Timestamp: P: 1761599269106492 usec, L: 0
1027 21:07:52.854075 (+ 417us) write_op.cc:270] Start()
1027 21:07:52.854165 (+ 90us) write_op.cc:276] Timestamp: P: 1761599269159518 usec, L: 0
1027 21:07:52.854434 (+ 269us) write_op.cc:270] Start()
1027 21:07:52.854459 (+ 25us) write_op.cc:276] Timestamp: P: 1761599269166791 usec, L: 0
1027 21:07:52.855194 (+ 735us) write_op.cc:270] Start()
1027 21:07:52.855220 (+ 26us) write_op.cc:276] Timestamp: P: 1761599269182030 usec, L: 0
1027 21:07:52.856135 (+ 915us) write_op.cc:270] Start()
1027 21:07:52.856268 (+ 133us) write_op.cc:276] Timestamp: P: 1761599269204045 usec, L: 0
1027 21:07:52.858304 (+ 2036us) write_op.cc:270] Start()
1027 21:07:52.858443 (+ 139us) write_op.cc:276] Timestamp: P: 1761599269211216 usec, L: 0
1027 21:07:52.860317 (+ 1874us) write_op.cc:270] Start()
1027 21:07:52.860342 (+ 25us) write_op.cc:276] Timestamp: P: 1761599269243111 usec, L: 0
1027 21:07:52.860482 (+ 140us) write_op.cc:270] Start()
1027 21:07:52.860501 (+ 19us) write_op.cc:276] Timestamp: P: 1761599269275171 usec, L: 0
1027 21:07:52.860611 (+ 110us) write_op.cc:270] Start()
1027 21:07:52.860626 (+ 15us) write_op.cc:276] Timestamp: P: 1761599269276537 usec, L: 0
1027 21:07:52.860755 (+ 129us) write_op.cc:270] Start()
1027 21:07:52.860774 (+ 19us) write_op.cc:276] Timestamp: P: 1761599269277387 usec, L: 0
1027 21:07:52.876028 (+ 15254us) write_op.cc:270] Start()
1027 21:07:52.876066 (+ 38us) write_op.cc:276] Timestamp: P: 1761599269331456 usec, L: 0
1027 21:07:52.876276 (+ 210us) write_op.cc:270] Start()
1027 21:07:52.876314 (+ 38us) write_op.cc:276] Timestamp: P: 1761599269332715 usec, L: 0
1027 21:07:52.876470 (+ 156us) write_op.cc:270] Start()
1027 21:07:52.876492 (+ 22us) write_op.cc:276] Timestamp: P: 1761599269333884 usec, L: 0
1027 21:07:52.877873 (+ 1381us) write_op.cc:270] Start()
1027 21:07:52.877898 (+ 25us) write_op.cc:276] Timestamp: P: 1761599269343451 usec, L: 0
1027 21:07:52.878061 (+ 163us) write_op.cc:270] Start()
1027 21:07:52.878099 (+ 38us) write_op.cc:276] Timestamp: P: 1761599269345627 usec, L: 0
1027 21:07:52.879094 (+ 995us) write_op.cc:270] Start()
1027 21:07:52.879117 (+ 23us) write_op.cc:276] Timestamp: P: 1761599
I20251027 21:07:53.488533 10031 tablet_bootstrap.cc:492] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:53.489034 10422 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:53.489248 10422 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: This master's current role is: LEADER
I20251027 21:07:53.489529 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4187b2f66f24055b1df170f1062cc88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.490159 10031 tablet_bootstrap.cc:654] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:53.502454 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet adaa4f66bbab491f8030d439d85cc7d1 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:53.503788 10029 tablet_bootstrap.cc:492] T 2b247829a11349b6835a2f18906e20a9 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:53.503985 10029 ts_tablet_manager.cc:1403] T 2b247829a11349b6835a2f18906e20a9 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.113s user 0.002s sys 0.001s
I20251027 21:07:53.504562 10358 raft_consensus.cc:493] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:53.504694 10358 raft_consensus.cc:515] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.505362 10358 leader_election.cc:290] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:53.506611 9923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8fcfa6b85e7b46d78a84947e9f11ae46" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:53.506788 9923 raft_consensus.cc:2468] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:53.509001 9321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" candidate_term: 4 candidate_status { last_received { term: 3 index: 349 } } ignore_live_leader: false dest_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" is_pre_election: true
I20251027 21:07:53.510987 9568 leader_election.cc:304] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:07:53.513160 10358 raft_consensus.cc:2804] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:53.513310 10358 raft_consensus.cc:493] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:53.513402 10358 raft_consensus.cc:3060] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:53.504874 10029 raft_consensus.cc:359] T 2b247829a11349b6835a2f18906e20a9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:53.514025 10029 raft_consensus.cc:385] T 2b247829a11349b6835a2f18906e20a9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:53.514111 10029 raft_consensus.cc:740] T 2b247829a11349b6835a2f18906e20a9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:53.514389 10029 consensus_queue.cc:260] T 2b247829a11349b6835a2f18906e20a9 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:53.516175 10358 raft_consensus.cc:515] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.516638 10043 ts_tablet_manager.cc:1929] T 10a44aecfade42969454126b990eca24 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:53.516747 10043 log.cc:1199] T 10a44aecfade42969454126b990eca24 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/10a44aecfade42969454126b990eca24
I20251027 21:07:53.517673 10043 ts_tablet_manager.cc:1950] T 10a44aecfade42969454126b990eca24 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:53.518253 10358 leader_election.cc:290] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:53.518417 9923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8fcfa6b85e7b46d78a84947e9f11ae46" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:07:53.518603 9923 raft_consensus.cc:3060] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:53.521265 9923 raft_consensus.cc:2468] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:53.521799 9568 leader_election.cc:304] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:07:53.522698 10358 raft_consensus.cc:2804] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:53.522882 10358 raft_consensus.cc:697] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:07:53.523265 10358 consensus_queue.cc:237] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.523631 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 10a44aecfade42969454126b990eca24 (table default.Table_b0f0589285ac4a56b7a2b23ace6248f6 [id=43ebd8ba9e694d17a9def870c95d7aa6]) successfully deleted
I20251027 21:07:53.524693 10031 tablet_bootstrap.cc:492] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:53.524847 10031 ts_tablet_manager.cc:1403] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.003s
I20251027 21:07:53.525591 10031 raft_consensus.cc:359] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.525792 10031 raft_consensus.cc:385] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:53.525856 10031 raft_consensus.cc:740] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:53.526789 10043 tablet_replica.cc:333] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:53.526989 10043 raft_consensus.cc:2243] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:07:53.527205 10043 pending_rounds.cc:70] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14: Trying to abort 1 pending ops.
I20251027 21:07:53.527281 10043 pending_rounds.cc:77] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510622255575040 op_type: NO_OP noop_request { }
I20251027 21:07:53.527372 10043 raft_consensus.cc:2889] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:07:53.527475 10043 raft_consensus.cc:2272] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:53.529237 10043 ts_tablet_manager.cc:1916] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:53.533154 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b247829a11349b6835a2f18906e20a9 with delete_type TABLET_DATA_DELETED (replaced by tablet fde0980aa64c4496861a662f7ed3ef75) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.534196 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 04c579bb008448e9baa841135b6dbc61
I20251027 21:07:53.535005 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 04c579bb008448e9baa841135b6dbc61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.535247 10031 consensus_queue.cc:260] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.535257 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 04c579bb008448e9baa841135b6dbc61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.538899 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 76e4fb2bea7f4a4696fe9f6bb892bccc
I20251027 21:07:53.539295 10031 ts_tablet_manager.cc:1434] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20251027 21:07:53.539686 9303 catalog_manager.cc:6719] ReplaceTablet: tablet 76e4fb2bea7f4a4696fe9f6bb892bccc deleted and replaced by tablet 5cb8ca3a50c148e6b602e949650353f8
I20251027 21:07:53.540343 9391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" candidate_term: 4 candidate_status { last_received { term: 3 index: 349 } } ignore_live_leader: false dest_uuid: "18b15927ce5744ad8ba81dda52725c11" is_pre_election: true
I20251027 21:07:53.540954 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 76e4fb2bea7f4a4696fe9f6bb892bccc with delete_type TABLET_DATA_DELETED (replaced by tablet 5cb8ca3a50c148e6b602e949650353f8) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.541294 10219 leader_election.cc:304] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e86ca5e83dd4df5b72f045e7b787e81; no voters: 18b15927ce5744ad8ba81dda52725c11, a4b8b31d47da48aaa386a8e64a5749c2
I20251027 21:07:53.541440 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 04c579bb008448e9baa841135b6dbc61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.542963 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet 76e4fb2bea7f4a4696fe9f6bb892bccc with delete_type TABLET_DATA_DELETED (replaced by tablet 5cb8ca3a50c148e6b602e949650353f8) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.545465 10032 tablet_bootstrap.cc:492] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:53.547789 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 4debc3f13f3a42939b6b83f6523aab7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.555764 10043 ts_tablet_manager.cc:1929] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:07:53.555879 10043 log.cc:1199] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/601c74459d3a4a41b812fdcdf8b0a62c
I20251027 21:07:53.556347 10043 ts_tablet_manager.cc:1950] T 601c74459d3a4a41b812fdcdf8b0a62c P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:53.557307 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 601c74459d3a4a41b812fdcdf8b0a62c (table default.Table_0ac2c74fb86c48459bad366bd8efccd5 [id=7c97543d81eb4a1fa0c81678857179ba]) successfully deleted
I20251027 21:07:53.558406 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7153e01438c54500a2e518e4c5ea51cf" } new_table_name: "default.Table_53424b483314460bb3b8eb15be3a21ea" modify_external_catalogs: true
I20251027 21:07:53.558616 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_537e495061924a4da35379e43642675a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.559333 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_537e495061924a4da35379e43642675a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:53.561060 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 9c2d713d72fd4b5ca34cb3e828c75761 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c2d713d72fd4b5ca34cb3e828c75761 already in progress: creating tablet
W20251027 21:07:53.562012 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 288b88e13fdf4427aae53f64b87733bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 288b88e13fdf4427aae53f64b87733bf already in progress: creating tablet
W20251027 21:07:53.562624 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 24ec2e0ba1994ab59c53977583bd2058 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24ec2e0ba1994ab59c53977583bd2058 already in progress: creating tablet
W20251027 21:07:53.563163 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 30ce703650c94c06b93bce976cb5d1fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30ce703650c94c06b93bce976cb5d1fd already in progress: creating tablet
W20251027 21:07:53.563715 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 94f2dbf47a5c4458bee54b26f272156e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94f2dbf47a5c4458bee54b26f272156e already in progress: creating tablet
I20251027 21:07:53.565048 10043 tablet_replica.cc:333] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:53.565471 10043 raft_consensus.cc:2243] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:53.566004 10043 raft_consensus.cc:2272] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:53.567893 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b6b952a4a4634e21a51d3dacc8f4431f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.568396 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6b952a4a4634e21a51d3dacc8f4431f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.566764 10043 ts_tablet_manager.cc:1916] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:53.571307 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_62ebc53a16ee4e8ab2022fc3fdbb95b1" } new_table_name: "default.Table_201f11890a6243e3915632a430cc0ab4" modify_external_catalogs: true
I20251027 21:07:53.573588 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_82b8023e5cb34761981a49e164eb36c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.574640 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82b8023e5cb34761981a49e164eb36c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.576596 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6ed444d67ba74d9ea15f80509d94f706"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:53.576785 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 986bf148b3e9482fa2fe53bbac54750d (DEFAULT_TABLE table=default.Table_e5aa7dc0a7da4c1cb2887c10710fbdf1 [id=0004aa305b9848a59125a57650d0590b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:53.577072 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ed444d67ba74d9ea15f80509d94f706 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.577167 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 986bf148b3e9482fa2fe53bbac54750d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.578294 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 76e4fb2bea7f4a4696fe9f6bb892bccc with delete_type TABLET_DATA_DELETED (replaced by tablet 5cb8ca3a50c148e6b602e949650353f8) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.579644 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3e9f6e0f115040639f405946e55a2637"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.580070 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e9f6e0f115040639f405946e55a2637 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.581010 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4a10fa72e7c84bf68191010a19693570" } new_table_name: "default.Table_e763bc3cda90472897f0f798424199b4" modify_external_catalogs: true
I20251027 21:07:53.581429 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f1103102a7544791874c67fa7cadbbbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.581895 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1103102a7544791874c67fa7cadbbbe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.583536 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 986bf148b3e9482fa2fe53bbac54750d (DEFAULT_TABLE table=default.Table_e5aa7dc0a7da4c1cb2887c10710fbdf1 [id=0004aa305b9848a59125a57650d0590b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:53.583761 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 43611e33512b49d58485cf15de788be7 has no consensus state
I20251027 21:07:53.588516 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8fcfa6b85e7b46d78a84947e9f11ae46" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:53.588861 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8fcfa6b85e7b46d78a84947e9f11ae46" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:07:53.592604 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dd9e6656d8df43afbb7f70706bc14585 consensus state has no leader
W20251027 21:07:53.593578 9566 leader_election.cc:343] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:07:53.593911 9566 leader_election.cc:343] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:53.603216 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a449297756a5460aac690849ef0074fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.603724 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a449297756a5460aac690849ef0074fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.603754 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b4d6f7e145e14280ac10fa486bd81385" } new_table_name: "default.Table_e08f0cbb71674a44a697686c1b6effc9" modify_external_catalogs: true
I20251027 21:07:53.604686 10043 ts_tablet_manager.cc:1929] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:53.604826 10043 log.cc:1199] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/f6249fc5c6174e8e81b9d02d598d5ebd
I20251027 21:07:53.604863 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_65b14aa010624a2d84b3d758fc00e92a" } new_table_name: "default.Table_93662efa6bc64f759a615db6db157bab" modify_external_catalogs: true
I20251027 21:07:53.605194 10043 ts_tablet_manager.cc:1950] T f6249fc5c6174e8e81b9d02d598d5ebd P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:53.606160 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e49807b03fc64f669376118d2ee01918"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:53.606240 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet f6249fc5c6174e8e81b9d02d598d5ebd (table default.Table_943417713b3b456ab4ca691d2fb168fd [id=1cb32eea25114c6dbcb3b28044036c08]) successfully deleted
W20251027 21:07:53.606604 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e49807b03fc64f669376118d2ee01918 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:53.612233 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 24ec2e0ba1994ab59c53977583bd2058 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24ec2e0ba1994ab59c53977583bd2058 already in progress: creating tablet
W20251027 21:07:53.613159 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 30ce703650c94c06b93bce976cb5d1fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30ce703650c94c06b93bce976cb5d1fd already in progress: creating tablet
I20251027 21:07:53.613261 9306 master_service.cc:954] ReplaceTablet: received request to replace tablet f22f013b25074d4db9c7f13caf5b9b42 from {username='slave'} at 127.0.0.1:41710
W20251027 21:07:53.613963 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 94f2dbf47a5c4458bee54b26f272156e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94f2dbf47a5c4458bee54b26f272156e already in progress: creating tablet
W20251027 21:07:53.614800 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 2a4aa1fe05114c1287ca580c1f411d3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a4aa1fe05114c1287ca580c1f411d3f already in progress: creating tablet
W20251027 21:07:53.615610 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 30ce703650c94c06b93bce976cb5d1fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30ce703650c94c06b93bce976cb5d1fd already in progress: creating tablet
I20251027 21:07:53.622460 10395 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251027 21:07:53.622650 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet f34512059f0c47fc9af7db2645f86b40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f34512059f0c47fc9af7db2645f86b40 already in progress: creating tablet
I20251027 21:07:53.625078 10043 tablet_replica.cc:333] T 019a892bfdd54e62bc3560a341d5c2da P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
W20251027 21:07:53.625205 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet f6789dc2f7cb4a3c826ac507d5c9a239 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6789dc2f7cb4a3c826ac507d5c9a239 already in progress: creating tablet
I20251027 21:07:53.625710 10043 raft_consensus.cc:2243] T 019a892bfdd54e62bc3560a341d5c2da P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:53.625905 10043 raft_consensus.cc:2272] T 019a892bfdd54e62bc3560a341d5c2da P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:53.626456 10043 ts_tablet_manager.cc:1916] T 019a892bfdd54e62bc3560a341d5c2da P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:53.628455 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5e19def4c4374da7ba3ee687fb55f0a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.628990 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e19def4c4374da7ba3ee687fb55f0a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.636888 10358 consensus_queue.cc:1048] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:07:53.641661 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet a9bf3a093be8498c9dd5511b53436b16 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:53.644481 10043 ts_tablet_manager.cc:1929] T 019a892bfdd54e62bc3560a341d5c2da P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:53.644593 10043 log.cc:1199] T 019a892bfdd54e62bc3560a341d5c2da P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/019a892bfdd54e62bc3560a341d5c2da
I20251027 21:07:53.644961 10043 ts_tablet_manager.cc:1950] T 019a892bfdd54e62bc3560a341d5c2da P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
W20251027 21:07:53.646303 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ba13a89a33549af921d3b55023faf15 has no consensus state
I20251027 21:07:53.651296 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 019a892bfdd54e62bc3560a341d5c2da (table default.Table_9c6b1e43fc5f4f4389620f1ee244db12 [id=2b0d4e085a8e42879c67963231a24b03]) successfully deleted
W20251027 21:07:53.651343 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aaaf70c950044387a63820079db3c829 consensus state has no leader
I20251027 21:07:53.652418 10043 tablet_replica.cc:333] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
W20251027 21:07:53.652784 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 423047dfbf084bfcb881efeb0352f88d has no consensus state
I20251027 21:07:53.653066 10043 raft_consensus.cc:2243] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:53.653352 10043 raft_consensus.cc:2272] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:53.654067 10043 ts_tablet_manager.cc:1916] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:53.656693 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_274350c56d3a43a4b3738ced49d24150"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.657361 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_274350c56d3a43a4b3738ced49d24150 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.659044 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a8906ade8f93405bb76d1ca9b7a0afa6" } modify_external_catalogs: true
I20251027 21:07:53.659247 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a8906ade8f93405bb76d1ca9b7a0afa6" } modify_external_catalogs: true
I20251027 21:07:53.660090 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_de808e6797d54833a87f262e64dcd576" } modify_external_catalogs: true
I20251027 21:07:53.660260 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_de808e6797d54833a87f262e64dcd576" } modify_external_catalogs: true
I20251027 21:07:53.663695 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9904948d307e479ea184d6dc441b5e61" } modify_external_catalogs: true
I20251027 21:07:53.663815 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f2775123ffc8464eaf6d5e7d4ee692b0" } modify_external_catalogs: true
I20251027 21:07:53.663852 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9904948d307e479ea184d6dc441b5e61" } modify_external_catalogs: true
I20251027 21:07:53.664016 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f2775123ffc8464eaf6d5e7d4ee692b0" } modify_external_catalogs: true
I20251027 21:07:53.668751 9311 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 0b9b939bccac4d3ab56329d8285cfcc8
I20251027 21:07:53.669924 10043 ts_tablet_manager.cc:1929] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:53.671550 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 59b6818d29624da5a49dd1a7d20994c6
I20251027 21:07:53.671660 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet cb95618d9fd54a4f8121b43e717d5616
I20251027 21:07:53.672806 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 59b6818d29624da5a49dd1a7d20994c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.673060 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet cb95618d9fd54a4f8121b43e717d5616 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.672580 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet cb95618d9fd54a4f8121b43e717d5616 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.672729 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 59b6818d29624da5a49dd1a7d20994c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.674352 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 9861a0efcba64141aae450663c5d83de
I20251027 21:07:53.674563 10043 log.cc:1199] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/d89c3c588a1747d7875ebf45924fb890
I20251027 21:07:53.675091 10043 ts_tablet_manager.cc:1950] T d89c3c588a1747d7875ebf45924fb890 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:53.675673 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 9861a0efcba64141aae450663c5d83de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.676398 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 9861a0efcba64141aae450663c5d83de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.676576 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet 9861a0efcba64141aae450663c5d83de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.676808 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet cb95618d9fd54a4f8121b43e717d5616 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.677723 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet d89c3c588a1747d7875ebf45924fb890 (table default.Table_f05923c6594b43378fa156d0ab2bd1da [id=ad541dd12124493583f4fdba032b7ed4]) successfully deleted
I20251027 21:07:53.679872 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet 59b6818d29624da5a49dd1a7d20994c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:53.681249 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 32f80614e30d41128f9fb2e0ea165223 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32f80614e30d41128f9fb2e0ea165223 already in progress: creating tablet
W20251027 21:07:53.689373 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 588e2f58f07b4a32a78dffd5b92ceb12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 588e2f58f07b4a32a78dffd5b92ceb12 already in progress: creating tablet
W20251027 21:07:53.690236 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet f4187b2f66f24055b1df170f1062cc88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4187b2f66f24055b1df170f1062cc88 already in progress: creating tablet
W20251027 21:07:53.690984 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet bbb4116d1edd492893adfbf5fb3ddeef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbb4116d1edd492893adfbf5fb3ddeef already in progress: creating tablet
W20251027 21:07:53.691761 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 1d64a13667194c12864fa388eb88a380 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d64a13667194c12864fa388eb88a380 already in progress: creating tablet
I20251027 21:07:53.691851 9305 master_service.cc:954] ReplaceTablet: received request to replace tablet d52d71f4976143e4a514350621ebe309 from {username='slave'} at 127.0.0.1:41710
W20251027 21:07:53.692539 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet b719dc40d7e348608cdfc78401c19824 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b719dc40d7e348608cdfc78401c19824 already in progress: creating tablet
I20251027 21:07:53.693089 10043 tablet_replica.cc:333] T efc0019840244d14bb95ae581793e74b P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:53.693317 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 986bf148b3e9482fa2fe53bbac54750d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.693564 10043 raft_consensus.cc:2243] T efc0019840244d14bb95ae581793e74b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:53.696187 10043 raft_consensus.cc:2272] T efc0019840244d14bb95ae581793e74b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:53.697299 10043 ts_tablet_manager.cc:1916] T efc0019840244d14bb95ae581793e74b P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:53.699308 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet d52d71f4976143e4a514350621ebe309
I20251027 21:07:53.700510 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet d52d71f4976143e4a514350621ebe309 with delete_type TABLET_DATA_DELETED (replaced by tablet e419e588b31d48c8bc19d7948081cc81) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.700631 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet d52d71f4976143e4a514350621ebe309 with delete_type TABLET_DATA_DELETED (replaced by tablet e419e588b31d48c8bc19d7948081cc81) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.702337 9305 catalog_manager.cc:6719] ReplaceTablet: tablet d52d71f4976143e4a514350621ebe309 deleted and replaced by tablet e419e588b31d48c8bc19d7948081cc81
I20251027 21:07:53.704121 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet d52d71f4976143e4a514350621ebe309 with delete_type TABLET_DATA_DELETED (replaced by tablet e419e588b31d48c8bc19d7948081cc81) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.707667 10041 tablet_replica.cc:333] T 9248f458baca409aa3505cc0db321b34 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:53.708076 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7425b6b0184e4d0b871777b25d904ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.709630 10041 raft_consensus.cc:2243] T 9248f458baca409aa3505cc0db321b34 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:53.710088 10041 raft_consensus.cc:2272] T 9248f458baca409aa3505cc0db321b34 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:53.714720 10031 tablet_bootstrap.cc:492] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:53.716429 10031 tablet_bootstrap.cc:654] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:53.717790 9614 tablet_service.cc:1505] Processing CreateTablet for tablet dd9e6656d8df43afbb7f70706bc14585 (DEFAULT_TABLE table=default.Table_201f11890a6243e3915632a430cc0ab4 [id=3cdf432ec43d42e8bc2d253a92eac8d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.718334 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd9e6656d8df43afbb7f70706bc14585. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.720621 10041 ts_tablet_manager.cc:1916] T 9248f458baca409aa3505cc0db321b34 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:53.708537 10032 tablet_bootstrap.cc:700] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent opening tablet: real 0.162s user 0.000s sys 0.001s
I20251027 21:07:53.722491 10032 tablet_bootstrap.cc:654] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:53.713022 9890 tablet_service.cc:1505] Processing CreateTablet for tablet dd9e6656d8df43afbb7f70706bc14585 (DEFAULT_TABLE table=default.Table_201f11890a6243e3915632a430cc0ab4 [id=3cdf432ec43d42e8bc2d253a92eac8d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.724186 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd9e6656d8df43afbb7f70706bc14585. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.726907 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 952583fd7fde4b4baccdb98c01bdf9e1 (DEFAULT_TABLE table=default.Table_917a7cf5e19547888798a43ded99f4b8 [id=7527117eef9d46a694977cbf2c073974]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.727279 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 952583fd7fde4b4baccdb98c01bdf9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.727471 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 952583fd7fde4b4baccdb98c01bdf9e1 (DEFAULT_TABLE table=default.Table_917a7cf5e19547888798a43ded99f4b8 [id=7527117eef9d46a694977cbf2c073974]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.727778 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 952583fd7fde4b4baccdb98c01bdf9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.734520 9749 tablet_service.cc:1505] Processing CreateTablet for tablet dd9e6656d8df43afbb7f70706bc14585 (DEFAULT_TABLE table=default.Table_201f11890a6243e3915632a430cc0ab4 [id=3cdf432ec43d42e8bc2d253a92eac8d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.734994 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd9e6656d8df43afbb7f70706bc14585. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.736151 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6ed444d67ba74d9ea15f80509d94f706" } modify_external_catalogs: true
I20251027 21:07:53.736421 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6ed444d67ba74d9ea15f80509d94f706" } modify_external_catalogs: true
I20251027 21:07:53.744064 9305 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 315ba31cbad94f3896b4d51e1a2ef319
I20251027 21:07:53.748660 10029 ts_tablet_manager.cc:1434] T 2b247829a11349b6835a2f18906e20a9 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.245s user 0.018s sys 0.028s
I20251027 21:07:53.749585 10029 tablet_bootstrap.cc:492] T 49b3d23798b740008c98cd0754ae33f4 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:53.751646 10029 tablet_bootstrap.cc:654] T 49b3d23798b740008c98cd0754ae33f4 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:53.761536 10043 ts_tablet_manager.cc:1929] T efc0019840244d14bb95ae581793e74b P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:53.764611 10031 tablet_bootstrap.cc:492] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:53.764801 10031 ts_tablet_manager.cc:1403] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.004s
I20251027 21:07:53.765635 10031 raft_consensus.cc:359] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.765920 10031 raft_consensus.cc:385] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:53.766081 10031 raft_consensus.cc:740] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:53.766619 10031 consensus_queue.cc:260] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.767315 10031 ts_tablet_manager.cc:1434] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:07:53.767647 10031 tablet_bootstrap.cc:492] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:53.771955 10031 tablet_bootstrap.cc:654] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:53.772115 10043 log.cc:1199] T efc0019840244d14bb95ae581793e74b P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/efc0019840244d14bb95ae581793e74b
I20251027 21:07:53.774819 10043 ts_tablet_manager.cc:1950] T efc0019840244d14bb95ae581793e74b P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:53.775923 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet efc0019840244d14bb95ae581793e74b (table default.Table_89407f0c58ec4ee7995b892701a6377f [id=f773d65a07a4473ab8b87cd82ac50bcf]) successfully deleted
W20251027 21:07:53.777705 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 4debc3f13f3a42939b6b83f6523aab7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4debc3f13f3a42939b6b83f6523aab7c already in progress: creating tablet
W20251027 21:07:53.780180 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet da26c83976df40328cf9b11734fcfa44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da26c83976df40328cf9b11734fcfa44 already in progress: creating tablet
I20251027 21:07:53.780963 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet d89c3c588a1747d7875ebf45924fb890 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.782985 10032 tablet_bootstrap.cc:492] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:53.783198 10032 ts_tablet_manager.cc:1403] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.238s user 0.000s sys 0.004s
I20251027 21:07:53.783993 10032 raft_consensus.cc:359] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.784263 10032 raft_consensus.cc:385] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:53.784384 10032 raft_consensus.cc:740] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:53.784942 10359 raft_consensus.cc:493] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:53.785080 10359 raft_consensus.cc:515] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.785769 10359 leader_election.cc:290] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:53.784749 10032 consensus_queue.cc:260] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:53.786900 9711 leader_election.cc:343] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 970fbf01a4c3406dafe1ea4cd005ea23
W20251027 21:07:53.787237 9710 leader_election.cc:343] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 970fbf01a4c3406dafe1ea4cd005ea23
I20251027 21:07:53.787348 10032 ts_tablet_manager.cc:1434] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251027 21:07:53.787361 9710 leader_election.cc:304] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:53.787621 10032 tablet_bootstrap.cc:492] T 94e94cd50c91429d8da9804149501c86 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:53.787786 10359 raft_consensus.cc:2749] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:53.803423 10040 ts_tablet_manager.cc:1929] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:53.803550 10040 log.cc:1199] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/36e168cc44274dae9682b0c1329d4558
I20251027 21:07:53.803903 10040 ts_tablet_manager.cc:1950] T 36e168cc44274dae9682b0c1329d4558 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:53.804205 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 293) took 1653 ms. Trace:
I20251027 21:07:53.804332 10040 rpcz_store.cc:276] 1027 21:07:52.150776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.150832 (+ 56us) service_pool.cc:225] Handling call
1027 21:07:53.403307 (+1252475us) ts_tablet_manager.cc:1151] Deleting tablet 36e168cc44274dae9682b0c1329d4558
1027 21:07:53.403320 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:53.408099 (+ 4779us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:53.803195 (+395096us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:53.804175 (+ 980us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1252279}
I20251027 21:07:53.804773 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 294) took 1641 ms. Trace:
I20251027 21:07:53.804868 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 36e168cc44274dae9682b0c1329d4558 (table default.Table_795e30871da94371958578747d1c898b [id=9a111e9480b84a55b5226f0940d999c4]) successfully deleted
I20251027 21:07:53.804915 10040 rpcz_store.cc:276] 1027 21:07:52.162812 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.162858 (+ 46us) service_pool.cc:225] Handling call
1027 21:07:53.804638 (+1641780us) ts_tablet_manager.cc:1151] Deleting tablet 5c34c8f1433b40d491ae05e7c6e60ba1
1027 21:07:53.804648 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:53.804743 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1641603}
W20251027 21:07:53.805342 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 5c34c8f1433b40d491ae05e7c6e60ba1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c34c8f1433b40d491ae05e7c6e60ba1 already in progress: creating tablet
I20251027 21:07:53.806416 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_762d79ee19da4feb90eca3f718458fc7" } new_table_name: "default.Table_2953868ab4054f75a9975b60db2e5e0a" modify_external_catalogs: true
I20251027 21:07:53.806505 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_da0e93f761894c259f883d648d7f25d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.807078 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da0e93f761894c259f883d648d7f25d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:53.809839 9711 consensus_peers.cc:597] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d659e4d73ea4ff3975a4b58a794090a. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:07:53.810725 10031 tablet_bootstrap.cc:492] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:53.810675 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6abd0422341f4d00a3dc8fe5a515e00a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:53.810858 10031 ts_tablet_manager.cc:1403] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.002s
W20251027 21:07:53.811220 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6abd0422341f4d00a3dc8fe5a515e00a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.811322 10043 tablet_replica.cc:333] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:53.811651 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a449297756a5460aac690849ef0074fc" } new_table_name: "default.Table_7c426eeeaca14f65ab36fef3162d4a3a" modify_external_catalogs: true
I20251027 21:07:53.812904 10043 raft_consensus.cc:2243] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:07:53.797785 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet 49d9e5c5960a4534a6603875006e6266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.815013 10031 raft_consensus.cc:359] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.815553 10043 pending_rounds.cc:70] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14: Trying to abort 1 pending ops.
I20251027 21:07:53.815913 10043 pending_rounds.cc:77] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510624427634688 op_type: NO_OP noop_request { }
W20251027 21:07:53.816083 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 83f5fbf5757a49df93ab949b7657c61f consensus state has no leader
I20251027 21:07:53.816298 10043 raft_consensus.cc:2889] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:07:53.816552 10043 raft_consensus.cc:2272] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251027 21:07:53.819782 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7c15b74f53bc45e1af33b043f2cf225f has no consensus state
I20251027 21:07:53.820228 10031 raft_consensus.cc:385] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:53.821584 10032 tablet_bootstrap.cc:654] T 94e94cd50c91429d8da9804149501c86 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:53.821841 10031 raft_consensus.cc:740] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:53.822350 10031 consensus_queue.cc:260] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.823369 10031 ts_tablet_manager.cc:1434] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20251027 21:07:53.824009 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 295) took 1657 ms. Trace:
I20251027 21:07:53.829516 10040 rpcz_store.cc:276] 1027 21:07:52.166153 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.166195 (+ 42us) service_pool.cc:225] Handling call
1027 21:07:53.823711 (+1657516us) ts_tablet_manager.cc:1151] Deleting tablet efc0019840244d14bb95ae581793e74b
1027 21:07:53.823851 (+ 140us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:53.823963 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2432,"tablet-delete.queue_time_us":1657342}
I20251027 21:07:53.829964 10043 ts_tablet_manager.cc:1916] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:53.831701 10041 ts_tablet_manager.cc:1929] T 9248f458baca409aa3505cc0db321b34 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251027 21:07:53.832149 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet efc0019840244d14bb95ae581793e74b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efc0019840244d14bb95ae581793e74b already in progress: creating tablet
I20251027 21:07:53.830078 10031 tablet_bootstrap.cc:492] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:53.834182 10031 tablet_bootstrap.cc:654] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:53.835090 10041 log.cc:1199] T 9248f458baca409aa3505cc0db321b34 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/9248f458baca409aa3505cc0db321b34
I20251027 21:07:53.835959 10041 ts_tablet_manager.cc:1950] T 9248f458baca409aa3505cc0db321b34 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:53.838187 10041 tablet_replica.cc:333] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:53.838828 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 9248f458baca409aa3505cc0db321b34 (table default.Table_b0a525314a0d4e69aee8e2c4a291d999 [id=64df643709144959945c55df034b77a1]) successfully deleted
I20251027 21:07:53.834463 10040 tablet_replica.cc:333] T cad3a3e910ac42399cf371ad82c89ff7 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:53.842775 10040 raft_consensus.cc:2243] T cad3a3e910ac42399cf371ad82c89ff7 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:53.842953 10040 raft_consensus.cc:2272] T cad3a3e910ac42399cf371ad82c89ff7 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:53.843498 10040 ts_tablet_manager.cc:1916] T cad3a3e910ac42399cf371ad82c89ff7 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:53.847259 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 952583fd7fde4b4baccdb98c01bdf9e1 (DEFAULT_TABLE table=default.Table_917a7cf5e19547888798a43ded99f4b8 [id=7527117eef9d46a694977cbf2c073974]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.848232 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 952583fd7fde4b4baccdb98c01bdf9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.850418 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_01129369d93842049a67974b77112073"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:53.850682 10041 raft_consensus.cc:2243] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:53.850874 10041 raft_consensus.cc:2272] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:53.850841 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5fe0ef16a24f4af9ae8e843adbe1643e" } new_table_name: "default.Table_4803a53377dc41709e7d86dc239bcea7" modify_external_catalogs: true
W20251027 21:07:53.851079 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01129369d93842049a67974b77112073 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.851408 10041 ts_tablet_manager.cc:1916] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:53.852442 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c9a81e9dabad4a7d9fbd427439a7587c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.853047 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9a81e9dabad4a7d9fbd427439a7587c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.836952 10032 tablet_bootstrap.cc:492] T 94e94cd50c91429d8da9804149501c86 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:53.854331 10032 ts_tablet_manager.cc:1403] T 94e94cd50c91429d8da9804149501c86 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.000s
I20251027 21:07:53.855113 10032 raft_consensus.cc:359] T 94e94cd50c91429d8da9804149501c86 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:53.855326 10032 raft_consensus.cc:385] T 94e94cd50c91429d8da9804149501c86 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:53.855430 10032 raft_consensus.cc:740] T 94e94cd50c91429d8da9804149501c86 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:53.855652 10032 consensus_queue.cc:260] T 94e94cd50c91429d8da9804149501c86 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:53.856171 10032 ts_tablet_manager.cc:1434] T 94e94cd50c91429d8da9804149501c86 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:07:53.858398 10031 tablet_bootstrap.cc:492] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:53.858552 10031 ts_tablet_manager.cc:1403] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.003s
I20251027 21:07:53.859305 10031 raft_consensus.cc:359] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.859474 10031 raft_consensus.cc:385] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:53.859571 10031 raft_consensus.cc:740] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:53.859628 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a83df71b24544fc0990359fb6768dffc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:53.859803 10031 consensus_queue.cc:260] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:53.860105 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a83df71b24544fc0990359fb6768dffc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.860059 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_185df62a04014137809bfc1bc3e5ecad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:53.860282 10031 ts_tablet_manager.cc:1434] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:07:53.860579 10031 tablet_bootstrap.cc:492] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:07:53.860574 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_185df62a04014137809bfc1bc3e5ecad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.861874 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_219373cdafa946c68b0330d75aa69cc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:53.862375 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b8bc4fe8cb374bd990d2c777eae1d4f0" } new_table_name: "default.Table_b462e4bf95dc4fc4ad3fd23b236fffa7" modify_external_catalogs: true
I20251027 21:07:53.862768 10032 tablet_bootstrap.cc:492] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:07:53.863765 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_219373cdafa946c68b0330d75aa69cc8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.868428 10032 tablet_bootstrap.cc:654] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:53.870378 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1f82ff66f4a14bb9ac990665fd6ec38e" } new_table_name: "default.Table_3e3e0c17a3bf452fbb217c6e3ee8bdcb" modify_external_catalogs: true
I20251027 21:07:53.873162 10411 raft_consensus.cc:493] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:53.873292 10411 raft_consensus.cc:515] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.873502 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f4272063203347a5810ed07554c2269b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:53.873850 10411 leader_election.cc:290] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:07:53.874042 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4272063203347a5810ed07554c2269b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.874650 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d8825b80cec4ba68da077ca3b8d89db" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:53.875519 9860 leader_election.cc:343] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:53.879113 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet 94e94cd50c91429d8da9804149501c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:53.880288 10041 ts_tablet_manager.cc:1929] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:53.881914 10040 ts_tablet_manager.cc:1929] T cad3a3e910ac42399cf371ad82c89ff7 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:53.885497 10040 log.cc:1199] T cad3a3e910ac42399cf371ad82c89ff7 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/cad3a3e910ac42399cf371ad82c89ff7
I20251027 21:07:53.886291 10040 ts_tablet_manager.cc:1950] T cad3a3e910ac42399cf371ad82c89ff7 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:53.887766 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9fa8f9725a2042b2aabd775502550393" } modify_external_catalogs: true
I20251027 21:07:53.888800 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2041587a20dc42f4bef58ce06f47958f" } modify_external_catalogs: true
I20251027 21:07:53.889743 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2041587a20dc42f4bef58ce06f47958f" } modify_external_catalogs: true
I20251027 21:07:53.890635 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 298) took 1685 ms. Trace:
I20251027 21:07:53.890772 10040 rpcz_store.cc:276] 1027 21:07:52.205086 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.205135 (+ 49us) service_pool.cc:225] Handling call
1027 21:07:53.834386 (+1629251us) ts_tablet_manager.cc:1151] Deleting tablet cad3a3e910ac42399cf371ad82c89ff7
1027 21:07:53.834406 (+ 20us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:53.849358 (+ 14952us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:53.881809 (+ 32451us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:53.890580 (+ 8771us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4864,"tablet-delete.queue_time_us":1629036}
I20251027 21:07:53.891222 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet cad3a3e910ac42399cf371ad82c89ff7 (table default.Table_b84197018ff44b40b7219e21846339f2 [id=0286a1b3137649eeaa5339cbdf97aa03]) successfully deleted
I20251027 21:07:53.882297 10031 tablet_bootstrap.cc:654] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:53.889075 9737 tablet_service.cc:1552] Processing DeleteTablet for tablet 10a44aecfade42969454126b990eca24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:53.891953 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aaa8910d9fdb4146945621ec4ebf969e consensus state has no leader
W20251027 21:07:53.892561 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4996d8fae094db8a96599b9057f1649 consensus state has no leader
I20251027 21:07:53.893932 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9fa8f9725a2042b2aabd775502550393" } modify_external_catalogs: true
I20251027 21:07:53.894202 10031 tablet_bootstrap.cc:492] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
W20251027 21:07:53.896095 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 45568792d7954b498581ce69fb611948 consensus state has no leader
I20251027 21:07:53.897156 10031 ts_tablet_manager.cc:1403] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.004s
I20251027 21:07:53.897950 10031 raft_consensus.cc:359] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.898829 10040 tablet_replica.cc:333] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:53.899030 10040 raft_consensus.cc:2243] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:53.899215 10040 raft_consensus.cc:2272] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:53.899718 10040 ts_tablet_manager.cc:1916] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:53.902786 10041 log.cc:1199] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/e07b8aa0a41241d79254c4dae8738e79
I20251027 21:07:53.904439 10041 ts_tablet_manager.cc:1950] T e07b8aa0a41241d79254c4dae8738e79 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:53.904670 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d8825b80cec4ba68da077ca3b8d89db" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:53.905251 9859 leader_election.cc:343] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:53.905403 9859 leader_election.cc:304] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:53.905838 10411 raft_consensus.cc:2749] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:53.905824 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet e07b8aa0a41241d79254c4dae8738e79 (table default.Table_e68b56d3ddd94f308243ec6a03ea9e6e [id=0a94e88b69094ffe8be3367470be86da]) successfully deleted
I20251027 21:07:53.906407 10041 tablet_replica.cc:333] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
W20251027 21:07:53.907263 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 2a4aa1fe05114c1287ca580c1f411d3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a4aa1fe05114c1287ca580c1f411d3f already in progress: creating tablet
I20251027 21:07:53.907248 9614 tablet_service.cc:1505] Processing CreateTablet for tablet aaaf70c950044387a63820079db3c829 (DEFAULT_TABLE table=default.Table_e08f0cbb71674a44a697686c1b6effc9 [id=471f86ddb9de40a7ae58ec202926a1fd]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:53.907526 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 30ce703650c94c06b93bce976cb5d1fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30ce703650c94c06b93bce976cb5d1fd already in progress: creating tablet
W20251027 21:07:53.907805 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet f6789dc2f7cb4a3c826ac507d5c9a239 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6789dc2f7cb4a3c826ac507d5c9a239 already in progress: creating tablet
I20251027 21:07:53.898221 10031 raft_consensus.cc:385] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251027 21:07:53.908058 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet f34512059f0c47fc9af7db2645f86b40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f34512059f0c47fc9af7db2645f86b40 already in progress: creating tablet
I20251027 21:07:53.908067 10031 raft_consensus.cc:740] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:53.908811 10031 consensus_queue.cc:260] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.909307 9308 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet d1e9d2925522491d87ac41e8273e55d3
I20251027 21:07:53.911468 10031 ts_tablet_manager.cc:1434] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251027 21:07:53.911756 10031 tablet_bootstrap.cc:492] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:53.913014 9737 tablet_service.cc:1505] Processing CreateTablet for tablet aaaf70c950044387a63820079db3c829 (DEFAULT_TABLE table=default.Table_e08f0cbb71674a44a697686c1b6effc9 [id=471f86ddb9de40a7ae58ec202926a1fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.913594 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaaf70c950044387a63820079db3c829. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.914604 9746 tablet_service.cc:1552] Processing DeleteTablet for tablet 019a892bfdd54e62bc3560a341d5c2da with delete_type TABLET_DATA_DELETED (replaced by tablet ccaf5c2f8e544842880ba8ba27300ae6) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:53.915921 10041 raft_consensus.cc:2243] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:53.916113 10041 raft_consensus.cc:2272] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:53.916635 10041 ts_tablet_manager.cc:1916] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:53.919721 9305 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 40f20e07d1c846459b7601ee74abc371
I20251027 21:07:53.911458 9897 tablet_service.cc:1505] Processing CreateTablet for tablet aaaf70c950044387a63820079db3c829 (DEFAULT_TABLE table=default.Table_e08f0cbb71674a44a697686c1b6effc9 [id=471f86ddb9de40a7ae58ec202926a1fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.923075 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaaf70c950044387a63820079db3c829. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.923388 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaaf70c950044387a63820079db3c829. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.926800 10043 ts_tablet_manager.cc:1929] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:07:53.928491 10043 log.cc:1199] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/8fcfa6b85e7b46d78a84947e9f11ae46
I20251027 21:07:53.930496 10043 ts_tablet_manager.cc:1950] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:53.931793 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 8fcfa6b85e7b46d78a84947e9f11ae46 (table default.Table_4723d51b934b4f6a90c790ee4bc7ea86 [id=e68b63cf420d49289a5bd1db81e1cbc1]) successfully deleted
I20251027 21:07:53.932870 10031 tablet_bootstrap.cc:654] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:53.938961 10029 tablet_bootstrap.cc:492] T 49b3d23798b740008c98cd0754ae33f4 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:53.939107 10029 ts_tablet_manager.cc:1403] T 49b3d23798b740008c98cd0754ae33f4 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.190s user 0.008s sys 0.012s
I20251027 21:07:53.939034 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5e19def4c4374da7ba3ee687fb55f0a1" } modify_external_catalogs: true
I20251027 21:07:53.939146 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3ac9f7acd5ac43f1a585c29ecae682eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:53.939244 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5e19def4c4374da7ba3ee687fb55f0a1" } modify_external_catalogs: true
W20251027 21:07:53.939705 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ac9f7acd5ac43f1a585c29ecae682eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.939831 10029 raft_consensus.cc:359] T 49b3d23798b740008c98cd0754ae33f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.940001 10029 raft_consensus.cc:385] T 49b3d23798b740008c98cd0754ae33f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:53.940101 10029 raft_consensus.cc:740] T 49b3d23798b740008c98cd0754ae33f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:53.940344 10029 consensus_queue.cc:260] T 49b3d23798b740008c98cd0754ae33f4 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.942431 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d4fb67aef18b41048fd4f61f7f1f2ebd" } modify_external_catalogs: true
I20251027 21:07:53.942735 10043 tablet_replica.cc:333] T 2b247829a11349b6835a2f18906e20a9 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:53.942876 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d4fb67aef18b41048fd4f61f7f1f2ebd" } modify_external_catalogs: true
I20251027 21:07:53.944032 10043 raft_consensus.cc:2243] T 2b247829a11349b6835a2f18906e20a9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:53.945699 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b36a756b7ab54252aaa0c08e563b2ea3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:53.946815 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b36a756b7ab54252aaa0c08e563b2ea3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:53.948189 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a83df71b24544fc0990359fb6768dffc" } modify_external_catalogs: true
I20251027 21:07:53.948338 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a83df71b24544fc0990359fb6768dffc" } modify_external_catalogs: true
I20251027 21:07:53.949602 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet 59b6818d29624da5a49dd1a7d20994c6 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:53.950510 10029 ts_tablet_manager.cc:1434] T 49b3d23798b740008c98cd0754ae33f4 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251027 21:07:53.952008 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 842c1a1298234d9ea2719bab29c54ae3 (DEFAULT_TABLE table=default.Table_7797741574f1487ba7211ddb3bb032dd [id=4372567fda9b4a178b8adc2cde3e1704]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.952425 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 842c1a1298234d9ea2719bab29c54ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:53.954564 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet 49b3d23798b740008c98cd0754ae33f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:53.957206 9308 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 12827d10d03746119c050344340885fc
I20251027 21:07:53.958021 10043 raft_consensus.cc:2272] T 2b247829a11349b6835a2f18906e20a9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:53.962584 10043 ts_tablet_manager.cc:1916] T 2b247829a11349b6835a2f18906e20a9 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:53.968997 10029 tablet_bootstrap.cc:492] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:53.970788 10359 raft_consensus.cc:493] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:53.970969 10029 tablet_bootstrap.cc:654] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:53.970924 10359 raft_consensus.cc:515] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.971570 10359 leader_election.cc:290] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:07:53.972693 9711 leader_election.cc:343] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: d89c3c588a1747d7875ebf45924fb890
W20251027 21:07:53.973690 9710 leader_election.cc:343] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: d89c3c588a1747d7875ebf45924fb890
I20251027 21:07:53.973794 9710 leader_election.cc:304] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:53.974249 10359 raft_consensus.cc:2749] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:53.975244 10031 tablet_bootstrap.cc:492] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:53.975426 10031 ts_tablet_manager.cc:1403] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.003s
I20251027 21:07:53.976224 10031 raft_consensus.cc:359] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.979673 10041 ts_tablet_manager.cc:1929] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:53.980221 10031 raft_consensus.cc:385] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:53.980472 10031 raft_consensus.cc:740] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:53.980108 10040 ts_tablet_manager.cc:1929] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:53.981957 10040 log.cc:1199] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/d8ef795c88da49b58082ba1264d83378
I20251027 21:07:53.982647 10040 ts_tablet_manager.cc:1950] T d8ef795c88da49b58082ba1264d83378 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:53.983034 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 302) took 1697 ms. Trace:
I20251027 21:07:53.983431 9310 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 3f813deac15a41639adbff6d5294f3c4
I20251027 21:07:53.983865 9305 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet b2d93350759c4ac995fc23e79c9bb3e5
I20251027 21:07:53.983326 10031 consensus_queue.cc:260] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:53.983814 10040 rpcz_store.cc:276] 1027 21:07:52.285725 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.285772 (+ 47us) service_pool.cc:225] Handling call
1027 21:07:53.898745 (+1612973us) ts_tablet_manager.cc:1151] Deleting tablet d8ef795c88da49b58082ba1264d83378
1027 21:07:53.898755 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:53.935424 (+ 36669us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:53.979976 (+ 44552us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:53.982992 (+ 3016us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1612736}
I20251027 21:07:53.986265 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet d8ef795c88da49b58082ba1264d83378 (table default.Table_df95aebbdcf94d8289b235722b7d2c8c [id=98143d42942742568f2c3a1d0fbcc100]) successfully deleted
I20251027 21:07:53.986686 10040 tablet_replica.cc:333] T 9c4369a10bf846fb94c9b71a70f21600 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:53.986878 10040 raft_consensus.cc:2243] T 9c4369a10bf846fb94c9b71a70f21600 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:53.987031 10040 raft_consensus.cc:2272] T 9c4369a10bf846fb94c9b71a70f21600 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:53.988695 10031 ts_tablet_manager.cc:1434] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251027 21:07:53.989025 10031 tablet_bootstrap.cc:492] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:53.992722 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 842c1a1298234d9ea2719bab29c54ae3 (DEFAULT_TABLE table=default.Table_7797741574f1487ba7211ddb3bb032dd [id=4372567fda9b4a178b8adc2cde3e1704]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.993170 10043 ts_tablet_manager.cc:1929] T 2b247829a11349b6835a2f18906e20a9 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:53.993343 10043 log.cc:1199] T 2b247829a11349b6835a2f18906e20a9 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/2b247829a11349b6835a2f18906e20a9
I20251027 21:07:53.994302 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 28a2e85021394f9d8a9ea524959b52a1 (DEFAULT_TABLE table=default.Table_b730ec55ffb24974928bd0b0d9416460 [id=d72c1295ca1a4ec6955476a6eb5b0bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:53.994441 10040 ts_tablet_manager.cc:1916] T 9c4369a10bf846fb94c9b71a70f21600 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:53.994671 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28a2e85021394f9d8a9ea524959b52a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.003172 10041 log.cc:1199] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/cfce0e5c837743ea969374244163be3e
I20251027 21:07:53.993851 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 28a2e85021394f9d8a9ea524959b52a1 (DEFAULT_TABLE table=default.Table_b730ec55ffb24974928bd0b0d9416460 [id=d72c1295ca1a4ec6955476a6eb5b0bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.003883 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28a2e85021394f9d8a9ea524959b52a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.004869 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet efc0019840244d14bb95ae581793e74b with delete_type TABLET_DATA_DELETED (replaced by tablet d2203ee62f7e404c95c5f86b5ae85feb) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:54.006397 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 842c1a1298234d9ea2719bab29c54ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.012305 10359 raft_consensus.cc:493] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:54.012475 10359 raft_consensus.cc:515] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.013127 10359 leader_election.cc:290] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:54.013512 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d13d00353b6740b5bed5a153936d2059" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:54.013787 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d13d00353b6740b5bed5a153936d2059" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:54.014339 9631 raft_consensus.cc:2393] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77b0513f17c0472088901a1ce8b08218 in current term 1: Already voted for candidate 51902b6b61ff4d5584ff21896e847e14 in this term.
I20251027 21:07:54.014387 9924 raft_consensus.cc:2393] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77b0513f17c0472088901a1ce8b08218 in current term 1: Already voted for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in this term.
I20251027 21:07:54.014958 9710 leader_election.cc:304] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:54.015765 10359 raft_consensus.cc:3060] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:54.016129 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 22dbfa50996b47269983c107cc399000 (DEFAULT_TABLE table=default.Table_b78874d9b16b4c76bdff139266ed9f32 [id=be549850a5104d288311e062e1362583]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.016499 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22dbfa50996b47269983c107cc399000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.020587 10359 raft_consensus.cc:2749] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:54.013870 10029 tablet_bootstrap.cc:492] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:54.021024 10029 ts_tablet_manager.cc:1403] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.000s
I20251027 21:07:54.021908 10029 raft_consensus.cc:359] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.022099 10029 raft_consensus.cc:385] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.022195 10029 raft_consensus.cc:740] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.022446 10029 consensus_queue.cc:260] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.022924 10029 ts_tablet_manager.cc:1434] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:07:54.023232 10029 tablet_bootstrap.cc:492] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:54.023339 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3dd9e38840f54ce2a2b0c5642eac1425" } new_table_name: "default.Table_022be94979014126958a69cf564e3239" modify_external_catalogs: true
I20251027 21:07:54.026243 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6c9db36d368044e487f90ca694832333"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.026021 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_979a3d9dccc74a86a0cf5c725bd8cf2b" } new_table_name: "default.Table_6df39c16cd96415fa08e5ad69b4b4504" modify_external_catalogs: true
W20251027 21:07:54.026798 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c9db36d368044e487f90ca694832333 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.028173 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 9248f458baca409aa3505cc0db321b34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:54.029676 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93a3b22cdc1543aebcc7fe72f2f2c0ae has no consensus state
I20251027 21:07:53.993819 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 842c1a1298234d9ea2719bab29c54ae3 (DEFAULT_TABLE table=default.Table_7797741574f1487ba7211ddb3bb032dd [id=4372567fda9b4a178b8adc2cde3e1704]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.034504 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 842c1a1298234d9ea2719bab29c54ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.034427 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 28a2e85021394f9d8a9ea524959b52a1 (DEFAULT_TABLE table=default.Table_b730ec55ffb24974928bd0b0d9416460 [id=d72c1295ca1a4ec6955476a6eb5b0bb4]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:54.035748 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 94f2819dd0394530ae93754bdd1e482f consensus state has no leader
I20251027 21:07:54.036008 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28a2e85021394f9d8a9ea524959b52a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.039628 10029 tablet_bootstrap.cc:654] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.040243 10031 tablet_bootstrap.cc:654] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.041368 10041 ts_tablet_manager.cc:1950] T cfce0e5c837743ea969374244163be3e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:54.042640 10043 ts_tablet_manager.cc:1950] T 2b247829a11349b6835a2f18906e20a9 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:54.046434 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 359) took 1023 ms. Trace:
I20251027 21:07:54.047646 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 2b247829a11349b6835a2f18906e20a9 (table default.Table_096b7cf08dbf49279d8d5c11b3b687ab [id=20b788f5a2d14b31a58e1b29fa954951]) successfully deleted
W20251027 21:07:54.047900 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 04c579bb008448e9baa841135b6dbc61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04c579bb008448e9baa841135b6dbc61 already in progress: creating tablet
I20251027 21:07:54.047849 10041 rpcz_store.cc:276] 1027 21:07:53.022463 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.022516 (+ 53us) service_pool.cc:225] Handling call
1027 21:07:53.906362 (+883846us) ts_tablet_manager.cc:1151] Deleting tablet cfce0e5c837743ea969374244163be3e
1027 21:07:53.906368 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:53.954332 (+ 47964us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:53.979559 (+ 25227us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:54.046375 (+ 66816us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":883677}
I20251027 21:07:54.048961 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet cfce0e5c837743ea969374244163be3e (table default.Table_4568794575f345c08a45afeb2d4f9648 [id=13a25b18846c4580a337b91cee116a3d]) successfully deleted
I20251027 21:07:54.049769 10029 tablet_bootstrap.cc:492] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:54.049916 10029 ts_tablet_manager.cc:1403] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20251027 21:07:54.050740 10029 raft_consensus.cc:359] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.050890 10029 raft_consensus.cc:385] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.050982 10029 raft_consensus.cc:740] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
W20251027 21:07:54.051723 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 32f80614e30d41128f9fb2e0ea165223 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32f80614e30d41128f9fb2e0ea165223 already in progress: creating tablet
I20251027 21:07:54.052717 10032 tablet_bootstrap.cc:492] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:54.052891 10032 ts_tablet_manager.cc:1403] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.190s user 0.001s sys 0.014s
I20251027 21:07:54.053658 10032 raft_consensus.cc:359] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.053913 10032 raft_consensus.cc:385] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.054033 10032 raft_consensus.cc:740] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.054390 10032 consensus_queue.cc:260] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.054994 10032 ts_tablet_manager.cc:1434] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 21:07:54.055617 10032 tablet_bootstrap.cc:492] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:54.055763 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 823620e5486e4cf982fdf9105e8cc4f1 (DEFAULT_TABLE table=default.Table_918877c62b6c4722a5dcf9c7ca26dd76 [id=69c0101c6db84e029d16092e446690a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.056161 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 823620e5486e4cf982fdf9105e8cc4f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.057854 10032 tablet_bootstrap.cc:654] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.059078 10043 tablet_replica.cc:333] T 49b3d23798b740008c98cd0754ae33f4 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:54.059315 10043 raft_consensus.cc:2243] T 49b3d23798b740008c98cd0754ae33f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.059566 10043 raft_consensus.cc:2272] T 49b3d23798b740008c98cd0754ae33f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:54.060123 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 27bd3b2a011648b2b8f2f92944513751 (DEFAULT_TABLE table=default.Table_4d2a5a0a0ac24d4bb907b89a6bee1aaf [id=8f16323b12524c71924a38207824dae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.060377 10043 ts_tablet_manager.cc:1916] T 49b3d23798b740008c98cd0754ae33f4 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:54.060509 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27bd3b2a011648b2b8f2f92944513751. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.061990 10041 tablet_replica.cc:333] T 1776590d8387494da5dfbd08db17f4b1 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:54.062249 10041 raft_consensus.cc:2243] T 1776590d8387494da5dfbd08db17f4b1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.062426 10041 raft_consensus.cc:2272] T 1776590d8387494da5dfbd08db17f4b1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:07:54.062755 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 588e2f58f07b4a32a78dffd5b92ceb12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 588e2f58f07b4a32a78dffd5b92ceb12 already in progress: creating tablet
I20251027 21:07:54.062974 10041 ts_tablet_manager.cc:1916] T 1776590d8387494da5dfbd08db17f4b1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:54.063015 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet bbb4116d1edd492893adfbf5fb3ddeef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbb4116d1edd492893adfbf5fb3ddeef already in progress: creating tablet
W20251027 21:07:54.063202 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet f4187b2f66f24055b1df170f1062cc88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4187b2f66f24055b1df170f1062cc88 already in progress: creating tablet
W20251027 21:07:54.063421 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 1d64a13667194c12864fa388eb88a380 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d64a13667194c12864fa388eb88a380 already in progress: creating tablet
W20251027 21:07:54.063594 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet b719dc40d7e348608cdfc78401c19824 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b719dc40d7e348608cdfc78401c19824 already in progress: creating tablet
I20251027 21:07:54.064199 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 823620e5486e4cf982fdf9105e8cc4f1 (DEFAULT_TABLE table=default.Table_918877c62b6c4722a5dcf9c7ca26dd76 [id=69c0101c6db84e029d16092e446690a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.064275 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 27bd3b2a011648b2b8f2f92944513751 (DEFAULT_TABLE table=default.Table_4d2a5a0a0ac24d4bb907b89a6bee1aaf [id=8f16323b12524c71924a38207824dae4]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:54.064560 9298 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 93a3b22cdc1543aebcc7fe72f2f2c0ae (table default.Table_022be94979014126958a69cf564e3239 [id=8646c99b90414f4fa266cd7a31123d8e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 93a3b22cdc1543aebcc7fe72f2f2c0ae has no consensus state
I20251027 21:07:54.064587 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 823620e5486e4cf982fdf9105e8cc4f1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:54.064731 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 76e4fb2bea7f4a4696fe9f6bb892bccc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76e4fb2bea7f4a4696fe9f6bb892bccc already in progress: creating tablet
W20251027 21:07:54.064965 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet cb95618d9fd54a4f8121b43e717d5616 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb95618d9fd54a4f8121b43e717d5616 already in progress: creating tablet
W20251027 21:07:54.065150 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 59b6818d29624da5a49dd1a7d20994c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59b6818d29624da5a49dd1a7d20994c6 already in progress: creating tablet
W20251027 21:07:54.065346 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 9861a0efcba64141aae450663c5d83de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9861a0efcba64141aae450663c5d83de already in progress: creating tablet
W20251027 21:07:54.065557 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet d52d71f4976143e4a514350621ebe309 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d52d71f4976143e4a514350621ebe309 already in progress: creating tablet
I20251027 21:07:54.065970 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 22dbfa50996b47269983c107cc399000 (DEFAULT_TABLE table=default.Table_b78874d9b16b4c76bdff139266ed9f32 [id=be549850a5104d288311e062e1362583]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.074749 9310 master_service.cc:954] ReplaceTablet: received request to replace tablet abc7458fcfb94985acf76d77c0292b08 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:54.075443 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22dbfa50996b47269983c107cc399000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.078289 10029 consensus_queue.cc:260] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.078903 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27bd3b2a011648b2b8f2f92944513751. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.085737 9306 master_service.cc:954] ReplaceTablet: received request to replace tablet 22d861810ed846c3b77a452d05e93d72 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:54.087944 9898 tablet_service.cc:1505] Processing CreateTablet for tablet d2203ee62f7e404c95c5f86b5ae85feb (DEFAULT_TABLE table=default.Table_89407f0c58ec4ee7995b892701a6377f [id=f773d65a07a4473ab8b87cd82ac50bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.088346 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 8cccd253ae2a46328dc9f767b8e9ee97 (DEFAULT_TABLE table=default.Table_fa13b0f459f9443e965078c8867a629c [id=501b230a824d4a17bb1903c8409407b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.088594 9893 tablet_service.cc:1505] Processing CreateTablet for tablet cdbde6272af147829ecadd67d634f049 (DEFAULT_TABLE table=default.Table_340ce2d28b454548af6c487cb35008ac [id=abc18d8f40b647c6afa5cbabd1cdaba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.088346 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2203ee62f7e404c95c5f86b5ae85feb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.088984 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cccd253ae2a46328dc9f767b8e9ee97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.089404 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdbde6272af147829ecadd67d634f049. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.097715 9900 tablet_service.cc:1552] Processing DeleteTablet for tablet ae124514964440c5b88b16ac9f8a2567 with delete_type TABLET_DATA_DELETED (replaced by tablet 01e9db9eaf8e41c6a2ee13b9ed127421) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:54.099179 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 0e4d35f229bc46e1859ded7c8196ab9d (DEFAULT_TABLE table=default.Table_00deabd570914e54b7fa9c4fc4eb02c7 [id=ba8a1f1eb0d445cbaf7f263429085ee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.099543 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e4d35f229bc46e1859ded7c8196ab9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.100212 9904 tablet_service.cc:1505] Processing CreateTablet for tablet c9c86253ce914916bee6584675a82067 (DEFAULT_TABLE table=default.Table_4469fd75838845e9b60fd07ddc845f5c [id=5076832e735b4894a0ce4f84c5606b83]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.100612 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9c86253ce914916bee6584675a82067. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.102931 9594 tablet_service.cc:1505] Processing CreateTablet for tablet c9c86253ce914916bee6584675a82067 (DEFAULT_TABLE table=default.Table_4469fd75838845e9b60fd07ddc845f5c [id=5076832e735b4894a0ce4f84c5606b83]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.103338 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9c86253ce914916bee6584675a82067. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.103348 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 72c1dc3b9eff47ee81df821deb789307 (DEFAULT_TABLE table=default.Table_052e3f023dfd462884130780e8c57936 [id=2a96776404334cf79ddb611a234a5844]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.103605 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 8cccd253ae2a46328dc9f767b8e9ee97 (DEFAULT_TABLE table=default.Table_fa13b0f459f9443e965078c8867a629c [id=501b230a824d4a17bb1903c8409407b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.103760 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72c1dc3b9eff47ee81df821deb789307. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.104149 9602 tablet_service.cc:1505] Processing CreateTablet for tablet cdbde6272af147829ecadd67d634f049 (DEFAULT_TABLE table=default.Table_340ce2d28b454548af6c487cb35008ac [id=abc18d8f40b647c6afa5cbabd1cdaba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.103338 9604 tablet_service.cc:1505] Processing CreateTablet for tablet 0e4d35f229bc46e1859ded7c8196ab9d (DEFAULT_TABLE table=default.Table_00deabd570914e54b7fa9c4fc4eb02c7 [id=ba8a1f1eb0d445cbaf7f263429085ee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.104801 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cccd253ae2a46328dc9f767b8e9ee97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.110255 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdbde6272af147829ecadd67d634f049. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.114255 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e4d35f229bc46e1859ded7c8196ab9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.115104 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 520f7142f8af4599ac4880b0692e3151 (DEFAULT_TABLE table=default.Table_e6955caf15904f3bae744add8221897f [id=bd9615f2023e45688ae88cd7d323e61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.115425 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 520f7142f8af4599ac4880b0692e3151. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.118374 10029 ts_tablet_manager.cc:1434] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.068s user 0.002s sys 0.000s
I20251027 21:07:54.118520 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 22d861810ed846c3b77a452d05e93d72
I20251027 21:07:54.118705 10029 tablet_bootstrap.cc:492] T c49775486ff24686b0f45c47e2757aba P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:54.118953 9306 catalog_manager.cc:6719] ReplaceTablet: tablet 22d861810ed846c3b77a452d05e93d72 deleted and replaced by tablet c0e2772d66404d74891e0893c49a66e0
I20251027 21:07:54.120868 10029 tablet_bootstrap.cc:654] T c49775486ff24686b0f45c47e2757aba P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.124603 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet 22d861810ed846c3b77a452d05e93d72 with delete_type TABLET_DATA_DELETED (replaced by tablet c0e2772d66404d74891e0893c49a66e0) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:54.124708 9594 tablet_service.cc:1505] Processing CreateTablet for tablet e6ed9d7a700446edafcd0daa02aa7ef6 (DEFAULT_TABLE table=default.Table_c814f5ad5b6f46f2aa8fc2ecf7ce2d73 [id=af0612025c07415fa46cf2fad2609787]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.125041 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6ed9d7a700446edafcd0daa02aa7ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.137164 10043 ts_tablet_manager.cc:1929] T 49b3d23798b740008c98cd0754ae33f4 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.137292 10043 log.cc:1199] T 49b3d23798b740008c98cd0754ae33f4 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/49b3d23798b740008c98cd0754ae33f4
I20251027 21:07:54.137665 10043 ts_tablet_manager.cc:1950] T 49b3d23798b740008c98cd0754ae33f4 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:54.139020 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7ed2c827a45d4c80a5416d12258c77c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.139314 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_35ea0e752f924d568bc7faf8412fe091"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.139886 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_82b8023e5cb34761981a49e164eb36c4" } modify_external_catalogs: true
I20251027 21:07:54.140053 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_82b8023e5cb34761981a49e164eb36c4" } modify_external_catalogs: true
I20251027 21:07:54.141213 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b6905d2ba9e649178b7b4cb0a420a516"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.141825 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_fad068c83e3f4abeb0eefca5ed055111"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.141939 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7ae5d7111eac45a7b3061078d4fc83d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.142437 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_30780e9f9901401ba8240a70a20ad769" } new_table_name: "default.Table_a2b19b07f5ec4668915f71b441eb1a37" modify_external_catalogs: true
I20251027 21:07:54.143561 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_022be94979014126958a69cf564e3239" } new_table_name: "default.Table_96870e913d9243568d09f6f4cb2ccbb4" modify_external_catalogs: true
I20251027 21:07:54.144603 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_53424b483314460bb3b8eb15be3a21ea" } new_table_name: "default.Table_02a5b78b84f1409b80d09d02e5069b9c" modify_external_catalogs: true
W20251027 21:07:54.150602 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fa39697360c46138ef4cd21e8c7a950 consensus state has no leader
I20251027 21:07:54.054590 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 520f7142f8af4599ac4880b0692e3151 (DEFAULT_TABLE table=default.Table_e6955caf15904f3bae744add8221897f [id=bd9615f2023e45688ae88cd7d323e61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.150930 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 520f7142f8af4599ac4880b0692e3151. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:54.152740 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93a3b22cdc1543aebcc7fe72f2f2c0ae has no consensus state
I20251027 21:07:54.155148 10358 raft_consensus.cc:493] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:54.155520 10358 raft_consensus.cc:515] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.157716 10358 leader_election.cc:290] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:54.159811 10043 tablet_replica.cc:333] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
W20251027 21:07:54.159931 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 43611e33512b49d58485cf15de788be7 has no consensus state
I20251027 21:07:54.159874 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f5b100cf12504aeaa64ef743bba49f18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.160233 10043 raft_consensus.cc:2243] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.160379 10043 raft_consensus.cc:2272] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:54.160036 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 49b3d23798b740008c98cd0754ae33f4 (table default.Table_5d5f327d728f4250841a9f4f10a2b533 [id=33dbf5fe986c415cb54fbf1b574651ac]) successfully deleted
I20251027 21:07:54.161258 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f99fba6d68834ef4b426a23a2ca11c9e" } modify_external_catalogs: true
I20251027 21:07:54.161180 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d5398f9a1dca4617be8062e32b583441"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.161407 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f99fba6d68834ef4b426a23a2ca11c9e" } modify_external_catalogs: true
W20251027 21:07:54.162081 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5398f9a1dca4617be8062e32b583441 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:54.162276 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6905d2ba9e649178b7b4cb0a420a516 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.162664 10043 ts_tablet_manager.cc:1916] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:54.162076 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5b100cf12504aeaa64ef743bba49f18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:54.164992 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ed2c827a45d4c80a5416d12258c77c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:54.165303 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ae5d7111eac45a7b3061078d4fc83d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:54.166510 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fad068c83e3f4abeb0eefca5ed055111 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:54.168202 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35ea0e752f924d568bc7faf8412fe091 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.170521 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 22dbfa50996b47269983c107cc399000 (DEFAULT_TABLE table=default.Table_b78874d9b16b4c76bdff139266ed9f32 [id=be549850a5104d288311e062e1362583]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.171877 9889 tablet_service.cc:1505] Processing CreateTablet for tablet e6ed9d7a700446edafcd0daa02aa7ef6 (DEFAULT_TABLE table=default.Table_c814f5ad5b6f46f2aa8fc2ecf7ce2d73 [id=af0612025c07415fa46cf2fad2609787]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.172251 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6ed9d7a700446edafcd0daa02aa7ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.173866 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 72c1dc3b9eff47ee81df821deb789307 (DEFAULT_TABLE table=default.Table_052e3f023dfd462884130780e8c57936 [id=2a96776404334cf79ddb611a234a5844]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.176575 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c90c85540a2142218c304c8da1a2df90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:54.177068 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c90c85540a2142218c304c8da1a2df90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.178324 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7a01697cdee04a929068e3297b598128" } modify_external_catalogs: true
I20251027 21:07:54.178571 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7a01697cdee04a929068e3297b598128" } modify_external_catalogs: true
I20251027 21:07:54.179237 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72c1dc3b9eff47ee81df821deb789307. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:54.183993 9568 leader_election.cc:343] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 9248f458baca409aa3505cc0db321b34
I20251027 21:07:54.184830 10359 raft_consensus.cc:493] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:54.186784 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_796614d1dc4148699dfa07316ea8febf" } modify_external_catalogs: true
I20251027 21:07:54.186841 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3e0f707f8e2844bfaf937150b747a935" } modify_external_catalogs: true
I20251027 21:07:54.186939 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_796614d1dc4148699dfa07316ea8febf" } modify_external_catalogs: true
I20251027 21:07:54.187068 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3e0f707f8e2844bfaf937150b747a935" } modify_external_catalogs: true
I20251027 21:07:54.187522 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4184f1a26ee84a88a37ee927169b9b44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:54.187952 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4184f1a26ee84a88a37ee927169b9b44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.193151 10043 ts_tablet_manager.cc:1929] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.196322 9753 tablet_service.cc:1505] Processing CreateTablet for tablet d2203ee62f7e404c95c5f86b5ae85feb (DEFAULT_TABLE table=default.Table_89407f0c58ec4ee7995b892701a6377f [id=f773d65a07a4473ab8b87cd82ac50bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.196707 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 27bd3b2a011648b2b8f2f92944513751 (DEFAULT_TABLE table=default.Table_4d2a5a0a0ac24d4bb907b89a6bee1aaf [id=8f16323b12524c71924a38207824dae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.197610 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 823620e5486e4cf982fdf9105e8cc4f1 (DEFAULT_TABLE table=default.Table_918877c62b6c4722a5dcf9c7ca26dd76 [id=69c0101c6db84e029d16092e446690a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.198659 10043 log.cc:1199] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/9248f458baca409aa3505cc0db321b34
I20251027 21:07:54.198871 9743 tablet_service.cc:1505] Processing CreateTablet for tablet 8cccd253ae2a46328dc9f767b8e9ee97 (DEFAULT_TABLE table=default.Table_fa13b0f459f9443e965078c8867a629c [id=501b230a824d4a17bb1903c8409407b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.198788 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9248f458baca409aa3505cc0db321b34" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:54.199460 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22dbfa50996b47269983c107cc399000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.199767 9740 tablet_service.cc:1505] Processing CreateTablet for tablet cdbde6272af147829ecadd67d634f049 (DEFAULT_TABLE table=default.Table_340ce2d28b454548af6c487cb35008ac [id=abc18d8f40b647c6afa5cbabd1cdaba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.201264 9747 tablet_service.cc:1505] Processing CreateTablet for tablet e6ed9d7a700446edafcd0daa02aa7ef6 (DEFAULT_TABLE table=default.Table_c814f5ad5b6f46f2aa8fc2ecf7ce2d73 [id=af0612025c07415fa46cf2fad2609787]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.201316 9741 tablet_service.cc:1505] Processing CreateTablet for tablet 0e4d35f229bc46e1859ded7c8196ab9d (DEFAULT_TABLE table=default.Table_00deabd570914e54b7fa9c4fc4eb02c7 [id=ba8a1f1eb0d445cbaf7f263429085ee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.201285 10359 raft_consensus.cc:515] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.202453 9744 tablet_service.cc:1505] Processing CreateTablet for tablet 72c1dc3b9eff47ee81df821deb789307 (DEFAULT_TABLE table=default.Table_052e3f023dfd462884130780e8c57936 [id=2a96776404334cf79ddb611a234a5844]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.203488 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 22d861810ed846c3b77a452d05e93d72 with delete_type TABLET_DATA_DELETED (replaced by tablet c0e2772d66404d74891e0893c49a66e0) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:54.203459 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 520f7142f8af4599ac4880b0692e3151 (DEFAULT_TABLE table=default.Table_e6955caf15904f3bae744add8221897f [id=bd9615f2023e45688ae88cd7d323e61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.204809 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27bd3b2a011648b2b8f2f92944513751. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.205076 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6ed9d7a700446edafcd0daa02aa7ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.205475 10359 leader_election.cc:290] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:54.205883 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cccd253ae2a46328dc9f767b8e9ee97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.207304 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e4d35f229bc46e1859ded7c8196ab9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.207511 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 823620e5486e4cf982fdf9105e8cc4f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.207710 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72c1dc3b9eff47ee81df821deb789307. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.208125 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 520f7142f8af4599ac4880b0692e3151. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.208918 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2203ee62f7e404c95c5f86b5ae85feb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.209137 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdbde6272af147829ecadd67d634f049. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.210078 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8fcfa6b85e7b46d78a84947e9f11ae46" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:54.210444 9924 raft_consensus.cc:2393] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77b0513f17c0472088901a1ce8b08218 in current term 1: Already voted for candidate 51902b6b61ff4d5584ff21896e847e14 in this term.
W20251027 21:07:54.212688 9711 consensus_peers.cc:597] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6249fc5c6174e8e81b9d02d598d5ebd. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:07:54.213165 9710 leader_election.cc:343] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 8fcfa6b85e7b46d78a84947e9f11ae46
I20251027 21:07:54.213294 9710 leader_election.cc:304] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:54.179379 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 22d861810ed846c3b77a452d05e93d72 with delete_type TABLET_DATA_DELETED (replaced by tablet c0e2772d66404d74891e0893c49a66e0) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:54.218286 10359 raft_consensus.cc:3060] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:54.220978 10359 raft_consensus.cc:2749] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:54.222555 9742 tablet_service.cc:1505] Processing CreateTablet for tablet c9c86253ce914916bee6584675a82067 (DEFAULT_TABLE table=default.Table_4469fd75838845e9b60fd07ddc845f5c [id=5076832e735b4894a0ce4f84c5606b83]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.224144 10411 raft_consensus.cc:493] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:54.224321 10411 raft_consensus.cc:515] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.225020 10411 leader_election.cc:290] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:54.228319 10041 ts_tablet_manager.cc:1929] T 1776590d8387494da5dfbd08db17f4b1 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.232270 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_bf6a0088d06247ca9eb20f7fbfb0f0e7" } new_table_name: "default.Table_e18ff5a3d21f467db8e09554b370ef9d" modify_external_catalogs: true
I20251027 21:07:54.234321 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae124514964440c5b88b16ac9f8a2567" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:54.235749 9710 consensus_peers.cc:597] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 -> Peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Couldn't send request to peer 51902b6b61ff4d5584ff21896e847e14. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6249fc5c6174e8e81b9d02d598d5ebd. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:07:54.237483 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a65e330a0b6d42efb7ae68d62db38755" } new_table_name: "default.Table_37d44f2ffd8b4816b4bff0b1960cd275" modify_external_catalogs: true
I20251027 21:07:54.237813 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_73afc9d8e0d740d098eb77c4018c3cb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:54.238425 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73afc9d8e0d740d098eb77c4018c3cb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:54.241940 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet abd3b8e71d3041b4b9e5f6663841bfcf consensus state has no leader
W20251027 21:07:54.235353 9859 leader_election.cc:343] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:54.245580 10041 log.cc:1199] T 1776590d8387494da5dfbd08db17f4b1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/1776590d8387494da5dfbd08db17f4b1
I20251027 21:07:54.246335 10041 ts_tablet_manager.cc:1950] T 1776590d8387494da5dfbd08db17f4b1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:54.248333 10032 tablet_bootstrap.cc:492] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:54.248559 10032 ts_tablet_manager.cc:1403] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.193s user 0.002s sys 0.006s
I20251027 21:07:54.249279 10032 raft_consensus.cc:359] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.249509 10032 raft_consensus.cc:385] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.249652 10032 raft_consensus.cc:740] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.249994 10032 consensus_queue.cc:260] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.250875 10032 ts_tablet_manager.cc:1434] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:54.251230 10032 tablet_bootstrap.cc:492] T 583fba7c750e4ac7b768da447002c211 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:54.253393 10040 ts_tablet_manager.cc:1929] T 9c4369a10bf846fb94c9b71a70f21600 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.253535 10040 log.cc:1199] T 9c4369a10bf846fb94c9b71a70f21600 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/9c4369a10bf846fb94c9b71a70f21600
I20251027 21:07:54.253865 10040 ts_tablet_manager.cc:1950] T 9c4369a10bf846fb94c9b71a70f21600 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:54.253968 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 1776590d8387494da5dfbd08db17f4b1 (table default.Table_b779f2756a924ab6ba1bc1702a412e06 [id=8ed572891f9b41cba8ed984911fa6b11]) successfully deleted
I20251027 21:07:54.254396 10041 tablet_replica.cc:333] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:54.255259 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 303) took 1932 ms. Trace:
I20251027 21:07:54.255371 10040 rpcz_store.cc:276] 1027 21:07:52.322347 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.322407 (+ 60us) service_pool.cc:225] Handling call
1027 21:07:53.986625 (+1664218us) ts_tablet_manager.cc:1151] Deleting tablet 9c4369a10bf846fb94c9b71a70f21600
1027 21:07:53.986636 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:54.200955 (+214319us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:54.253308 (+ 52353us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:54.255208 (+ 1900us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1663844}
I20251027 21:07:54.255721 10041 raft_consensus.cc:2243] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.255856 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 301) took 2009 ms. Trace:
I20251027 21:07:54.255959 10040 rpcz_store.cc:276] 1027 21:07:52.246093 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.328448 (+ 82355us) service_pool.cc:225] Handling call
1027 21:07:54.255678 (+1927230us) ts_tablet_manager.cc:1151] Deleting tablet 01e9db9eaf8e41c6a2ee13b9ed127421
1027 21:07:54.255687 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:54.255792 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1926974}
I20251027 21:07:54.256284 10040 tablet_replica.cc:333] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:54.256456 10040 raft_consensus.cc:2243] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.256067 10041 raft_consensus.cc:2272] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:54.256628 10040 raft_consensus.cc:2272] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:54.256971 10032 tablet_bootstrap.cc:654] T 583fba7c750e4ac7b768da447002c211 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.257182 10040 ts_tablet_manager.cc:1916] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:54.257534 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 9c4369a10bf846fb94c9b71a70f21600 (table default.Table_ec86de5add82433b81e1060e76d559e3 [id=731981b3bb2c4078a597da73199e31d8]) successfully deleted
W20251027 21:07:54.257858 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 01e9db9eaf8e41c6a2ee13b9ed127421 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01e9db9eaf8e41c6a2ee13b9ed127421 already in progress: creating tablet
W20251027 21:07:54.202105 9566 leader_election.cc:343] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:54.261396 10041 ts_tablet_manager.cc:1916] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:54.262020 9566 leader_election.cc:304] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
W20251027 21:07:54.262332 9566 raft_consensus.cc:2686] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251027 21:07:54.262537 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 60482.6us
I20251027 21:07:54.210841 9614 tablet_service.cc:1505] Processing CreateTablet for tablet d2203ee62f7e404c95c5f86b5ae85feb (DEFAULT_TABLE table=default.Table_89407f0c58ec4ee7995b892701a6377f [id=f773d65a07a4473ab8b87cd82ac50bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.225854 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae124514964440c5b88b16ac9f8a2567" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:54.263478 9860 leader_election.cc:343] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:54.263617 9860 leader_election.cc:304] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:54.264005 10411 raft_consensus.cc:2749] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:54.264752 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2203ee62f7e404c95c5f86b5ae85feb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.271636 10043 ts_tablet_manager.cc:1950] T 9248f458baca409aa3505cc0db321b34 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:54.277005 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 9248f458baca409aa3505cc0db321b34 (table default.Table_b0a525314a0d4e69aee8e2c4a291d999 [id=64df643709144959945c55df034b77a1]) successfully deleted
I20251027 21:07:54.279163 10031 tablet_bootstrap.cc:492] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:54.279398 10031 ts_tablet_manager.cc:1403] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.290s user 0.006s sys 0.006s
I20251027 21:07:54.280103 10411 raft_consensus.cc:493] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:54.280251 10031 raft_consensus.cc:359] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.280246 10411 raft_consensus.cc:515] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.280411 10031 raft_consensus.cc:385] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.280472 10031 raft_consensus.cc:740] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.280670 10031 consensus_queue.cc:260] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.280879 10411 leader_election.cc:290] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:54.281172 10031 ts_tablet_manager.cc:1434] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:07:54.281477 10031 tablet_bootstrap.cc:492] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:54.281622 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49d9e5c5960a4534a6603875006e6266" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:54.282284 9860 leader_election.cc:343] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:07:54.282820 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59300e16b3e148a08a217fc24d1e572f has no consensus state
I20251027 21:07:54.282954 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49d9e5c5960a4534a6603875006e6266" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:54.283555 9859 leader_election.cc:343] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:54.283667 9859 leader_election.cc:304] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:54.284052 10411 raft_consensus.cc:2749] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:07:54.284026 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 22d861810ed846c3b77a452d05e93d72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22d861810ed846c3b77a452d05e93d72 already in progress: creating tablet
I20251027 21:07:54.286433 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9c86253ce914916bee6584675a82067. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.287451 10032 tablet_bootstrap.cc:492] T 583fba7c750e4ac7b768da447002c211 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:54.287766 10032 ts_tablet_manager.cc:1403] T 583fba7c750e4ac7b768da447002c211 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.001s
I20251027 21:07:54.289146 10032 raft_consensus.cc:359] T 583fba7c750e4ac7b768da447002c211 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.289719 10032 raft_consensus.cc:385] T 583fba7c750e4ac7b768da447002c211 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.289899 10032 raft_consensus.cc:740] T 583fba7c750e4ac7b768da447002c211 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.290382 10032 consensus_queue.cc:260] T 583fba7c750e4ac7b768da447002c211 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.291568 10032 ts_tablet_manager.cc:1434] T 583fba7c750e4ac7b768da447002c211 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:07:54.292634 10032 tablet_bootstrap.cc:492] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:54.293457 10041 ts_tablet_manager.cc:1929] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.294245 10041 log.cc:1199] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/7d8825b80cec4ba68da077ca3b8d89db
I20251027 21:07:54.296188 10041 ts_tablet_manager.cc:1950] T 7d8825b80cec4ba68da077ca3b8d89db P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:54.300027 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 7d8825b80cec4ba68da077ca3b8d89db (table default.Table_943417713b3b456ab4ca691d2fb168fd [id=1cb32eea25114c6dbcb3b28044036c08]) successfully deleted
W20251027 21:07:54.301661 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 4debc3f13f3a42939b6b83f6523aab7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4debc3f13f3a42939b6b83f6523aab7c already in progress: creating tablet
I20251027 21:07:54.302172 10032 tablet_bootstrap.cc:654] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:54.304085 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet da26c83976df40328cf9b11734fcfa44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da26c83976df40328cf9b11734fcfa44 already in progress: creating tablet
I20251027 21:07:54.308894 10041 tablet_replica.cc:333] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:54.309170 10041 raft_consensus.cc:2243] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.309422 10041 raft_consensus.cc:2272] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:54.311061 10040 ts_tablet_manager.cc:1929] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.311185 10040 log.cc:1199] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/33e41145c79b4465b8d564741174b8e0
I20251027 21:07:54.311568 10040 ts_tablet_manager.cc:1950] T 33e41145c79b4465b8d564741174b8e0 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:54.312158 10041 ts_tablet_manager.cc:1916] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:54.314610 10031 tablet_bootstrap.cc:654] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.323354 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 304) took 1987 ms. Trace:
I20251027 21:07:54.323664 10040 rpcz_store.cc:276] 1027 21:07:52.335511 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.335754 (+ 243us) service_pool.cc:225] Handling call
1027 21:07:54.256214 (+1920460us) ts_tablet_manager.cc:1151] Deleting tablet 33e41145c79b4465b8d564741174b8e0
1027 21:07:54.256221 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:54.307686 (+ 51465us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:54.310954 (+ 3268us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:54.323307 (+ 12353us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1920213}
I20251027 21:07:54.324388 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 33e41145c79b4465b8d564741174b8e0 (table default.Table_e4d0e9336a834d6d8ab29ac0b1de7ace [id=af5b5b94fcef4677affe7e7afaa7f023]) successfully deleted
I20251027 21:07:54.324872 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 305) took 1978 ms. Trace:
I20251027 21:07:54.325392 10040 rpcz_store.cc:276] 1027 21:07:52.346653 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.378207 (+ 31554us) service_pool.cc:225] Handling call
1027 21:07:54.324709 (+1946502us) ts_tablet_manager.cc:1151] Deleting tablet 3103c8e5a7c744be90be097179904ba3
1027 21:07:54.324719 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:54.324831 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1946238}
W20251027 21:07:54.326169 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 3103c8e5a7c744be90be097179904ba3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3103c8e5a7c744be90be097179904ba3 already in progress: creating tablet
I20251027 21:07:54.326653 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 306) took 1932 ms. Trace:
I20251027 21:07:54.297712 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 49b3d23798b740008c98cd0754ae33f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:54.329084 10040 rpcz_store.cc:276] 1027 21:07:52.394025 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.394222 (+ 197us) service_pool.cc:225] Handling call
1027 21:07:54.326524 (+1932302us) ts_tablet_manager.cc:1151] Deleting tablet 533f231e3ff64531ae2331409a961425
1027 21:07:54.326533 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:54.326616 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1932043}
I20251027 21:07:54.329563 10041 ts_tablet_manager.cc:1929] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251027 21:07:54.329632 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 533f231e3ff64531ae2331409a961425 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 533f231e3ff64531ae2331409a961425 already in progress: creating tablet
I20251027 21:07:54.330019 10040 tablet_replica.cc:333] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:54.330471 10040 raft_consensus.cc:2243] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.331261 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet 288b88e13fdf4427aae53f64b87733bf from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:54.332386 10040 raft_consensus.cc:2272] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:54.333149 10040 ts_tablet_manager.cc:1916] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:54.331404 10032 tablet_bootstrap.cc:492] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:54.345216 10032 ts_tablet_manager.cc:1403] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.000s
I20251027 21:07:54.345570 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet f775a7e42bdc4910bdb604b26a28a7f4 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:54.346040 10032 raft_consensus.cc:359] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.346330 10032 raft_consensus.cc:385] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.346446 10031 tablet_bootstrap.cc:492] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:54.346470 10032 raft_consensus.cc:740] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.346628 10031 ts_tablet_manager.cc:1403] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.065s user 0.001s sys 0.005s
I20251027 21:07:54.346786 10032 consensus_queue.cc:260] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.347368 10031 raft_consensus.cc:359] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.347577 10031 raft_consensus.cc:385] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.347702 10031 raft_consensus.cc:740] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.347975 10031 consensus_queue.cc:260] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.348429 10032 ts_tablet_manager.cc:1434] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:07:54.348814 10032 tablet_bootstrap.cc:492] T 6db0f31c12d14e66a4ec7fc722d62c3d P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:54.350450 10032 tablet_bootstrap.cc:654] T 6db0f31c12d14e66a4ec7fc722d62c3d P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.354310 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b247829a11349b6835a2f18906e20a9 with delete_type TABLET_DATA_DELETED (replaced by tablet fde0980aa64c4496861a662f7ed3ef75) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:54.354511 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b6b952a4a4634e21a51d3dacc8f4431f" } new_table_name: "default.Table_7929c591af15477aa34ef5952495dfc6" modify_external_catalogs: true
I20251027 21:07:54.356575 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_35ea0e752f924d568bc7faf8412fe091" } new_table_name: "default.Table_ef27c46c59a94abfbc41c240b0477362" modify_external_catalogs: true
I20251027 21:07:54.357393 10041 log.cc:1199] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/8fcfa6b85e7b46d78a84947e9f11ae46
I20251027 21:07:54.357854 10041 ts_tablet_manager.cc:1950] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:54.358883 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 8fcfa6b85e7b46d78a84947e9f11ae46 (table default.Table_4723d51b934b4f6a90c790ee4bc7ea86 [id=e68b63cf420d49289a5bd1db81e1cbc1]) successfully deleted
W20251027 21:07:54.359545 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 04c579bb008448e9baa841135b6dbc61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04c579bb008448e9baa841135b6dbc61 already in progress: creating tablet
W20251027 21:07:54.360486 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 76e4fb2bea7f4a4696fe9f6bb892bccc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76e4fb2bea7f4a4696fe9f6bb892bccc already in progress: creating tablet
W20251027 21:07:54.361428 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 9861a0efcba64141aae450663c5d83de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9861a0efcba64141aae450663c5d83de already in progress: creating tablet
W20251027 21:07:54.362006 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet cb95618d9fd54a4f8121b43e717d5616 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb95618d9fd54a4f8121b43e717d5616 already in progress: creating tablet
W20251027 21:07:54.362607 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 59b6818d29624da5a49dd1a7d20994c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59b6818d29624da5a49dd1a7d20994c6 already in progress: creating tablet
W20251027 21:07:54.363193 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet d52d71f4976143e4a514350621ebe309 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d52d71f4976143e4a514350621ebe309 already in progress: creating tablet
W20251027 21:07:54.363406 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 618f1657a10947fd98976820564c20b3 has no consensus state
I20251027 21:07:54.363509 10041 tablet_replica.cc:333] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:54.363698 10041 raft_consensus.cc:2243] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.364281 10041 raft_consensus.cc:2272] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:54.365069 10041 ts_tablet_manager.cc:1916] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:54.366261 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4081f3b9db1e4af2a1a8faa1d337461e" } new_table_name: "default.Table_3b49d647ca194ad9bf5524812c6837a2" modify_external_catalogs: true
W20251027 21:07:54.374049 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64e39cc210664c3d83eaf1fc576f83ce has no consensus state
I20251027 21:07:54.375270 10029 tablet_bootstrap.cc:492] T c49775486ff24686b0f45c47e2757aba P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:54.375502 10029 ts_tablet_manager.cc:1403] T c49775486ff24686b0f45c47e2757aba P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.257s user 0.021s sys 0.011s
I20251027 21:07:54.376259 10029 raft_consensus.cc:359] T c49775486ff24686b0f45c47e2757aba P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.376509 10029 raft_consensus.cc:385] T c49775486ff24686b0f45c47e2757aba P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.376632 10029 raft_consensus.cc:740] T c49775486ff24686b0f45c47e2757aba P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.376941 10029 consensus_queue.cc:260] T c49775486ff24686b0f45c47e2757aba P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.377514 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_907ada415b694cf4817ef1e6f6f68e9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:54.378125 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_907ada415b694cf4817ef1e6f6f68e9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:54.380636 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9351a369dc1e4ac49f6ad419eab3a4c5 has no consensus state
I20251027 21:07:54.380678 10041 ts_tablet_manager.cc:1929] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.381352 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_602a5846b60c4860b33575da44b5015b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.381515 10029 ts_tablet_manager.cc:1434] T c49775486ff24686b0f45c47e2757aba P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
W20251027 21:07:54.381945 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_602a5846b60c4860b33575da44b5015b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.385462 10029 tablet_bootstrap.cc:492] T 1e5f5b57f5d948f3b2352451ccaacb09 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:54.387100 10029 tablet_bootstrap.cc:654] T 1e5f5b57f5d948f3b2352451ccaacb09 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.391402 10041 log.cc:1199] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/49d9e5c5960a4534a6603875006e6266
I20251027 21:07:54.391958 10041 ts_tablet_manager.cc:1950] T 49d9e5c5960a4534a6603875006e6266 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:54.393271 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 49d9e5c5960a4534a6603875006e6266 (table default.Table_52c53bafe66b4c66b99e5b26e4df7ef5 [id=27dc835f316f41e091be9b2b1f560bbf]) successfully deleted
I20251027 21:07:54.395376 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_011f55ad6c164cdd8753c07e2359334c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.395814 10041 tablet_replica.cc:333] T 94e94cd50c91429d8da9804149501c86 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:54.396147 10041 raft_consensus.cc:2243] T 94e94cd50c91429d8da9804149501c86 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.396641 10041 raft_consensus.cc:2272] T 94e94cd50c91429d8da9804149501c86 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:07:54.398623 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_011f55ad6c164cdd8753c07e2359334c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.400139 10041 ts_tablet_manager.cc:1916] T 94e94cd50c91429d8da9804149501c86 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:54.400476 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3ed917cd5e964f3b8f54abfcf705aba7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:54.401044 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ed917cd5e964f3b8f54abfcf705aba7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.403656 10029 tablet_bootstrap.cc:492] T 1e5f5b57f5d948f3b2352451ccaacb09 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:54.403831 10029 ts_tablet_manager.cc:1403] T 1e5f5b57f5d948f3b2352451ccaacb09 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.005s
I20251027 21:07:54.404596 10029 raft_consensus.cc:359] T 1e5f5b57f5d948f3b2352451ccaacb09 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.404841 10029 raft_consensus.cc:385] T 1e5f5b57f5d948f3b2352451ccaacb09 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.405009 10029 raft_consensus.cc:740] T 1e5f5b57f5d948f3b2352451ccaacb09 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.397588 10032 tablet_bootstrap.cc:492] T 6db0f31c12d14e66a4ec7fc722d62c3d P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:54.405421 10029 consensus_queue.cc:260] T 1e5f5b57f5d948f3b2352451ccaacb09 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.405673 10032 ts_tablet_manager.cc:1403] T 6db0f31c12d14e66a4ec7fc722d62c3d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.057s user 0.009s sys 0.003s
I20251027 21:07:54.406905 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_ff202033c9ec4fba9cc9525ea6767cac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.407016 10032 raft_consensus.cc:359] T 6db0f31c12d14e66a4ec7fc722d62c3d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.407217 10032 raft_consensus.cc:385] T 6db0f31c12d14e66a4ec7fc722d62c3d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.407282 10032 raft_consensus.cc:740] T 6db0f31c12d14e66a4ec7fc722d62c3d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
W20251027 21:07:54.407351 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff202033c9ec4fba9cc9525ea6767cac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.407523 10032 consensus_queue.cc:260] T 6db0f31c12d14e66a4ec7fc722d62c3d P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.408959 10032 ts_tablet_manager.cc:1434] T 6db0f31c12d14e66a4ec7fc722d62c3d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:07:54.409296 10032 tablet_bootstrap.cc:492] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:54.410318 10029 ts_tablet_manager.cc:1434] T 1e5f5b57f5d948f3b2352451ccaacb09 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251027 21:07:54.412421 10029 tablet_bootstrap.cc:492] T e07b8aa0a41241d79254c4dae8738e79 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:54.414109 10029 tablet_bootstrap.cc:654] T e07b8aa0a41241d79254c4dae8738e79 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.416380 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c8cb97c3834e406ca7d8a2d047d7d249"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.416862 10032 tablet_bootstrap.cc:654] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:54.417248 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8cb97c3834e406ca7d8a2d047d7d249 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.424376 10040 ts_tablet_manager.cc:1929] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.424331 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a328d75fa0f44ad9b92e05cd96556f06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:54.424786 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a328d75fa0f44ad9b92e05cd96556f06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.427585 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7d20b20b1f274f4f94a9497e2d030235"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:54.428117 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d20b20b1f274f4f94a9497e2d030235 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.430126 10031 ts_tablet_manager.cc:1434] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.074s user 0.001s sys 0.000s
I20251027 21:07:54.430191 10040 log.cc:1199] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/94a00ff844f0415fa419613d70a318e6
I20251027 21:07:54.430698 10040 ts_tablet_manager.cc:1950] T 94a00ff844f0415fa419613d70a318e6 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:54.430760 10031 tablet_bootstrap.cc:492] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:54.431088 10411 raft_consensus.cc:493] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:54.431238 10411 raft_consensus.cc:515] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.431890 10411 leader_election.cc:290] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:54.432526 10031 tablet_bootstrap.cc:654] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.432739 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b87e809ff897417c88e5edc19e2d591e" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:54.433696 9860 leader_election.cc:343] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:54.434764 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 308) took 2037 ms. Trace:
I20251027 21:07:54.434880 10040 rpcz_store.cc:276] 1027 21:07:52.397246 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.397295 (+ 49us) service_pool.cc:225] Handling call
1027 21:07:54.329956 (+1932661us) ts_tablet_manager.cc:1151] Deleting tablet 94a00ff844f0415fa419613d70a318e6
1027 21:07:54.329965 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:54.350081 (+ 20116us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:54.424264 (+ 74183us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:54.434698 (+ 10434us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1932495}
I20251027 21:07:54.435108 10041 ts_tablet_manager.cc:1929] T 94e94cd50c91429d8da9804149501c86 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.435220 10041 log.cc:1199] T 94e94cd50c91429d8da9804149501c86 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/94e94cd50c91429d8da9804149501c86
I20251027 21:07:54.435335 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 94a00ff844f0415fa419613d70a318e6 (table default.Table_89407f0c58ec4ee7995b892701a6377f [id=f773d65a07a4473ab8b87cd82ac50bcf]) successfully deleted
I20251027 21:07:54.435570 10041 ts_tablet_manager.cc:1950] T 94e94cd50c91429d8da9804149501c86 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:54.435583 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b87e809ff897417c88e5edc19e2d591e" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:54.436076 9859 leader_election.cc:343] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:54.436196 9859 leader_election.cc:304] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:54.436560 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 94e94cd50c91429d8da9804149501c86 (table default.Table_b23313ec4d7b4e32a4283dd404578616 [id=6a545a959eb74d0eb589f1be92a0bc0a]) successfully deleted
I20251027 21:07:54.436826 10411 raft_consensus.cc:2749] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:54.437160 10040 tablet_replica.cc:333] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:54.437404 10040 raft_consensus.cc:2243] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.437906 10040 raft_consensus.cc:2272] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:54.438494 10040 ts_tablet_manager.cc:1916] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:54.438649 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6a4f975247ee46deac07892f07bf19e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:54.439152 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a4f975247ee46deac07892f07bf19e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.439373 10032 tablet_bootstrap.cc:492] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:54.439565 10032 ts_tablet_manager.cc:1403] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20251027 21:07:54.440354 10032 raft_consensus.cc:359] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.440574 10032 raft_consensus.cc:385] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.440707 10032 raft_consensus.cc:740] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.440961 10032 consensus_queue.cc:260] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.442428 10032 ts_tablet_manager.cc:1434] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251027 21:07:54.442749 10032 tablet_bootstrap.cc:492] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:54.444622 10032 tablet_bootstrap.cc:654] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.445679 10041 tablet_replica.cc:333] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:54.445938 10041 raft_consensus.cc:2243] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.448503 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4803a53377dc41709e7d86dc239bcea7" } new_table_name: "default.Table_d8da07f0725347338012d992737c5d04" modify_external_catalogs: true
I20251027 21:07:54.451135 10041 raft_consensus.cc:2272] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:54.452041 10041 ts_tablet_manager.cc:1916] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:54.463390 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aaa8910d9fdb4146945621ec4ebf969e consensus state has no leader
I20251027 21:07:54.465562 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05
I20251027 21:07:54.466784 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:54.467480 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:54.467602 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:54.468293 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 already in progress: creating tablet
I20251027 21:07:54.468920 9311 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 1e5f5b57f5d948f3b2352451ccaacb09 (table default.Table_f0792045d49041578f5ff6459c99f77e [id=7f83f7c0f9524147bb778a06dd233a8e]). Expected version 1 got 0
I20251027 21:07:54.469960 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet c49775486ff24686b0f45c47e2757aba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:54.470291 10043 tablet_replica.cc:333] T c49775486ff24686b0f45c47e2757aba P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:54.470497 10043 raft_consensus.cc:2243] T c49775486ff24686b0f45c47e2757aba P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.470680 10043 raft_consensus.cc:2272] T c49775486ff24686b0f45c47e2757aba P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:54.471550 10043 ts_tablet_manager.cc:1916] T c49775486ff24686b0f45c47e2757aba P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:54.473786 9305 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 995284883d834bbb9ef583b736ca8402 (table default.Table_3895a18e2b7a4456a3681679d7c8ba4f [id=186c9cce53d74040b9799ea386fdffd7]). Expected version 3 got 0
I20251027 21:07:54.476943 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet 583fba7c750e4ac7b768da447002c211 with delete_type TABLET_DATA_DELETED (replaced by tablet f34512059f0c47fc9af7db2645f86b40) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:54.477202 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet b87e809ff897417c88e5edc19e2d591e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:54.477479 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet 6db0f31c12d14e66a4ec7fc722d62c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:54.479092 10043 ts_tablet_manager.cc:1929] T c49775486ff24686b0f45c47e2757aba P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.479305 10043 log.cc:1199] T c49775486ff24686b0f45c47e2757aba P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/c49775486ff24686b0f45c47e2757aba
I20251027 21:07:54.479714 10043 ts_tablet_manager.cc:1950] T c49775486ff24686b0f45c47e2757aba P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:54.480809 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet c49775486ff24686b0f45c47e2757aba (table default.Table_b534fdb9e3084d6f9656dfdbb0e01750 [id=dcefc291ce454343964e99be8876837c]) successfully deleted
I20251027 21:07:54.481573 10029 tablet_bootstrap.cc:492] T e07b8aa0a41241d79254c4dae8738e79 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:54.481840 10029 ts_tablet_manager.cc:1403] T e07b8aa0a41241d79254c4dae8738e79 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.005s
I20251027 21:07:54.482483 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 334c580862bd4bf09be3f1710aa6a092 (DEFAULT_TABLE table=default.Table_3ac9f7acd5ac43f1a585c29ecae682eb [id=cf9029e184114ac0a5b641a496701122]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.482878 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 334c580862bd4bf09be3f1710aa6a092. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.483656 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 334c580862bd4bf09be3f1710aa6a092 (DEFAULT_TABLE table=default.Table_3ac9f7acd5ac43f1a585c29ecae682eb [id=cf9029e184114ac0a5b641a496701122]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.484021 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 334c580862bd4bf09be3f1710aa6a092. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.484284 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 334c580862bd4bf09be3f1710aa6a092 (DEFAULT_TABLE table=default.Table_3ac9f7acd5ac43f1a585c29ecae682eb [id=cf9029e184114ac0a5b641a496701122]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.484696 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 334c580862bd4bf09be3f1710aa6a092. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.489249 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 6210d3d10fc44b9e9e2b0780883186ce (DEFAULT_TABLE table=default.Table_f4272063203347a5810ed07554c2269b [id=20a2daa7746f40e69803fc010dc4bcbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.489586 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6210d3d10fc44b9e9e2b0780883186ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.489620 10040 ts_tablet_manager.cc:1929] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.489724 10040 log.cc:1199] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/61cd8ee8e1f8459eb7af5dd48c018e02
I20251027 21:07:54.490612 10040 ts_tablet_manager.cc:1950] T 61cd8ee8e1f8459eb7af5dd48c018e02 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:54.491384 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 309) took 2093 ms. Trace:
I20251027 21:07:54.491482 10040 rpcz_store.cc:276] 1027 21:07:52.397501 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.397539 (+ 38us) service_pool.cc:225] Handling call
1027 21:07:54.437080 (+2039541us) ts_tablet_manager.cc:1151] Deleting tablet 61cd8ee8e1f8459eb7af5dd48c018e02
1027 21:07:54.437089 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:54.453079 (+ 15990us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:54.456801 (+ 3722us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:54.491351 (+ 34550us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2039377}
I20251027 21:07:54.491905 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 61cd8ee8e1f8459eb7af5dd48c018e02 (table default.Table_52c53bafe66b4c66b99e5b26e4df7ef5 [id=27dc835f316f41e091be9b2b1f560bbf]) successfully deleted
I20251027 21:07:54.492192 10040 tablet_replica.cc:333] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:54.492368 10040 raft_consensus.cc:2243] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.492511 10040 raft_consensus.cc:2272] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:54.493002 10040 ts_tablet_manager.cc:1916] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:54.492956 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7a535e8483da496498c4c7f7e76d6670"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:54.493551 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a535e8483da496498c4c7f7e76d6670 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.495075 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6abd0422341f4d00a3dc8fe5a515e00a" } new_table_name: "default.Table_5502390635a249d5a5b5eaf0788236fd" modify_external_catalogs: true
I20251027 21:07:54.495150 10041 ts_tablet_manager.cc:1929] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.495311 10041 log.cc:1199] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/ae124514964440c5b88b16ac9f8a2567
I20251027 21:07:54.495666 10041 ts_tablet_manager.cc:1950] T ae124514964440c5b88b16ac9f8a2567 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:54.496307 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet a033551a591146039b017b4193b221ca
I20251027 21:07:54.496639 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 9ed4f9c61ad54e1292281b30bcd19533
I20251027 21:07:54.496686 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet 53cac7817db440e79e7416d625a9b2c8 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:54.497527 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ed4f9c61ad54e1292281b30bcd19533 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:54.497910 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet a033551a591146039b017b4193b221ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:54.498116 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ed4f9c61ad54e1292281b30bcd19533 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:54.499076 10029 raft_consensus.cc:359] T e07b8aa0a41241d79254c4dae8738e79 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.499295 10029 raft_consensus.cc:385] T e07b8aa0a41241d79254c4dae8738e79 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.499374 10029 raft_consensus.cc:740] T e07b8aa0a41241d79254c4dae8738e79 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.499581 10029 consensus_queue.cc:260] T e07b8aa0a41241d79254c4dae8738e79 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.502038 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ed4f9c61ad54e1292281b30bcd19533 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:54.502480 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet ae124514964440c5b88b16ac9f8a2567 (table default.Table_11fbadd9eed14cdba7664715e389f7e5 [id=83ec44dba5c444aba48846811a5f5fc9]) successfully deleted
I20251027 21:07:54.503980 10032 tablet_bootstrap.cc:492] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:54.504209 10032 ts_tablet_manager.cc:1403] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.003s
I20251027 21:07:54.504468 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet a033551a591146039b017b4193b221ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:54.505172 10032 raft_consensus.cc:359] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.505429 10032 raft_consensus.cc:385] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.505550 10032 raft_consensus.cc:740] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.505879 10032 consensus_queue.cc:260] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.508014 10029 ts_tablet_manager.cc:1434] T e07b8aa0a41241d79254c4dae8738e79 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251027 21:07:54.508343 10029 tablet_bootstrap.cc:492] T 779b7f63b0fa4f9ab47cf5799b3e085d P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:54.509331 9309 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 8fc2d7d6601d4cc2849064f4cdddc90d (table default.Table_f343a39b880b4c7a827bff89fd3fe411 [id=6f39910d23dd455f87a4ac847917b87f]). Expected version 4 got 0
I20251027 21:07:54.510589 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e7df6f0acec34d399aa25b78c3ed7c7a
I20251027 21:07:54.510980 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet f1aa5f7f76cb4a1ba15f016942f17067
W20251027 21:07:54.511178 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 22d861810ed846c3b77a452d05e93d72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22d861810ed846c3b77a452d05e93d72 already in progress: creating tablet
I20251027 21:07:54.511991 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet f1aa5f7f76cb4a1ba15f016942f17067 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:54.512029 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet e7df6f0acec34d399aa25b78c3ed7c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:54.512323 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet e7df6f0acec34d399aa25b78c3ed7c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:54.512315 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet e7df6f0acec34d399aa25b78c3ed7c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:54.512531 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet f1aa5f7f76cb4a1ba15f016942f17067 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:54.512708 10032 ts_tablet_manager.cc:1434] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251027 21:07:54.513139 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet f775a7e42bdc4910bdb604b26a28a7f4
W20251027 21:07:54.513305 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 already in progress: creating tablet
I20251027 21:07:54.513628 9307 catalog_manager.cc:6719] ReplaceTablet: tablet f775a7e42bdc4910bdb604b26a28a7f4 deleted and replaced by tablet 11627a84ff9b478c87c323617a09570b
I20251027 21:07:54.513947 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet f775a7e42bdc4910bdb604b26a28a7f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 11627a84ff9b478c87c323617a09570b) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:54.514534 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet f775a7e42bdc4910bdb604b26a28a7f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 11627a84ff9b478c87c323617a09570b) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:54.517273 10031 tablet_bootstrap.cc:492] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:54.517509 10031 ts_tablet_manager.cc:1403] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.087s user 0.006s sys 0.003s
I20251027 21:07:54.517886 9307 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 8fc2d7d6601d4cc2849064f4cdddc90d (table default.Table_f343a39b880b4c7a827bff89fd3fe411 [id=6f39910d23dd455f87a4ac847917b87f]). Expected version 4 got 0
I20251027 21:07:54.518599 10031 raft_consensus.cc:359] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.518862 10031 raft_consensus.cc:385] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.518987 10031 raft_consensus.cc:740] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.516669 10032 tablet_bootstrap.cc:492] T 9c239216cb8846f0b1aaa360ddb83321 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:54.519343 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 6210d3d10fc44b9e9e2b0780883186ce (DEFAULT_TABLE table=default.Table_f4272063203347a5810ed07554c2269b [id=20a2daa7746f40e69803fc010dc4bcbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.516990 10041 tablet_replica.cc:333] T 583fba7c750e4ac7b768da447002c211 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:54.519861 10031 consensus_queue.cc:260] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.519984 10041 raft_consensus.cc:2243] T 583fba7c750e4ac7b768da447002c211 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.522431 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 6210d3d10fc44b9e9e2b0780883186ce (DEFAULT_TABLE table=default.Table_f4272063203347a5810ed07554c2269b [id=20a2daa7746f40e69803fc010dc4bcbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.522809 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6210d3d10fc44b9e9e2b0780883186ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.525568 10041 raft_consensus.cc:2272] T 583fba7c750e4ac7b768da447002c211 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:54.520181 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6210d3d10fc44b9e9e2b0780883186ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.530925 10041 ts_tablet_manager.cc:1916] T 583fba7c750e4ac7b768da447002c211 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:54.531204 10031 ts_tablet_manager.cc:1434] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251027 21:07:54.532547 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet c49775486ff24686b0f45c47e2757aba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:54.531114 10411 raft_consensus.cc:493] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:54.532894 10411 raft_consensus.cc:515] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.533905 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 465be1b24d16457e834023a2b2194049 (DEFAULT_TABLE table=default.Table_185df62a04014137809bfc1bc3e5ecad [id=80175945bd1e436c92630ce62cc1c4ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.534461 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet f1aa5f7f76cb4a1ba15f016942f17067 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:54.534658 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet a2d26784bbb24ae3893103802c41affb from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:54.535583 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet f775a7e42bdc4910bdb604b26a28a7f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 11627a84ff9b478c87c323617a09570b) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:54.535687 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 9ed4f9c61ad54e1292281b30bcd19533 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ed4f9c61ad54e1292281b30bcd19533 already in progress: creating tablet
I20251027 21:07:54.535899 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "995284883d834bbb9ef583b736ca8402" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:54.536491 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 465be1b24d16457e834023a2b2194049 (DEFAULT_TABLE table=default.Table_185df62a04014137809bfc1bc3e5ecad [id=80175945bd1e436c92630ce62cc1c4ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.536857 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 465be1b24d16457e834023a2b2194049. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:54.537047 9298 catalog_manager.cc:4952] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet e7df6f0acec34d399aa25b78c3ed7c7a because the tablet was not found. No further retry: Not found: Tablet not found: e7df6f0acec34d399aa25b78c3ed7c7a
W20251027 21:07:54.539927 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet f1aa5f7f76cb4a1ba15f016942f17067 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1aa5f7f76cb4a1ba15f016942f17067 already in progress: creating tablet
W20251027 21:07:54.542032 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93d2f7df14a74731aea84927f0b7ed45 consensus state has no leader
I20251027 21:07:54.542229 10411 leader_election.cc:290] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:54.542698 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 465be1b24d16457e834023a2b2194049 (DEFAULT_TABLE table=default.Table_185df62a04014137809bfc1bc3e5ecad [id=80175945bd1e436c92630ce62cc1c4ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.542923 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet e07b8aa0a41241d79254c4dae8738e79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:54.543093 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 465be1b24d16457e834023a2b2194049. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.543200 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet a033551a591146039b017b4193b221ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:54.543224 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 53cac7817db440e79e7416d625a9b2c8
I20251027 21:07:54.543915 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet 53cac7817db440e79e7416d625a9b2c8 with delete_type TABLET_DATA_DELETED (replaced by tablet ed66f9eeeb2b4b4bb77062c241d2140f) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:54.543936 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet f775a7e42bdc4910bdb604b26a28a7f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f775a7e42bdc4910bdb604b26a28a7f4 already in progress: creating tablet
I20251027 21:07:54.544270 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet 53cac7817db440e79e7416d625a9b2c8 with delete_type TABLET_DATA_DELETED (replaced by tablet ed66f9eeeb2b4b4bb77062c241d2140f) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:54.544382 10043 tablet_replica.cc:333] T e07b8aa0a41241d79254c4dae8738e79 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:54.544399 9311 catalog_manager.cc:6719] ReplaceTablet: tablet 53cac7817db440e79e7416d625a9b2c8 deleted and replaced by tablet ed66f9eeeb2b4b4bb77062c241d2140f
I20251027 21:07:54.544588 10043 raft_consensus.cc:2243] T e07b8aa0a41241d79254c4dae8738e79 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.544770 10043 raft_consensus.cc:2272] T e07b8aa0a41241d79254c4dae8738e79 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:54.544967 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 465be1b24d16457e834023a2b2194049. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.545943 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 53cac7817db440e79e7416d625a9b2c8 with delete_type TABLET_DATA_DELETED (replaced by tablet ed66f9eeeb2b4b4bb77062c241d2140f) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:54.546213 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "995284883d834bbb9ef583b736ca8402" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:54.546366 9860 leader_election.cc:343] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:54.546954 10029 tablet_bootstrap.cc:654] T 779b7f63b0fa4f9ab47cf5799b3e085d P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:54.550436 9859 leader_election.cc:343] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:54.550578 9859 leader_election.cc:304] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:54.550805 10411 raft_consensus.cc:2749] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:54.553539 10031 tablet_bootstrap.cc:492] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:54.555065 10031 tablet_bootstrap.cc:654] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.555583 10043 ts_tablet_manager.cc:1916] T e07b8aa0a41241d79254c4dae8738e79 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:54.556701 10041 ts_tablet_manager.cc:1929] T 583fba7c750e4ac7b768da447002c211 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.557619 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet a2d26784bbb24ae3893103802c41affb
I20251027 21:07:54.558992 9307 catalog_manager.cc:6719] ReplaceTablet: tablet a2d26784bbb24ae3893103802c41affb deleted and replaced by tablet 9ace4188d3494a05a5678381f8d8fd17
I20251027 21:07:54.559808 9600 tablet_service.cc:1505] Processing CreateTablet for tablet d8691aa22fe84cf29893ed4e79d156cc (DEFAULT_TABLE table=default.Table_da0e93f761894c259f883d648d7f25d8 [id=a3955ead580245ebaad587dbe2d98a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.560242 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8691aa22fe84cf29893ed4e79d156cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.563980 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet a2d26784bbb24ae3893103802c41affb with delete_type TABLET_DATA_DELETED (replaced by tablet 9ace4188d3494a05a5678381f8d8fd17) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:54.556890 10032 tablet_bootstrap.cc:654] T 9c239216cb8846f0b1aaa360ddb83321 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.557855 9890 tablet_service.cc:1505] Processing CreateTablet for tablet d8691aa22fe84cf29893ed4e79d156cc (DEFAULT_TABLE table=default.Table_da0e93f761894c259f883d648d7f25d8 [id=a3955ead580245ebaad587dbe2d98a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.558454 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet a2d26784bbb24ae3893103802c41affb with delete_type TABLET_DATA_DELETED (replaced by tablet 9ace4188d3494a05a5678381f8d8fd17) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:54.559020 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet a2d26784bbb24ae3893103802c41affb with delete_type TABLET_DATA_DELETED (replaced by tablet 9ace4188d3494a05a5678381f8d8fd17) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:54.562093 9740 tablet_service.cc:1505] Processing CreateTablet for tablet d8691aa22fe84cf29893ed4e79d156cc (DEFAULT_TABLE table=default.Table_da0e93f761894c259f883d648d7f25d8 [id=a3955ead580245ebaad587dbe2d98a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.565696 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8691aa22fe84cf29893ed4e79d156cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.565699 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8691aa22fe84cf29893ed4e79d156cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.566249 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_89407f0c58ec4ee7995b892701a6377f" } new_table_name: "default.Table_0432e7918b494dccb2f6b921657c0d81" modify_external_catalogs: true
I20251027 21:07:54.569628 10041 log.cc:1199] T 583fba7c750e4ac7b768da447002c211 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/583fba7c750e4ac7b768da447002c211
I20251027 21:07:54.570215 10041 ts_tablet_manager.cc:1950] T 583fba7c750e4ac7b768da447002c211 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:54.571632 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 583fba7c750e4ac7b768da447002c211 (table default.Table_660eb77e69884fe5bd5e65b47ea91dde [id=14daa3e199fe4fa3ba2de06dd6be4330]) successfully deleted
I20251027 21:07:54.572088 10414 catalog_manager.cc:1269] Loaded cluster ID: 074cb2d5dc574bb39dc828c9d64a191a
I20251027 21:07:54.572201 10414 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: loading cluster ID for follower catalog manager: success
I20251027 21:07:54.574551 10041 tablet_replica.cc:333] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:54.576143 10041 raft_consensus.cc:2243] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.576925 10041 raft_consensus.cc:2272] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:54.577898 10041 ts_tablet_manager.cc:1916] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:54.578948 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d2203ee62f7e404c95c5f86b5ae85feb consensus state has no leader
I20251027 21:07:54.579607 10040 ts_tablet_manager.cc:1929] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.579666 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2953868ab4054f75a9975b60db2e5e0a" } new_table_name: "default.Table_d985344dc40f440cab5f2b52c2ba4e27" modify_external_catalogs: true
I20251027 21:07:54.580777 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 76bee322d0b94abe9166170733608853 (DEFAULT_TABLE table=default.Table_274350c56d3a43a4b3738ced49d24150 [id=c9e7aa435d5b48e096c2b67b0e678eeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.581171 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76bee322d0b94abe9166170733608853. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.584201 9614 tablet_service.cc:1505] Processing CreateTablet for tablet e1e58f6228254e3eb25dbf627a505ca1 (DEFAULT_TABLE table=default.Table_3e9f6e0f115040639f405946e55a2637 [id=dd2a4ed46c1c4a6c8687547e809f2677]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.584558 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1e58f6228254e3eb25dbf627a505ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.582784 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 76bee322d0b94abe9166170733608853 (DEFAULT_TABLE table=default.Table_274350c56d3a43a4b3738ced49d24150 [id=c9e7aa435d5b48e096c2b67b0e678eeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.586849 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76bee322d0b94abe9166170733608853. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.592324 10029 tablet_bootstrap.cc:492] T 779b7f63b0fa4f9ab47cf5799b3e085d P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:54.592811 10041 ts_tablet_manager.cc:1929] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.593092 10029 ts_tablet_manager.cc:1403] T 779b7f63b0fa4f9ab47cf5799b3e085d P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.085s user 0.002s sys 0.002s
I20251027 21:07:54.594053 10029 raft_consensus.cc:359] T 779b7f63b0fa4f9ab47cf5799b3e085d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.595185 10029 raft_consensus.cc:385] T 779b7f63b0fa4f9ab47cf5799b3e085d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.595436 10029 raft_consensus.cc:740] T 779b7f63b0fa4f9ab47cf5799b3e085d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
W20251027 21:07:54.595930 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 83f5fbf5757a49df93ab949b7657c61f consensus state has no leader
I20251027 21:07:54.595866 10029 consensus_queue.cc:260] T 779b7f63b0fa4f9ab47cf5799b3e085d P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.598099 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_88e8bdcff395438e9684379b653b965e" } new_table_name: "default.Table_4755cbfcd1464ec1b6f0b909b99f0c3f" modify_external_catalogs: true
I20251027 21:07:54.600461 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c260261e06b74bf097a23ef87e39eccd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.600878 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 42371496a7494e77a7bf1169de5b083a (DEFAULT_TABLE table=default.Table_6f3e1c9642e3433eab3f8e5a409470c4 [id=da9123bd1cc24db8b3be1d53b992292f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:54.601104 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c260261e06b74bf097a23ef87e39eccd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.601218 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42371496a7494e77a7bf1169de5b083a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.603737 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet 779b7f63b0fa4f9ab47cf5799b3e085d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:54.604328 10029 ts_tablet_manager.cc:1434] T 779b7f63b0fa4f9ab47cf5799b3e085d P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20251027 21:07:54.594875 9889 tablet_service.cc:1505] Processing CreateTablet for tablet e1e58f6228254e3eb25dbf627a505ca1 (DEFAULT_TABLE table=default.Table_3e9f6e0f115040639f405946e55a2637 [id=dd2a4ed46c1c4a6c8687547e809f2677]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.606453 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1e58f6228254e3eb25dbf627a505ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.607195 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b0eb812d94f1489892d520a5b4df323c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:54.607712 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0eb812d94f1489892d520a5b4df323c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.608644 10358 raft_consensus.cc:493] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:54.608777 10358 raft_consensus.cc:515] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.609323 10358 leader_election.cc:290] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:54.595703 10032 tablet_bootstrap.cc:492] T 9c239216cb8846f0b1aaa360ddb83321 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:54.609527 10032 ts_tablet_manager.cc:1403] T 9c239216cb8846f0b1aaa360ddb83321 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.093s user 0.005s sys 0.002s
I20251027 21:07:54.610313 10032 raft_consensus.cc:359] T 9c239216cb8846f0b1aaa360ddb83321 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.610508 10032 raft_consensus.cc:385] T 9c239216cb8846f0b1aaa360ddb83321 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.610594 10032 raft_consensus.cc:740] T 9c239216cb8846f0b1aaa360ddb83321 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.610806 10032 consensus_queue.cc:260] T 9c239216cb8846f0b1aaa360ddb83321 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.611241 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb1686fc7c3a4996aa4462884d30c41f" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:54.611464 9924 raft_consensus.cc:2468] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:54.611414 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 00c90ff84d3f4a7f92fda235416eed1d (DEFAULT_TABLE table=default.Table_b36a756b7ab54252aaa0c08e563b2ea3 [id=14a74e3dd6ef4f3ba0ef5f190c0f393c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.611732 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00c90ff84d3f4a7f92fda235416eed1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.611302 10032 ts_tablet_manager.cc:1434] T 9c239216cb8846f0b1aaa360ddb83321 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:54.612119 9568 leader_election.cc:304] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:07:54.612432 10032 tablet_bootstrap.cc:492] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:54.612485 10358 raft_consensus.cc:2804] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:54.612557 10358 raft_consensus.cc:493] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:54.612645 10358 raft_consensus.cc:3060] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:54.613245 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 00c90ff84d3f4a7f92fda235416eed1d (DEFAULT_TABLE table=default.Table_b36a756b7ab54252aaa0c08e563b2ea3 [id=14a74e3dd6ef4f3ba0ef5f190c0f393c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.613621 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00c90ff84d3f4a7f92fda235416eed1d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:54.614217 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 84b9de82d55447bba2a897edb683e5c5 consensus state has no leader
I20251027 21:07:54.614421 10032 tablet_bootstrap.cc:654] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.615386 10358 raft_consensus.cc:515] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.618875 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb1686fc7c3a4996aa4462884d30c41f" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:07:54.619110 9924 raft_consensus.cc:3060] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:54.620319 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 42371496a7494e77a7bf1169de5b083a (DEFAULT_TABLE table=default.Table_6f3e1c9642e3433eab3f8e5a409470c4 [id=da9123bd1cc24db8b3be1d53b992292f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.620774 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42371496a7494e77a7bf1169de5b083a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.622987 10358 leader_election.cc:290] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:54.624387 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 371051fb3491426e8cf7797317baa3db (DEFAULT_TABLE table=default.Table_6ef774fe4a464edc8b6ac91a3011fd41 [id=3b0044836ccc4b4082775de6eb111e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.624621 9924 raft_consensus.cc:2468] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:54.624771 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 371051fb3491426e8cf7797317baa3db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.625074 9568 leader_election.cc:304] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:07:54.625383 10358 raft_consensus.cc:2804] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:54.625836 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e2cb137bea0d419b9791305f97bc0efd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:54.626413 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2cb137bea0d419b9791305f97bc0efd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.627561 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b5b5182f58364f46849e1af83b27c6d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:54.627983 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5b5182f58364f46849e1af83b27c6d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.628847 10041 log.cc:1199] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/b87e809ff897417c88e5edc19e2d591e
I20251027 21:07:54.629233 10041 ts_tablet_manager.cc:1950] T b87e809ff897417c88e5edc19e2d591e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:54.630520 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet b87e809ff897417c88e5edc19e2d591e (table default.Table_d2bf88aca4cc44df99d13783ea2722f8 [id=7de1278fe67640ff8540392837b2eeae]) successfully deleted
I20251027 21:07:54.634582 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9d77e1ff90f047748d72fe89fbcc0c49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:54.635277 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d77e1ff90f047748d72fe89fbcc0c49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.636119 10041 tablet_replica.cc:333] T 6db0f31c12d14e66a4ec7fc722d62c3d P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:54.637297 10029 tablet_bootstrap.cc:492] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:54.638411 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9601abc37cc5482aa4b9a13b818e9074"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:54.638887 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9601abc37cc5482aa4b9a13b818e9074 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.642320 10041 raft_consensus.cc:2243] T 6db0f31c12d14e66a4ec7fc722d62c3d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.642568 10041 raft_consensus.cc:2272] T 6db0f31c12d14e66a4ec7fc722d62c3d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:54.642962 9306 master_service.cc:954] ReplaceTablet: received request to replace tablet 630ae4d66f99474382814cc32cc60ea0 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:54.643391 10041 ts_tablet_manager.cc:1916] T 6db0f31c12d14e66a4ec7fc722d62c3d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:54.648620 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c239216cb8846f0b1aaa360ddb83321 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:54.652853 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_10a096bf3cc34dc89007e7abeb182f61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.653647 10029 tablet_bootstrap.cc:654] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.582674 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 76bee322d0b94abe9166170733608853 (DEFAULT_TABLE table=default.Table_274350c56d3a43a4b3738ced49d24150 [id=c9e7aa435d5b48e096c2b67b0e678eeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.586851 10452 raft_consensus.cc:493] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:54.655735 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 371051fb3491426e8cf7797317baa3db (DEFAULT_TABLE table=default.Table_6ef774fe4a464edc8b6ac91a3011fd41 [id=3b0044836ccc4b4082775de6eb111e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.655920 10452 raft_consensus.cc:515] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.656210 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 371051fb3491426e8cf7797317baa3db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.656826 10452 leader_election.cc:290] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:54.658838 10414 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: acquiring CA information for follower catalog manager: success
W20251027 21:07:54.660624 9710 leader_election.cc:343] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 10a44aecfade42969454126b990eca24
I20251027 21:07:54.661188 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76bee322d0b94abe9166170733608853. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:54.661506 9711 leader_election.cc:343] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 10a44aecfade42969454126b990eca24
I20251027 21:07:54.661617 9711 leader_election.cc:304] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:54.662060 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 0e64d21abef342de86555e876f020787 (DEFAULT_TABLE table=default.Table_dac5ed8bed8045668aa636907042bf0a [id=cfdf2d0e9745403ea4bbeca6b4ed7439]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.662243 10452 raft_consensus.cc:2749] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:54.662621 10043 ts_tablet_manager.cc:1929] T e07b8aa0a41241d79254c4dae8738e79 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.663259 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e64d21abef342de86555e876f020787. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.664222 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 0e64d21abef342de86555e876f020787 (DEFAULT_TABLE table=default.Table_dac5ed8bed8045668aa636907042bf0a [id=cfdf2d0e9745403ea4bbeca6b4ed7439]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.664599 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e64d21abef342de86555e876f020787. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.665699 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 93a3b22cdc1543aebcc7fe72f2f2c0ae (DEFAULT_TABLE table=default.Table_96870e913d9243568d09f6f4cb2ccbb4 [id=8646c99b90414f4fa266cd7a31123d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.666489 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93a3b22cdc1543aebcc7fe72f2f2c0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.669128 10041 ts_tablet_manager.cc:1929] T 6db0f31c12d14e66a4ec7fc722d62c3d P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.670418 10041 log.cc:1199] T 6db0f31c12d14e66a4ec7fc722d62c3d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/6db0f31c12d14e66a4ec7fc722d62c3d
W20251027 21:07:54.671818 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10a096bf3cc34dc89007e7abeb182f61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.672322 10041 ts_tablet_manager.cc:1950] T 6db0f31c12d14e66a4ec7fc722d62c3d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:54.672595 10452 raft_consensus.cc:493] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:54.672727 10452 raft_consensus.cc:515] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.673492 10452 leader_election.cc:290] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:54.596637 9740 tablet_service.cc:1505] Processing CreateTablet for tablet e1e58f6228254e3eb25dbf627a505ca1 (DEFAULT_TABLE table=default.Table_3e9f6e0f115040639f405946e55a2637 [id=dd2a4ed46c1c4a6c8687547e809f2677]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.673874 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 371051fb3491426e8cf7797317baa3db (DEFAULT_TABLE table=default.Table_6ef774fe4a464edc8b6ac91a3011fd41 [id=3b0044836ccc4b4082775de6eb111e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.674284 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 371051fb3491426e8cf7797317baa3db. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:54.674294 9710 leader_election.cc:343] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 2b247829a11349b6835a2f18906e20a9
W20251027 21:07:54.674523 9711 leader_election.cc:343] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 2b247829a11349b6835a2f18906e20a9
I20251027 21:07:54.674679 9711 leader_election.cc:304] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:54.675225 10452 raft_consensus.cc:2749] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:54.674623 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1e58f6228254e3eb25dbf627a505ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.675740 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 6db0f31c12d14e66a4ec7fc722d62c3d (table default.Table_42431224890f43859abe3dedfae4db51 [id=a6784a116865476b85fe6ff9c32bb9db]) successfully deleted
W20251027 21:07:54.676430 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 9ed4f9c61ad54e1292281b30bcd19533 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ed4f9c61ad54e1292281b30bcd19533 already in progress: creating tablet
W20251027 21:07:54.677225 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet a033551a591146039b017b4193b221ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a033551a591146039b017b4193b221ca already in progress: creating tablet
I20251027 21:07:54.609524 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 42371496a7494e77a7bf1169de5b083a (DEFAULT_TABLE table=default.Table_6f3e1c9642e3433eab3f8e5a409470c4 [id=da9123bd1cc24db8b3be1d53b992292f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.609863 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb1686fc7c3a4996aa4462884d30c41f" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:54.678046 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet f1aa5f7f76cb4a1ba15f016942f17067 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1aa5f7f76cb4a1ba15f016942f17067 already in progress: creating tablet
I20251027 21:07:54.611842 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 00c90ff84d3f4a7f92fda235416eed1d (DEFAULT_TABLE table=default.Table_b36a756b7ab54252aaa0c08e563b2ea3 [id=14a74e3dd6ef4f3ba0ef5f190c0f393c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.616904 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb1686fc7c3a4996aa4462884d30c41f" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:54.651170 10359 raft_consensus.cc:493] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:54.678486 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00c90ff84d3f4a7f92fda235416eed1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.678360 10359 raft_consensus.cc:515] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:54.678619 9566 leader_election.cc:343] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:54.679059 10359 leader_election.cc:290] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:07:54.679994 9711 leader_election.cc:343] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 019a892bfdd54e62bc3560a341d5c2da
I20251027 21:07:54.683241 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 0e64d21abef342de86555e876f020787 (DEFAULT_TABLE table=default.Table_dac5ed8bed8045668aa636907042bf0a [id=cfdf2d0e9745403ea4bbeca6b4ed7439]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:54.683513 9710 leader_election.cc:343] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 019a892bfdd54e62bc3560a341d5c2da
I20251027 21:07:54.683631 9710 leader_election.cc:304] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:54.683777 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e64d21abef342de86555e876f020787. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:54.684077 9566 leader_election.cc:343] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:54.684448 10359 raft_consensus.cc:2749] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:07:54.686453 9299 catalog_manager.cc:4952] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet e7df6f0acec34d399aa25b78c3ed7c7a because the tablet was not found. No further retry: Not found: Tablet not found: e7df6f0acec34d399aa25b78c3ed7c7a
W20251027 21:07:54.687109 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet f775a7e42bdc4910bdb604b26a28a7f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f775a7e42bdc4910bdb604b26a28a7f4 already in progress: creating tablet
W20251027 21:07:54.687692 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 53cac7817db440e79e7416d625a9b2c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53cac7817db440e79e7416d625a9b2c8 already in progress: creating tablet
I20251027 21:07:54.687847 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42371496a7494e77a7bf1169de5b083a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.687953 10040 log.cc:1199] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/cd57440d3b2a4f37a8db7d1fa4bc4dbe
W20251027 21:07:54.688225 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet a2d26784bbb24ae3893103802c41affb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2d26784bbb24ae3893103802c41affb already in progress: creating tablet
I20251027 21:07:54.688352 10040 ts_tablet_manager.cc:1950] T cd57440d3b2a4f37a8db7d1fa4bc4dbe P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:54.688524 10041 tablet_replica.cc:333] T 9c239216cb8846f0b1aaa360ddb83321 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:54.688719 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 310) took 2290 ms. Trace:
I20251027 21:07:54.688767 10041 raft_consensus.cc:2243] T 9c239216cb8846f0b1aaa360ddb83321 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.688863 10040 rpcz_store.cc:276] 1027 21:07:52.397734 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.397769 (+ 35us) service_pool.cc:225] Handling call
1027 21:07:54.492136 (+2094367us) ts_tablet_manager.cc:1151] Deleting tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe
1027 21:07:54.492145 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:54.496205 (+ 4060us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:54.579509 (+ 83304us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:54.688672 (+109163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2094206}
I20251027 21:07:54.689342 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet cd57440d3b2a4f37a8db7d1fa4bc4dbe (table default.Table_4c5433970bd64291a2b1289a84f35590 [id=a1b2d46db3764c7f97712a0c5f891881]) successfully deleted
I20251027 21:07:54.689779 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 307) took 2293 ms. Trace:
I20251027 21:07:54.689917 10040 rpcz_store.cc:276] 1027 21:07:52.396092 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.397813 (+ 1721us) service_pool.cc:225] Handling call
1027 21:07:54.689621 (+2291808us) ts_tablet_manager.cc:1151] Deleting tablet 6bf9c2a78aac42528d67846dc58adc40
1027 21:07:54.689629 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:54.689725 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2962,"tablet-delete.queue_time_us":2291686}
I20251027 21:07:54.690297 10041 raft_consensus.cc:2272] T 9c239216cb8846f0b1aaa360ddb83321 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:07:54.690397 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 6bf9c2a78aac42528d67846dc58adc40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bf9c2a78aac42528d67846dc58adc40 already in progress: creating tablet
I20251027 21:07:54.690718 10040 tablet_replica.cc:333] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:54.690943 10040 raft_consensus.cc:2243] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:07:54.691242 10040 pending_rounds.cc:70] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
I20251027 21:07:54.691354 10040 pending_rounds.cc:77] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510620764233728 op_type: NO_OP noop_request { }
I20251027 21:07:54.691509 10040 raft_consensus.cc:2889] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:07:54.691650 10040 raft_consensus.cc:2272] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:54.692238 10040 ts_tablet_manager.cc:1916] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:54.692494 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 630ae4d66f99474382814cc32cc60ea0
I20251027 21:07:54.692837 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_580942c54cc24c14b0849e1f7c209975"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.693113 9306 catalog_manager.cc:6719] ReplaceTablet: tablet 630ae4d66f99474382814cc32cc60ea0 deleted and replaced by tablet 2f641bc4dbed4c0abd0a55b253352507
I20251027 21:07:54.693068 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c3f0afa8e76249b69b9fbe4b3f67d020" } modify_external_catalogs: true
I20251027 21:07:54.693238 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c3f0afa8e76249b69b9fbe4b3f67d020" } modify_external_catalogs: true
W20251027 21:07:54.693419 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_580942c54cc24c14b0849e1f7c209975 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.693876 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 630ae4d66f99474382814cc32cc60ea0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2f641bc4dbed4c0abd0a55b253352507) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:54.694672 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_96870e913d9243568d09f6f4cb2ccbb4" } new_table_name: "default.Table_581b993e984d49cd9178284ab5c6a500" modify_external_catalogs: true
I20251027 21:07:54.696441 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_691240a058cc4f879e5737f73a4c12e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:54.697033 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_691240a058cc4f879e5737f73a4c12e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.698297 10041 ts_tablet_manager.cc:1916] T 9c239216cb8846f0b1aaa360ddb83321 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:54.698381 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet 630ae4d66f99474382814cc32cc60ea0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2f641bc4dbed4c0abd0a55b253352507) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:54.699772 10358 raft_consensus.cc:697] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:07:54.700099 10358 consensus_queue.cc:237] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.704396 10041 ts_tablet_manager.cc:1929] T 9c239216cb8846f0b1aaa360ddb83321 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.705112 10041 log.cc:1199] T 9c239216cb8846f0b1aaa360ddb83321 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/9c239216cb8846f0b1aaa360ddb83321
I20251027 21:07:54.705538 10041 ts_tablet_manager.cc:1950] T 9c239216cb8846f0b1aaa360ddb83321 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:54.706507 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 9c239216cb8846f0b1aaa360ddb83321 (table default.Table_2d758ba690f8478ba7f983e8fd770219 [id=849a0760791548a38aceb18d0533f5f4]) successfully deleted
I20251027 21:07:54.706967 9306 master_service.cc:954] ReplaceTablet: received request to replace tablet 132a01405a63447ab27b2137b803224f from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:54.707234 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2992b8e207024e06957ce7befb657acb" } modify_external_catalogs: true
I20251027 21:07:54.707473 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2992b8e207024e06957ce7befb657acb" } modify_external_catalogs: true
I20251027 21:07:54.707638 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6f3e1c9642e3433eab3f8e5a409470c4" } modify_external_catalogs: true
I20251027 21:07:54.708335 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6f3e1c9642e3433eab3f8e5a409470c4" } modify_external_catalogs: true
I20251027 21:07:54.711941 10029 tablet_bootstrap.cc:492] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:54.712304 10029 ts_tablet_manager.cc:1403] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.075s user 0.002s sys 0.002s
I20251027 21:07:54.713529 10029 raft_consensus.cc:359] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.713671 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet 630ae4d66f99474382814cc32cc60ea0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2f641bc4dbed4c0abd0a55b253352507) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:54.714473 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 630ae4d66f99474382814cc32cc60ea0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 630ae4d66f99474382814cc32cc60ea0 already in progress: creating tablet
I20251027 21:07:54.714480 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet cef2bfee222f45e6b6f20c6d63ff44d4
W20251027 21:07:54.714931 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93a3b22cdc1543aebcc7fe72f2f2c0ae consensus state has no leader
I20251027 21:07:54.715794 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet cef2bfee222f45e6b6f20c6d63ff44d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:54.716993 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_340ce2d28b454548af6c487cb35008ac" } modify_external_catalogs: true
I20251027 21:07:54.717247 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet cef2bfee222f45e6b6f20c6d63ff44d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:54.717756 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet cef2bfee222f45e6b6f20c6d63ff44d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cef2bfee222f45e6b6f20c6d63ff44d4 already in progress: creating tablet
I20251027 21:07:54.718070 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_340ce2d28b454548af6c487cb35008ac" } modify_external_catalogs: true
I20251027 21:07:54.720293 10411 raft_consensus.cc:493] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:54.720427 10411 raft_consensus.cc:515] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.720999 10411 leader_election.cc:290] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:54.721304 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d94c695320f24a7f9776c493e462ed24" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:54.721783 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4184f1a26ee84a88a37ee927169b9b44" } modify_external_catalogs: true
I20251027 21:07:54.721951 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4184f1a26ee84a88a37ee927169b9b44" } modify_external_catalogs: true
W20251027 21:07:54.722048 9859 leader_election.cc:343] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:54.721608 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d94c695320f24a7f9776c493e462ed24" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:54.722450 10029 raft_consensus.cc:385] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.722545 10029 raft_consensus.cc:740] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.722776 10029 consensus_queue.cc:260] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:54.723423 9860 leader_election.cc:343] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:54.723510 10029 ts_tablet_manager.cc:1434] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20251027 21:07:54.723551 9860 leader_election.cc:304] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:54.723799 10029 tablet_bootstrap.cc:492] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:54.723956 10411 raft_consensus.cc:2749] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:54.725773 10043 log.cc:1199] T e07b8aa0a41241d79254c4dae8738e79 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/e07b8aa0a41241d79254c4dae8738e79
I20251027 21:07:54.726769 10031 tablet_bootstrap.cc:492] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:54.726959 10031 ts_tablet_manager.cc:1403] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.193s user 0.004s sys 0.005s
I20251027 21:07:54.727380 10032 tablet_bootstrap.cc:492] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:54.727579 10032 ts_tablet_manager.cc:1403] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.115s user 0.004s sys 0.006s
I20251027 21:07:54.727875 10031 raft_consensus.cc:359] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.728091 10031 raft_consensus.cc:385] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.728221 10031 raft_consensus.cc:740] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.728374 10032 raft_consensus.cc:359] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.728612 10032 raft_consensus.cc:385] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.728681 10032 raft_consensus.cc:740] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.728510 10031 consensus_queue.cc:260] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.728962 10032 consensus_queue.cc:260] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.729090 10414 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251027 21:07:54.729460 10032 ts_tablet_manager.cc:1434] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:54.729928 10032 tablet_bootstrap.cc:492] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:54.733433 9308 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 78884c2be75f45f881f5424d3b3f4e08 (table default.Table_1376974f614548189208cb9cf0fd5353 [id=0eb078640c174f8fadb959306a35cada]). Expected version 1 got 0
I20251027 21:07:54.733592 10031 ts_tablet_manager.cc:1434] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251027 21:07:54.734231 10359 raft_consensus.cc:493] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:54.734385 10359 raft_consensus.cc:515] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.735585 10359 leader_election.cc:290] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:07:54.735721 9710 leader_election.cc:343] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 49b3d23798b740008c98cd0754ae33f4
I20251027 21:07:54.735924 10031 tablet_bootstrap.cc:492] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:54.736392 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet e07b8aa0a41241d79254c4dae8738e79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:54.737080 9711 leader_election.cc:343] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 49b3d23798b740008c98cd0754ae33f4
I20251027 21:07:54.737208 9711 leader_election.cc:304] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:54.738094 10359 raft_consensus.cc:2749] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:54.739843 9311 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 78884c2be75f45f881f5424d3b3f4e08 (table default.Table_1376974f614548189208cb9cf0fd5353 [id=0eb078640c174f8fadb959306a35cada]). Expected version 1 got 0
I20251027 21:07:54.740724 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 93a3b22cdc1543aebcc7fe72f2f2c0ae (DEFAULT_TABLE table=default.Table_581b993e984d49cd9178284ab5c6a500 [id=8646c99b90414f4fa266cd7a31123d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.741449 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 59300e16b3e148a08a217fc24d1e572f (DEFAULT_TABLE table=default.Table_37d44f2ffd8b4816b4bff0b1960cd275 [id=a1dea3b74aa94995991d3123fbb2bf8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.741904 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93a3b22cdc1543aebcc7fe72f2f2c0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.742444 10032 tablet_bootstrap.cc:654] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.743223 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59300e16b3e148a08a217fc24d1e572f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.743324 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 59300e16b3e148a08a217fc24d1e572f (DEFAULT_TABLE table=default.Table_37d44f2ffd8b4816b4bff0b1960cd275 [id=a1dea3b74aa94995991d3123fbb2bf8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.743737 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59300e16b3e148a08a217fc24d1e572f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.743825 10043 ts_tablet_manager.cc:1950] T e07b8aa0a41241d79254c4dae8738e79 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:54.743925 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 93a3b22cdc1543aebcc7fe72f2f2c0ae (DEFAULT_TABLE table=default.Table_581b993e984d49cd9178284ab5c6a500 [id=8646c99b90414f4fa266cd7a31123d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.744316 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93a3b22cdc1543aebcc7fe72f2f2c0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.745613 10043 tablet_replica.cc:333] T 779b7f63b0fa4f9ab47cf5799b3e085d P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:54.745787 10043 raft_consensus.cc:2243] T 779b7f63b0fa4f9ab47cf5799b3e085d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.745949 10043 raft_consensus.cc:2272] T 779b7f63b0fa4f9ab47cf5799b3e085d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:54.726446 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet cef2bfee222f45e6b6f20c6d63ff44d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:54.729598 10468 raft_consensus.cc:493] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:54.746466 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet e07b8aa0a41241d79254c4dae8738e79 (table default.Table_e68b56d3ddd94f308243ec6a03ea9e6e [id=0a94e88b69094ffe8be3367470be86da]) successfully deleted
I20251027 21:07:54.743335 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 4ba13a89a33549af921d3b55023faf15 (DEFAULT_TABLE table=default.Table_e763bc3cda90472897f0f798424199b4 [id=714f3076ca9f4d399fcc7a36e701ac85]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:54.746727 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet a033551a591146039b017b4193b221ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a033551a591146039b017b4193b221ca already in progress: creating tablet
I20251027 21:07:54.746806 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ba13a89a33549af921d3b55023faf15. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:54.746932 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 53cac7817db440e79e7416d625a9b2c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53cac7817db440e79e7416d625a9b2c8 already in progress: creating tablet
W20251027 21:07:54.747143 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet a2d26784bbb24ae3893103802c41affb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2d26784bbb24ae3893103802c41affb already in progress: creating tablet
I20251027 21:07:54.750450 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f4272063203347a5810ed07554c2269b" } new_table_name: "default.Table_6c3cfd4d761643dca83d16b2441b0c64" modify_external_catalogs: true
I20251027 21:07:54.746174 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_bae8db615e424af398f1f9baec11cb76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:54.754590 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bae8db615e424af398f1f9baec11cb76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.754572 10468 raft_consensus.cc:515] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.755869 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d8825b80cec4ba68da077ca3b8d89db" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:54.756436 9566 leader_election.cc:343] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:54.756794 10468 leader_election.cc:290] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:07:54.757751 9568 leader_election.cc:343] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 7d8825b80cec4ba68da077ca3b8d89db
I20251027 21:07:54.757941 9568 leader_election.cc:304] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:54.758515 10468 raft_consensus.cc:2749] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:54.760151 10029 tablet_bootstrap.cc:654] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.760375 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 815e12edcd3847ff8485ba768291a824
I20251027 21:07:54.762085 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet 815e12edcd3847ff8485ba768291a824 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:54.764242 9299 catalog_manager.cc:4952] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 815e12edcd3847ff8485ba768291a824 because the tablet was not found. No further retry: Not found: Tablet not found: 815e12edcd3847ff8485ba768291a824
I20251027 21:07:54.764768 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 815e12edcd3847ff8485ba768291a824 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:54.765381 9304 catalog_manager.cc:5649] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } }
I20251027 21:07:54.767686 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 4ba13a89a33549af921d3b55023faf15 (DEFAULT_TABLE table=default.Table_e763bc3cda90472897f0f798424199b4 [id=714f3076ca9f4d399fcc7a36e701ac85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.767690 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 4ba13a89a33549af921d3b55023faf15 (DEFAULT_TABLE table=default.Table_e763bc3cda90472897f0f798424199b4 [id=714f3076ca9f4d399fcc7a36e701ac85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.768019 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ba13a89a33549af921d3b55023faf15. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.768074 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ba13a89a33549af921d3b55023faf15. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.777158 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 0d2d977e360b4ed29c9ed63a309783d6 (DEFAULT_TABLE table=default.Table_048e52e3f32d46a1a3d1619888efc5ae [id=9ebc5854bbdc40bfacda411c948f98d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.777269 9305 master_service.cc:954] ReplaceTablet: received request to replace tablet 1e5f5b57f5d948f3b2352451ccaacb09 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:54.777559 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d2d977e360b4ed29c9ed63a309783d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.777551 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 0d2d977e360b4ed29c9ed63a309783d6 (DEFAULT_TABLE table=default.Table_048e52e3f32d46a1a3d1619888efc5ae [id=9ebc5854bbdc40bfacda411c948f98d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.777861 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d2d977e360b4ed29c9ed63a309783d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.778703 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 9be6b996df8145b9b0df1a2d2dc0ef03 (DEFAULT_TABLE table=default.Table_d09f1941d15f4517ac7561a66dd9b5e9 [id=b57564104d854b67a653c03de0650c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.779247 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9be6b996df8145b9b0df1a2d2dc0ef03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.779330 10043 ts_tablet_manager.cc:1916] T 779b7f63b0fa4f9ab47cf5799b3e085d P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:54.780776 9893 tablet_service.cc:1505] Processing CreateTablet for tablet a9558d53008640739ac6fb45f1352fc0 (DEFAULT_TABLE table=default.Table_0c6076a969244f39ac709114449efb98 [id=c575fb3af44c46db8c198569241a425f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.781106 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9558d53008640739ac6fb45f1352fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.781214 9594 tablet_service.cc:1505] Processing CreateTablet for tablet a9558d53008640739ac6fb45f1352fc0 (DEFAULT_TABLE table=default.Table_0c6076a969244f39ac709114449efb98 [id=c575fb3af44c46db8c198569241a425f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.781316 9737 tablet_service.cc:1505] Processing CreateTablet for tablet a9558d53008640739ac6fb45f1352fc0 (DEFAULT_TABLE table=default.Table_0c6076a969244f39ac709114449efb98 [id=c575fb3af44c46db8c198569241a425f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.781618 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9558d53008640739ac6fb45f1352fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.782220 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9558d53008640739ac6fb45f1352fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.782413 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 9be6b996df8145b9b0df1a2d2dc0ef03 (DEFAULT_TABLE table=default.Table_d09f1941d15f4517ac7561a66dd9b5e9 [id=b57564104d854b67a653c03de0650c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.782745 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9be6b996df8145b9b0df1a2d2dc0ef03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.786087 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 9be6b996df8145b9b0df1a2d2dc0ef03 (DEFAULT_TABLE table=default.Table_d09f1941d15f4517ac7561a66dd9b5e9 [id=b57564104d854b67a653c03de0650c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.786335 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c260261e06b74bf097a23ef87e39eccd" } new_table_name: "default.Table_8291d411936d4f4e8d267fdc28efc533" modify_external_catalogs: true
I20251027 21:07:54.786474 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9be6b996df8145b9b0df1a2d2dc0ef03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.788055 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 0d2d977e360b4ed29c9ed63a309783d6 (DEFAULT_TABLE table=default.Table_048e52e3f32d46a1a3d1619888efc5ae [id=9ebc5854bbdc40bfacda411c948f98d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.788419 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d2d977e360b4ed29c9ed63a309783d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.790284 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 815e12edcd3847ff8485ba768291a824 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:54.791764 10031 tablet_bootstrap.cc:654] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.794822 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 59300e16b3e148a08a217fc24d1e572f (DEFAULT_TABLE table=default.Table_37d44f2ffd8b4816b4bff0b1960cd275 [id=a1dea3b74aa94995991d3123fbb2bf8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.795157 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59300e16b3e148a08a217fc24d1e572f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.805474 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 371aa8a57aee466ab01652d6061db4c2 (DEFAULT_TABLE table=default.Table_cd4bceadd9e4409c9ed9dc8365e4ed8e [id=9529122fab2c414e9d718265e42f26ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.806011 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 371aa8a57aee466ab01652d6061db4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.806891 10029 tablet_bootstrap.cc:492] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:54.807036 10029 ts_tablet_manager.cc:1403] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.001s
I20251027 21:07:54.807789 10029 raft_consensus.cc:359] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.807991 10029 raft_consensus.cc:385] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.808074 10029 raft_consensus.cc:740] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.808277 10029 consensus_queue.cc:260] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.811776 9312 master_service.cc:954] ReplaceTablet: received request to replace tablet 7292bc6c22ca4ab5b8f53a69520833ba from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:54.813109 10040 ts_tablet_manager.cc:1929] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:07:54.813282 10040 log.cc:1199] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/2d659e4d73ea4ff3975a4b58a794090a
I20251027 21:07:54.813712 10040 ts_tablet_manager.cc:1950] T 2d659e4d73ea4ff3975a4b58a794090a P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:54.814810 10029 ts_tablet_manager.cc:1434] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251027 21:07:54.815790 10029 tablet_bootstrap.cc:492] T a9cc64ad85334daa9a09538e24ebd8a5 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:54.817258 9897 tablet_service.cc:1505] Processing CreateTablet for tablet e10ae34eaaf3476594813de8e96c7e77 (DEFAULT_TABLE table=default.Table_107ad52a34d44600aa08dbe6983ab081 [id=bbc85bf038474c93ae8312bdca3a7029]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.817462 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 371aa8a57aee466ab01652d6061db4c2 (DEFAULT_TABLE table=default.Table_cd4bceadd9e4409c9ed9dc8365e4ed8e [id=9529122fab2c414e9d718265e42f26ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.818320 9742 tablet_service.cc:1505] Processing CreateTablet for tablet e10ae34eaaf3476594813de8e96c7e77 (DEFAULT_TABLE table=default.Table_107ad52a34d44600aa08dbe6983ab081 [id=bbc85bf038474c93ae8312bdca3a7029]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.819669 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e10ae34eaaf3476594813de8e96c7e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.821198 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 8dc358445526415d8cd4f1ac6779126d (DEFAULT_TABLE table=default.Table_1b463a3e59984a129fc2824928138b26 [id=7422045718af4223aadf1927576e0b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.821661 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dc358445526415d8cd4f1ac6779126d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.821720 9610 tablet_service.cc:1505] Processing CreateTablet for tablet e10ae34eaaf3476594813de8e96c7e77 (DEFAULT_TABLE table=default.Table_107ad52a34d44600aa08dbe6983ab081 [id=bbc85bf038474c93ae8312bdca3a7029]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.822105 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e10ae34eaaf3476594813de8e96c7e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.823354 10031 tablet_bootstrap.cc:492] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:54.823510 10031 ts_tablet_manager.cc:1403] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.001s
I20251027 21:07:54.824651 10031 raft_consensus.cc:359] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.824829 10031 raft_consensus.cc:385] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.824898 10031 raft_consensus.cc:740] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.825233 10031 consensus_queue.cc:260] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.817889 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 371aa8a57aee466ab01652d6061db4c2 (DEFAULT_TABLE table=default.Table_cd4bceadd9e4409c9ed9dc8365e4ed8e [id=9529122fab2c414e9d718265e42f26ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.827302 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 371aa8a57aee466ab01652d6061db4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.821331 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 8dc358445526415d8cd4f1ac6779126d (DEFAULT_TABLE table=default.Table_1b463a3e59984a129fc2824928138b26 [id=7422045718af4223aadf1927576e0b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.829422 10032 tablet_bootstrap.cc:492] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:54.829685 10032 ts_tablet_manager.cc:1403] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.100s user 0.004s sys 0.000s
I20251027 21:07:54.829747 10031 ts_tablet_manager.cc:1434] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251027 21:07:54.829751 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dc358445526415d8cd4f1ac6779126d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.830543 10032 raft_consensus.cc:359] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.830785 10032 raft_consensus.cc:385] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.831449 10032 raft_consensus.cc:740] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.831725 10032 consensus_queue.cc:260] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.831990 10031 tablet_bootstrap.cc:492] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:54.832355 10032 ts_tablet_manager.cc:1434] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:54.832695 10032 tablet_bootstrap.cc:492] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:54.833528 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 8dc358445526415d8cd4f1ac6779126d (DEFAULT_TABLE table=default.Table_1b463a3e59984a129fc2824928138b26 [id=7422045718af4223aadf1927576e0b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.834760 10032 tablet_bootstrap.cc:654] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.836020 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 315) took 2295 ms. Trace:
I20251027 21:07:54.836165 10040 rpcz_store.cc:276] 1027 21:07:52.540568 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.542405 (+ 1837us) service_pool.cc:225] Handling call
1027 21:07:54.690640 (+2148235us) ts_tablet_manager.cc:1151] Deleting tablet 2d659e4d73ea4ff3975a4b58a794090a
1027 21:07:54.690648 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:54.762486 (+ 71838us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:54.813007 (+ 50521us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:54.835952 (+ 22945us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2146331}
I20251027 21:07:54.836687 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 2d659e4d73ea4ff3975a4b58a794090a (table default.Table_e9fd9bee6a2a40d7ad92d2d31b03afce [id=af6cb904f8cd4a929be7a978f9bc2001]) successfully deleted
I20251027 21:07:54.836851 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 316) took 2288 ms. Trace:
I20251027 21:07:54.836947 10040 rpcz_store.cc:276] 1027 21:07:52.547963 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.548006 (+ 43us) service_pool.cc:225] Handling call
1027 21:07:54.836661 (+2288655us) ts_tablet_manager.cc:1151] Deleting tablet f22f013b25074d4db9c7f13caf5b9b42
1027 21:07:54.836670 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:54.836803 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2288467}
W20251027 21:07:54.837380 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet f22f013b25074d4db9c7f13caf5b9b42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f22f013b25074d4db9c7f13caf5b9b42 already in progress: creating tablet
I20251027 21:07:54.837755 10040 tablet_replica.cc:333] T 81e914ecc3c74b869e9fc3ad102887ec P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:54.838013 10040 raft_consensus.cc:2243] T 81e914ecc3c74b869e9fc3ad102887ec P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.838260 10040 raft_consensus.cc:2272] T 81e914ecc3c74b869e9fc3ad102887ec P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:54.838979 10040 ts_tablet_manager.cc:1916] T 81e914ecc3c74b869e9fc3ad102887ec P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:54.839244 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 371aa8a57aee466ab01652d6061db4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.839484 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dc358445526415d8cd4f1ac6779126d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.846613 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e10ae34eaaf3476594813de8e96c7e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.849242 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1d1b4e58d3da402188147c3d83dbe958" } new_table_name: "default.Table_5a6211c8465040dbad1cc86377a370ea" modify_external_catalogs: true
I20251027 21:07:54.849771 10029 tablet_bootstrap.cc:654] T a9cc64ad85334daa9a09538e24ebd8a5 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.850911 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 42371496a7494e77a7bf1169de5b083a
I20251027 21:07:54.851550 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet 42371496a7494e77a7bf1169de5b083a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:54.852066 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet cdbde6272af147829ecadd67d634f049
I20251027 21:07:54.853094 9309 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 4af2f4efd9874fc694fb71f0d8a1e6ba
I20251027 21:07:54.853305 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet cdbde6272af147829ecadd67d634f049 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:54.853907 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet 42371496a7494e77a7bf1169de5b083a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:54.854475 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 42371496a7494e77a7bf1169de5b083a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42371496a7494e77a7bf1169de5b083a already in progress: creating tablet
I20251027 21:07:54.854755 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet cdbde6272af147829ecadd67d634f049 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:54.855278 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet cdbde6272af147829ecadd67d634f049 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cdbde6272af147829ecadd67d634f049 already in progress: creating tablet
I20251027 21:07:54.855556 9737 tablet_service.cc:1552] Processing DeleteTablet for tablet cdbde6272af147829ecadd67d634f049 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:54.856829 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 42371496a7494e77a7bf1169de5b083a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:54.857077 10043 ts_tablet_manager.cc:1929] T 779b7f63b0fa4f9ab47cf5799b3e085d P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.857219 10043 log.cc:1199] T 779b7f63b0fa4f9ab47cf5799b3e085d P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/779b7f63b0fa4f9ab47cf5799b3e085d
I20251027 21:07:54.857580 10043 ts_tablet_manager.cc:1950] T 779b7f63b0fa4f9ab47cf5799b3e085d P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:54.858016 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 1a48067b40324c79b1ee4eb431ce4a46 (DEFAULT_TABLE table=default.Table_c3c17096bae64f378a84096c3b6f5ab5 [id=00404f9f36a34a2e8e1244799468d85e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.858392 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a48067b40324c79b1ee4eb431ce4a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.858763 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 779b7f63b0fa4f9ab47cf5799b3e085d (table default.Table_d98a9c0b61d649f2945cb6c199f70a64 [id=6c62d93feb804d6aa1d442f8458b7ef8]) successfully deleted
I20251027 21:07:54.864137 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 1a48067b40324c79b1ee4eb431ce4a46 (DEFAULT_TABLE table=default.Table_c3c17096bae64f378a84096c3b6f5ab5 [id=00404f9f36a34a2e8e1244799468d85e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.864545 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a48067b40324c79b1ee4eb431ce4a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.864650 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 43611e33512b49d58485cf15de788be7 (DEFAULT_TABLE table=default.Table_02a5b78b84f1409b80d09d02e5069b9c [id=b3524d24371a43038db124bb4f0c5793]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.865061 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43611e33512b49d58485cf15de788be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.871006 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 43611e33512b49d58485cf15de788be7 (DEFAULT_TABLE table=default.Table_02a5b78b84f1409b80d09d02e5069b9c [id=b3524d24371a43038db124bb4f0c5793]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.871394 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43611e33512b49d58485cf15de788be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.871762 10031 tablet_bootstrap.cc:654] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.877493 10359 raft_consensus.cc:493] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:54.877564 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ed32ba2bd26544e1a9165f6246ee5cb9" } new_table_name: "default.Table_b2fe93c466204947aa406fbece12b682" modify_external_catalogs: true
I20251027 21:07:54.877616 10359 raft_consensus.cc:515] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.878273 10359 leader_election.cc:290] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:07:54.878604 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 630ae4d66f99474382814cc32cc60ea0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 630ae4d66f99474382814cc32cc60ea0 already in progress: creating tablet
W20251027 21:07:54.878839 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet cef2bfee222f45e6b6f20c6d63ff44d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cef2bfee222f45e6b6f20c6d63ff44d4 already in progress: creating tablet
W20251027 21:07:54.879074 9298 catalog_manager.cc:4952] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 815e12edcd3847ff8485ba768291a824 because the tablet was not found. No further retry: Not found: Tablet not found: 815e12edcd3847ff8485ba768291a824
W20251027 21:07:54.879317 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 42371496a7494e77a7bf1169de5b083a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42371496a7494e77a7bf1169de5b083a already in progress: creating tablet
W20251027 21:07:54.879346 9710 leader_election.cc:343] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: e07b8aa0a41241d79254c4dae8738e79
W20251027 21:07:54.879550 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet cdbde6272af147829ecadd67d634f049 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cdbde6272af147829ecadd67d634f049 already in progress: creating tablet
I20251027 21:07:54.880283 10029 tablet_bootstrap.cc:492] T a9cc64ad85334daa9a09538e24ebd8a5 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:54.880141 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f8b57b86738a4f61b6a63a06b73016dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.880436 10029 ts_tablet_manager.cc:1403] T a9cc64ad85334daa9a09538e24ebd8a5 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.001s
W20251027 21:07:54.880976 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8b57b86738a4f61b6a63a06b73016dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.881182 10029 raft_consensus.cc:359] T a9cc64ad85334daa9a09538e24ebd8a5 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.881351 10029 raft_consensus.cc:385] T a9cc64ad85334daa9a09538e24ebd8a5 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.881417 10029 raft_consensus.cc:740] T a9cc64ad85334daa9a09538e24ebd8a5 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.881649 10029 consensus_queue.cc:260] T a9cc64ad85334daa9a09538e24ebd8a5 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.882293 10029 ts_tablet_manager.cc:1434] T a9cc64ad85334daa9a09538e24ebd8a5 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:54.885037 10029 tablet_bootstrap.cc:492] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:54.886751 10029 tablet_bootstrap.cc:654] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:54.888527 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 133e474f9133487f89c44dbfa6266ca7 (DEFAULT_TABLE table=default.Table_537e495061924a4da35379e43642675a [id=3b80e606aa374fd9ad1e7b0c5c20bb6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.888854 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 133e474f9133487f89c44dbfa6266ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.890013 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 133e474f9133487f89c44dbfa6266ca7 (DEFAULT_TABLE table=default.Table_537e495061924a4da35379e43642675a [id=3b80e606aa374fd9ad1e7b0c5c20bb6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.890372 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 133e474f9133487f89c44dbfa6266ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.890417 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 57ef07c764664deaa4b2e9bcae7e7973 (DEFAULT_TABLE table=default.Table_73d87eb46d0c436db67f3b5c9ccbb8db [id=a541c8d30a9540f5a7d4dd98a5d3739a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.890799 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57ef07c764664deaa4b2e9bcae7e7973. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:54.891217 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6210d3d10fc44b9e9e2b0780883186ce consensus state has no leader
I20251027 21:07:54.897235 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_cc31649fd9f544e985ed00eb483164ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.897557 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_1a08b666f7f4426eb19757e15584dd53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:54.897693 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc31649fd9f544e985ed00eb483164ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:54.898074 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a08b666f7f4426eb19757e15584dd53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.898087 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 43611e33512b49d58485cf15de788be7 (DEFAULT_TABLE table=default.Table_02a5b78b84f1409b80d09d02e5069b9c [id=b3524d24371a43038db124bb4f0c5793]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.898481 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43611e33512b49d58485cf15de788be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.899871 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 1e5f5b57f5d948f3b2352451ccaacb09
I20251027 21:07:54.900354 9305 catalog_manager.cc:6719] ReplaceTablet: tablet 1e5f5b57f5d948f3b2352451ccaacb09 deleted and replaced by tablet 82366f8bf4154ffd87ef45e4a4d5f54b
I20251027 21:07:54.897208 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3d031fe204804a95b5bc07e8b54573d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.900691 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e5f5b57f5d948f3b2352451ccaacb09 with delete_type TABLET_DATA_DELETED (replaced by tablet 82366f8bf4154ffd87ef45e4a4d5f54b) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:54.900946 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d031fe204804a95b5bc07e8b54573d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.901960 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet 0a58cd07e1bb4ffdbf39e7e16a46022a from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:54.902348 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 5cb8ca3a50c148e6b602e949650353f8 (DEFAULT_TABLE table=default.Table_24aac5cecb8745d7be92287080a1b6e0 [id=080bf50609c045bd9b05f4e9144feac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.902518 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 57ef07c764664deaa4b2e9bcae7e7973 (DEFAULT_TABLE table=default.Table_73d87eb46d0c436db67f3b5c9ccbb8db [id=a541c8d30a9540f5a7d4dd98a5d3739a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.902669 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cb8ca3a50c148e6b602e949650353f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.902809 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57ef07c764664deaa4b2e9bcae7e7973. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.906288 10043 tablet_replica.cc:333] T 1e5f5b57f5d948f3b2352451ccaacb09 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:54.906585 10043 raft_consensus.cc:2243] T 1e5f5b57f5d948f3b2352451ccaacb09 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.906787 10043 raft_consensus.cc:2272] T 1e5f5b57f5d948f3b2352451ccaacb09 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:54.907428 10043 ts_tablet_manager.cc:1916] T 1e5f5b57f5d948f3b2352451ccaacb09 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:54.910276 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e5f5b57f5d948f3b2352451ccaacb09 with delete_type TABLET_DATA_DELETED (replaced by tablet 82366f8bf4154ffd87ef45e4a4d5f54b) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:54.910421 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 5cb8ca3a50c148e6b602e949650353f8 (DEFAULT_TABLE table=default.Table_24aac5cecb8745d7be92287080a1b6e0 [id=080bf50609c045bd9b05f4e9144feac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.910756 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cb8ca3a50c148e6b602e949650353f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.922394 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 9351a369dc1e4ac49f6ad419eab3a4c5 (DEFAULT_TABLE table=default.Table_3b49d647ca194ad9bf5524812c6837a2 [id=a12d4ab777c34431beaa8f87ddbc7135]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.922765 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9351a369dc1e4ac49f6ad419eab3a4c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.926699 10029 tablet_bootstrap.cc:492] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:54.926838 10029 ts_tablet_manager.cc:1403] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.001s
I20251027 21:07:54.927595 10029 raft_consensus.cc:359] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.927765 10029 raft_consensus.cc:385] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:54.927838 10029 raft_consensus.cc:740] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:54.928077 10029 consensus_queue.cc:260] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:54.929481 10029 ts_tablet_manager.cc:1434] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:07:54.929811 10029 tablet_bootstrap.cc:492] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:54.928838 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_989e3ba3204f41f6b0d7f408eab2f4a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.930264 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet a9cc64ad85334daa9a09538e24ebd8a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:54.930348 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_989e3ba3204f41f6b0d7f408eab2f4a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.930665 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet 533f231e3ff64531ae2331409a961425 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:54.931520 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d8825b80cec4ba68da077ca3b8d89db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:54.931735 10040 ts_tablet_manager.cc:1929] T 81e914ecc3c74b869e9fc3ad102887ec P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.931891 10040 log.cc:1199] T 81e914ecc3c74b869e9fc3ad102887ec P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/81e914ecc3c74b869e9fc3ad102887ec
I20251027 21:07:54.932343 9740 tablet_service.cc:1505] Processing CreateTablet for tablet e7df6f0acec34d399aa25b78c3ed7c7a (DEFAULT_TABLE table=default.Table_82b8023e5cb34761981a49e164eb36c4 [id=635f029832b249c590d149e5c44ca8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.932723 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7df6f0acec34d399aa25b78c3ed7c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.933234 10040 ts_tablet_manager.cc:1950] T 81e914ecc3c74b869e9fc3ad102887ec P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
W20251027 21:07:54.933512 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0820648719a34f379507efced9cc434b has no consensus state
I20251027 21:07:54.934173 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 9351a369dc1e4ac49f6ad419eab3a4c5 (DEFAULT_TABLE table=default.Table_3b49d647ca194ad9bf5524812c6837a2 [id=a12d4ab777c34431beaa8f87ddbc7135]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.934542 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9351a369dc1e4ac49f6ad419eab3a4c5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:54.934721 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9339c6b1154400eace53d765528fdf9 consensus state has no leader
I20251027 21:07:54.935310 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 7292bc6c22ca4ab5b8f53a69520833ba
I20251027 21:07:54.935827 9312 catalog_manager.cc:6719] ReplaceTablet: tablet 7292bc6c22ca4ab5b8f53a69520833ba deleted and replaced by tablet 37ae3b6bf590473e8e09092fc5f7df1b
I20251027 21:07:54.936056 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet 7292bc6c22ca4ab5b8f53a69520833ba with delete_type TABLET_DATA_DELETED (replaced by tablet 37ae3b6bf590473e8e09092fc5f7df1b) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:54.936844 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 345d3795a37b4a40b40967cfacf0f893 consensus state has no leader
I20251027 21:07:54.937166 10043 ts_tablet_manager.cc:1929] T 1e5f5b57f5d948f3b2352451ccaacb09 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:54.937832 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 314) took 2444 ms. Trace:
I20251027 21:07:54.938791 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 95446c4333424f10a71829ce32d857e4 (DEFAULT_TABLE table=default.Table_219373cdafa946c68b0330d75aa69cc8 [id=3246662942e949b9ac37a58dd81f0ae9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.939252 9737 tablet_service.cc:1552] Processing DeleteTablet for tablet 7292bc6c22ca4ab5b8f53a69520833ba with delete_type TABLET_DATA_DELETED (replaced by tablet 37ae3b6bf590473e8e09092fc5f7df1b) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:54.939525 10043 log.cc:1199] T 1e5f5b57f5d948f3b2352451ccaacb09 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/1e5f5b57f5d948f3b2352451ccaacb09
I20251027 21:07:54.937248 9600 tablet_service.cc:1505] Processing CreateTablet for tablet e7df6f0acec34d399aa25b78c3ed7c7a (DEFAULT_TABLE table=default.Table_82b8023e5cb34761981a49e164eb36c4 [id=635f029832b249c590d149e5c44ca8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.937263 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 95446c4333424f10a71829ce32d857e4 (DEFAULT_TABLE table=default.Table_219373cdafa946c68b0330d75aa69cc8 [id=3246662942e949b9ac37a58dd81f0ae9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.940152 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95446c4333424f10a71829ce32d857e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.941759 10359 raft_consensus.cc:493] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:54.943905 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7df6f0acec34d399aa25b78c3ed7c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.945415 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b126a056ef2c41b28a178327ef69e524"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.944164 10029 tablet_bootstrap.cc:654] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:54.945948 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b126a056ef2c41b28a178327ef69e524 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.944301 10043 ts_tablet_manager.cc:1950] T 1e5f5b57f5d948f3b2352451ccaacb09 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:54.948156 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 1e5f5b57f5d948f3b2352451ccaacb09 (table default.Table_f0792045d49041578f5ff6459c99f77e [id=7f83f7c0f9524147bb778a06dd233a8e]) successfully deleted
I20251027 21:07:54.948498 10043 tablet_replica.cc:333] T a9cc64ad85334daa9a09538e24ebd8a5 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:54.948706 10043 raft_consensus.cc:2243] T a9cc64ad85334daa9a09538e24ebd8a5 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.948908 10043 raft_consensus.cc:2272] T a9cc64ad85334daa9a09538e24ebd8a5 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:54.949770 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 1a48067b40324c79b1ee4eb431ce4a46 (DEFAULT_TABLE table=default.Table_c3c17096bae64f378a84096c3b6f5ab5 [id=00404f9f36a34a2e8e1244799468d85e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.950047 9306 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 7f7ad06a15c04efbb701634e14904351 (table default.Table_318d41ccfb2c4d2c99bafcea875143a1 [id=2bc39b5bc02645ef830eee14e92707db]). Expected version 1 got 0
I20251027 21:07:54.950327 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4d2a5a0a0ac24d4bb907b89a6bee1aaf" } modify_external_catalogs: true
I20251027 21:07:54.950498 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4d2a5a0a0ac24d4bb907b89a6bee1aaf" } modify_external_catalogs: true
I20251027 21:07:54.950807 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5315576a7f87464dbbfded0081ddef2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:54.951263 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5315576a7f87464dbbfded0081ddef2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:54.951686 9711 leader_election.cc:343] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: e07b8aa0a41241d79254c4dae8738e79
I20251027 21:07:54.951956 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a48067b40324c79b1ee4eb431ce4a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.954238 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e5f5b57f5d948f3b2352451ccaacb09 with delete_type TABLET_DATA_DELETED (replaced by tablet 82366f8bf4154ffd87ef45e4a4d5f54b) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:54.954744 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 5cb8ca3a50c148e6b602e949650353f8 (DEFAULT_TABLE table=default.Table_24aac5cecb8745d7be92287080a1b6e0 [id=080bf50609c045bd9b05f4e9144feac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.955042 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_35709e995c574069b4df49b470f87b26" } new_table_name: "default.Table_a3f6fbc547e4462097650dbde068ef25" modify_external_catalogs: true
I20251027 21:07:54.955672 9901 tablet_service.cc:1505] Processing CreateTablet for tablet 95446c4333424f10a71829ce32d857e4 (DEFAULT_TABLE table=default.Table_219373cdafa946c68b0330d75aa69cc8 [id=3246662942e949b9ac37a58dd81f0ae9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.958356 9304 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 7f7ad06a15c04efbb701634e14904351 (table default.Table_318d41ccfb2c4d2c99bafcea875143a1 [id=2bc39b5bc02645ef830eee14e92707db]). Expected version 1 got 0
I20251027 21:07:54.958379 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 133e474f9133487f89c44dbfa6266ca7 (DEFAULT_TABLE table=default.Table_537e495061924a4da35379e43642675a [id=3b80e606aa374fd9ad1e7b0c5c20bb6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.962494 10040 rpcz_store.cc:276] 1027 21:07:52.493146 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.604979 (+111833us) service_pool.cc:225] Handling call
1027 21:07:54.837675 (+2232696us) ts_tablet_manager.cc:1151] Deleting tablet 81e914ecc3c74b869e9fc3ad102887ec
1027 21:07:54.837684 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:54.861544 (+ 23860us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:54.914915 (+ 53371us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:54.937795 (+ 22880us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2232502}
I20251027 21:07:54.962890 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 319) took 2332 ms. Trace:
I20251027 21:07:54.962990 10040 rpcz_store.cc:276] 1027 21:07:52.630467 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.630522 (+ 55us) service_pool.cc:225] Handling call
1027 21:07:54.962763 (+2332241us) ts_tablet_manager.cc:1151] Deleting tablet fe24af9838954a9bb680474800642e62
1027 21:07:54.962783 (+ 20us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:54.962865 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2332006}
I20251027 21:07:54.963290 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 322) took 2288 ms. Trace:
I20251027 21:07:54.963382 10040 rpcz_store.cc:276] 1027 21:07:52.674600 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.674781 (+ 181us) service_pool.cc:225] Handling call
1027 21:07:54.963179 (+2288398us) ts_tablet_manager.cc:1151] Deleting tablet 288b88e13fdf4427aae53f64b87733bf
1027 21:07:54.963186 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:54.963266 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2288199}
I20251027 21:07:54.964751 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 57ef07c764664deaa4b2e9bcae7e7973 (DEFAULT_TABLE table=default.Table_73d87eb46d0c436db67f3b5c9ccbb8db [id=a541c8d30a9540f5a7d4dd98a5d3739a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.965257 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cb8ca3a50c148e6b602e949650353f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.965837 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 321) took 2291 ms. Trace:
I20251027 21:07:54.965941 10040 rpcz_store.cc:276] 1027 21:07:52.674480 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.674532 (+ 52us) service_pool.cc:225] Handling call
1027 21:07:54.965688 (+2291156us) ts_tablet_manager.cc:1151] Deleting tablet 9c2d713d72fd4b5ca34cb3e828c75761
1027 21:07:54.965698 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:54.965807 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2288160}
I20251027 21:07:54.966293 10040 tablet_replica.cc:333] T 601c74459d3a4a41b812fdcdf8b0a62c P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:54.966266 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d00ba224ec354b72a605f17e11591b78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.966496 10040 raft_consensus.cc:2243] T 601c74459d3a4a41b812fdcdf8b0a62c P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:54.966643 10040 raft_consensus.cc:2272] T 601c74459d3a4a41b812fdcdf8b0a62c P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251027 21:07:54.966732 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d00ba224ec354b72a605f17e11591b78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.967099 10040 ts_tablet_manager.cc:1916] T 601c74459d3a4a41b812fdcdf8b0a62c P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:54.970453 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 8ae222307a374a91916f560a0fbe8d8f (DEFAULT_TABLE table=default.Table_6c9db36d368044e487f90ca694832333 [id=d89790f6beb646f8b2580feeddea18f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.970219 9898 tablet_service.cc:1505] Processing CreateTablet for tablet e7df6f0acec34d399aa25b78c3ed7c7a (DEFAULT_TABLE table=default.Table_82b8023e5cb34761981a49e164eb36c4 [id=635f029832b249c590d149e5c44ca8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.970809 10032 tablet_bootstrap.cc:492] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:54.970901 10359 raft_consensus.cc:515] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:54.970983 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet 7292bc6c22ca4ab5b8f53a69520833ba with delete_type TABLET_DATA_DELETED (replaced by tablet 37ae3b6bf590473e8e09092fc5f7df1b) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:54.971522 10359 leader_election.cc:290] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:54.972033 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95446c4333424f10a71829ce32d857e4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:54.972672 9710 leader_election.cc:343] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: efc0019840244d14bb95ae581793e74b
I20251027 21:07:54.973958 9309 master_service.cc:954] ReplaceTablet: received request to replace tablet 533f34d4872444278477abcce89da9ec from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:54.976172 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3f462ba7e3ab4705a1daa311661c4f3d" } modify_external_catalogs: true
I20251027 21:07:54.976413 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3f462ba7e3ab4705a1daa311661c4f3d" } modify_external_catalogs: true
I20251027 21:07:54.978865 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_235e62a9895345fdb8b0b428777ff0bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:54.979760 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_235e62a9895345fdb8b0b428777ff0bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.986445 9711 leader_election.cc:304] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:54.986554 9901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95446c4333424f10a71829ce32d857e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.987291 10041 tablet_replica.cc:333] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:54.987673 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 8ae222307a374a91916f560a0fbe8d8f (DEFAULT_TABLE table=default.Table_6c9db36d368044e487f90ca694832333 [id=d89790f6beb646f8b2580feeddea18f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.988070 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ae222307a374a91916f560a0fbe8d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.988619 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 81e914ecc3c74b869e9fc3ad102887ec (table default.Table_a7f999b292d04332b831f454d4aa6743 [id=a6803bcbb3c2427992f31518cc1de040]) successfully deleted
W20251027 21:07:54.988838 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet fe24af9838954a9bb680474800642e62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe24af9838954a9bb680474800642e62 already in progress: creating tablet
W20251027 21:07:54.989005 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 288b88e13fdf4427aae53f64b87733bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 288b88e13fdf4427aae53f64b87733bf already in progress: creating tablet
W20251027 21:07:54.989291 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 9c2d713d72fd4b5ca34cb3e828c75761 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c2d713d72fd4b5ca34cb3e828c75761 already in progress: creating tablet
I20251027 21:07:54.989665 9750 tablet_service.cc:1505] Processing CreateTablet for tablet fe94124b239e442dbf644e4a483615fc (DEFAULT_TABLE table=default.Table_01129369d93842049a67974b77112073 [id=13f784ef7e0b4ede81d4d11e5e2a4138]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.990016 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe94124b239e442dbf644e4a483615fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.990414 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 7c15b74f53bc45e1af33b043f2cf225f (DEFAULT_TABLE table=default.Table_7c426eeeaca14f65ab36fef3162d4a3a [id=32fe13ae5b414c62aba38f34c648d6ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.990787 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c15b74f53bc45e1af33b043f2cf225f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.991792 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 0a58cd07e1bb4ffdbf39e7e16a46022a
I20251027 21:07:54.992316 9303 catalog_manager.cc:6719] ReplaceTablet: tablet 0a58cd07e1bb4ffdbf39e7e16a46022a deleted and replaced by tablet 4d88aacbf47144cb837b99d68f6331e2
I20251027 21:07:54.992507 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet d872cf3ef39042ff9a615b55e7195afa from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:54.994374 9741 tablet_service.cc:1505] Processing CreateTablet for tablet 423047dfbf084bfcb881efeb0352f88d (DEFAULT_TABLE table=default.Table_93662efa6bc64f759a615db6db157bab [id=25d8790f144f475d92de13eae144bc44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.994692 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423047dfbf084bfcb881efeb0352f88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.995005 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 46185d382f0a4dea803aad692ed3c89a (DEFAULT_TABLE table=default.Table_c6f57d0c3113494cb54754d67cfefa71 [id=068e2fce2c4a4915acd15991f2fdcc43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.995321 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46185d382f0a4dea803aad692ed3c89a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.998395 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 4c67c5cb3b244feb8478f9594b834fbf (DEFAULT_TABLE table=default.Table_e49807b03fc64f669376118d2ee01918 [id=642101fc044642829702e0ac1341329f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.998700 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c67c5cb3b244feb8478f9594b834fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.998979 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 27bd3b2a011648b2b8f2f92944513751
W20251027 21:07:55.000077 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67cb71ce43244d8ab9e71ead6bb315c9 consensus state has no leader
I20251027 21:07:55.001497 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 533f34d4872444278477abcce89da9ec
I20251027 21:07:55.001986 9309 catalog_manager.cc:6719] ReplaceTablet: tablet 533f34d4872444278477abcce89da9ec deleted and replaced by tablet 5054bbc3e27243298b4df6b0a87af8be
I20251027 21:07:55.002388 9747 tablet_service.cc:1505] Processing CreateTablet for tablet 93d2f7df14a74731aea84927f0b7ed45 (DEFAULT_TABLE table=default.Table_5502390635a249d5a5b5eaf0788236fd [id=c24a9debcdfc413ba606517b997cd19c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.002643 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 61ba8d7df6fd45abb6f5231ee8c4ec38
I20251027 21:07:55.002737 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d2f7df14a74731aea84927f0b7ed45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.006474 9744 tablet_service.cc:1505] Processing CreateTablet for tablet 2752b86151a6428c88bfaac1ff72cfb2 (DEFAULT_TABLE table=default.Table_110dd5476e984ca299e60f37066a8058 [id=ba21340acd344a59865d26d398ee1937]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.006855 10359 raft_consensus.cc:2749] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:55.006990 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2752b86151a6428c88bfaac1ff72cfb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.007066 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet d872cf3ef39042ff9a615b55e7195afa
I20251027 21:07:55.007990 9303 catalog_manager.cc:6719] ReplaceTablet: tablet d872cf3ef39042ff9a615b55e7195afa deleted and replaced by tablet 6f901626832f4adb948ccb65f8573e86
I20251027 21:07:55.009022 9900 tablet_service.cc:1505] Processing CreateTablet for tablet 9351a369dc1e4ac49f6ad419eab3a4c5 (DEFAULT_TABLE table=default.Table_3b49d647ca194ad9bf5524812c6837a2 [id=a12d4ab777c34431beaa8f87ddbc7135]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.009330 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 46185d382f0a4dea803aad692ed3c89a (DEFAULT_TABLE table=default.Table_c6f57d0c3113494cb54754d67cfefa71 [id=068e2fce2c4a4915acd15991f2fdcc43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.987383 9899 tablet_service.cc:1505] Processing CreateTablet for tablet fe94124b239e442dbf644e4a483615fc (DEFAULT_TABLE table=default.Table_01129369d93842049a67974b77112073 [id=13f784ef7e0b4ede81d4d11e5e2a4138]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.011545 10032 ts_tablet_manager.cc:1403] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.138s user 0.004s sys 0.000s
I20251027 21:07:55.012373 10032 raft_consensus.cc:359] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:55.012571 10032 raft_consensus.cc:385] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:55.012647 10032 raft_consensus.cc:740] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:55.012842 10032 consensus_queue.cc:260] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:55.013312 10032 ts_tablet_manager.cc:1434] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:07:55.013643 10032 tablet_bootstrap.cc:492] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:55.014048 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ae222307a374a91916f560a0fbe8d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.015408 10041 raft_consensus.cc:2243] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:55.017710 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 7c15b74f53bc45e1af33b043f2cf225f (DEFAULT_TABLE table=default.Table_7c426eeeaca14f65ab36fef3162d4a3a [id=32fe13ae5b414c62aba38f34c648d6ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.019171 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 4c67c5cb3b244feb8478f9594b834fbf (DEFAULT_TABLE table=default.Table_e49807b03fc64f669376118d2ee01918 [id=642101fc044642829702e0ac1341329f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.022228 9303 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 69da799c886841a19002d3c7c5084e73 (table default.Table_c2e56e5918cb4007b681c71977325233 [id=0cf4bae5f10d45c1a41880b45fa0b0f0]). Expected version 1 got 0
I20251027 21:07:55.024217 10041 raft_consensus.cc:2272] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:55.024749 10041 ts_tablet_manager.cc:1916] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:55.026923 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 423047dfbf084bfcb881efeb0352f88d (DEFAULT_TABLE table=default.Table_93662efa6bc64f759a615db6db157bab [id=25d8790f144f475d92de13eae144bc44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.027297 9886 tablet_service.cc:1505] Processing CreateTablet for tablet dafbb806923c4cf595abde7b7daa1637 (DEFAULT_TABLE table=default.Table_4863860f9b4c419fb46dca7a45e5b3ff [id=759288d4d08247518fb011022363036f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.027446 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe94124b239e442dbf644e4a483615fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.039439 9903 tablet_service.cc:1505] Processing CreateTablet for tablet 93d2f7df14a74731aea84927f0b7ed45 (DEFAULT_TABLE table=default.Table_5502390635a249d5a5b5eaf0788236fd [id=c24a9debcdfc413ba606517b997cd19c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.040716 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423047dfbf084bfcb881efeb0352f88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.041138 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7df6f0acec34d399aa25b78c3ed7c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.041553 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 2752b86151a6428c88bfaac1ff72cfb2 (DEFAULT_TABLE table=default.Table_110dd5476e984ca299e60f37066a8058 [id=ba21340acd344a59865d26d398ee1937]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.043429 9886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dafbb806923c4cf595abde7b7daa1637. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.043706 9894 tablet_service.cc:1552] Processing DeleteTablet for tablet 27bd3b2a011648b2b8f2f92944513751 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:55.044081 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2752b86151a6428c88bfaac1ff72cfb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.046238 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a58cd07e1bb4ffdbf39e7e16a46022a with delete_type TABLET_DATA_DELETED (replaced by tablet 4d88aacbf47144cb837b99d68f6331e2) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:55.046234 9894 tablet_service.cc:1552] Processing DeleteTablet for tablet 533f34d4872444278477abcce89da9ec with delete_type TABLET_DATA_DELETED (replaced by tablet 5054bbc3e27243298b4df6b0a87af8be) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:55.046612 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 61ba8d7df6fd45abb6f5231ee8c4ec38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:55.046854 9895 tablet_service.cc:1505] Processing CreateTablet for tablet fb7fdc91156e41de80872b5c941c5d95 (DEFAULT_TABLE table=default.Table_f1103102a7544791874c67fa7cadbbbe [id=08bb0e89b4514dafb2758847b79e64c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.047499 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46185d382f0a4dea803aad692ed3c89a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.048543 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c67c5cb3b244feb8478f9594b834fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.048569 9894 tablet_service.cc:1552] Processing DeleteTablet for tablet d872cf3ef39042ff9a615b55e7195afa with delete_type TABLET_DATA_DELETED (replaced by tablet 6f901626832f4adb948ccb65f8573e86) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:55.049173 9894 tablet_service.cc:1505] Processing CreateTablet for tablet e1385beb1dc14ee0b42a2f4e1f66f532 (DEFAULT_TABLE table=default.Table_92242f48713b48e5969d30e05bed6686 [id=3867dc60b7dd4012b55fc6e187d76b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.049580 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d2f7df14a74731aea84927f0b7ed45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.054229 9896 tablet_service.cc:1505] Processing CreateTablet for tablet e419e588b31d48c8bc19d7948081cc81 (DEFAULT_TABLE table=default.Table_c305a5155cce4754bcbc792f3900b690 [id=82b0b6e6d2e145a7870a13ed5b407b47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.058403 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c15b74f53bc45e1af33b043f2cf225f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.059252 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ff202033c9ec4fba9cc9525ea6767cac" } modify_external_catalogs: true
I20251027 21:07:55.059453 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ff202033c9ec4fba9cc9525ea6767cac" } modify_external_catalogs: true
I20251027 21:07:55.060046 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9c6b1e43fc5f4f4389620f1ee244db12" } modify_external_catalogs: true
I20251027 21:07:55.060150 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b730ec55ffb24974928bd0b0d9416460" } modify_external_catalogs: true
I20251027 21:07:55.060249 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9c6b1e43fc5f4f4389620f1ee244db12" } modify_external_catalogs: true
I20251027 21:07:55.060348 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b730ec55ffb24974928bd0b0d9416460" } modify_external_catalogs: true
I20251027 21:07:55.070361 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e419e588b31d48c8bc19d7948081cc81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.070911 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1385beb1dc14ee0b42a2f4e1f66f532. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.072911 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb7fdc91156e41de80872b5c941c5d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.073101 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 133e474f9133487f89c44dbfa6266ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.073570 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 61ba8d7df6fd45abb6f5231ee8c4ec38 (DEFAULT_TABLE table=default.Table_3f462ba7e3ab4705a1daa311661c4f3d [id=39097e9cb16c449682ec4fa71836e4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.074435 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9351a369dc1e4ac49f6ad419eab3a4c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.075915 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 4bbb69d62b9a41e3b88a5527f7549881
I20251027 21:07:55.076890 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 28a2e85021394f9d8a9ea524959b52a1
I20251027 21:07:55.081276 9893 tablet_service.cc:1505] Processing CreateTablet for tablet cc4635c00e9b40c49ce04debc76a04bf (DEFAULT_TABLE table=default.Table_3c1a4867db0849ceb59d5900eb57eb73 [id=1af04a7db72b4f939ffae873e6e3e688]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.081951 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet ccaf5c2f8e544842880ba8ba27300ae6
I20251027 21:07:55.082259 9901 tablet_service.cc:1505] Processing CreateTablet for tablet a7d8d060916941a5b7e944e8b08a2882 (DEFAULT_TABLE table=default.Table_b547b70a001f404fb96dc4f8fa6d70e9 [id=e1f4a77dfc0a497080c3c0ad4fe2e370]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.085765 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57ef07c764664deaa4b2e9bcae7e7973. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.086252 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61ba8d7df6fd45abb6f5231ee8c4ec38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.086460 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc4635c00e9b40c49ce04debc76a04bf. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:55.087066 9298 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 67cb71ce43244d8ab9e71ead6bb315c9 (table default.Table_a3f6fbc547e4462097650dbde068ef25 [id=3eb54d245ec64ee48022c541c2e49b17], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 67cb71ce43244d8ab9e71ead6bb315c9 consensus state has no leader
I20251027 21:07:55.087177 9901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7d8d060916941a5b7e944e8b08a2882. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.970649 9605 tablet_service.cc:1505] Processing CreateTablet for tablet fe94124b239e442dbf644e4a483615fc (DEFAULT_TABLE table=default.Table_01129369d93842049a67974b77112073 [id=13f784ef7e0b4ede81d4d11e5e2a4138]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.089012 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe94124b239e442dbf644e4a483615fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.972122 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 46185d382f0a4dea803aad692ed3c89a (DEFAULT_TABLE table=default.Table_c6f57d0c3113494cb54754d67cfefa71 [id=068e2fce2c4a4915acd15991f2fdcc43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.090436 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46185d382f0a4dea803aad692ed3c89a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.091542 9905 tablet_service.cc:1505] Processing CreateTablet for tablet e477ca4b69c84327b04618d90406e19a (DEFAULT_TABLE table=default.Table_7392db44fc434fd1a795922799aaf639 [id=2b5a90a29b994f1bad6491e84865172d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.094094 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e477ca4b69c84327b04618d90406e19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:54.976135 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 7c15b74f53bc45e1af33b043f2cf225f (DEFAULT_TABLE table=default.Table_7c426eeeaca14f65ab36fef3162d4a3a [id=32fe13ae5b414c62aba38f34c648d6ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.978020 9607 tablet_service.cc:1505] Processing CreateTablet for tablet dafbb806923c4cf595abde7b7daa1637 (DEFAULT_TABLE table=default.Table_4863860f9b4c419fb46dca7a45e5b3ff [id=759288d4d08247518fb011022363036f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.094483 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c15b74f53bc45e1af33b043f2cf225f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:55.094794 9711 leader_election.cc:343] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: efc0019840244d14bb95ae581793e74b
I20251027 21:07:55.097860 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_728cbf4f936c4df79d77156e2d3ac59c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:54.979686 9604 tablet_service.cc:1505] Processing CreateTablet for tablet 423047dfbf084bfcb881efeb0352f88d (DEFAULT_TABLE table=default.Table_93662efa6bc64f759a615db6db157bab [id=25d8790f144f475d92de13eae144bc44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.981352 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 4c67c5cb3b244feb8478f9594b834fbf (DEFAULT_TABLE table=default.Table_e49807b03fc64f669376118d2ee01918 [id=642101fc044642829702e0ac1341329f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.983021 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 93d2f7df14a74731aea84927f0b7ed45 (DEFAULT_TABLE table=default.Table_5502390635a249d5a5b5eaf0788236fd [id=c24a9debcdfc413ba606517b997cd19c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.984822 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 2752b86151a6428c88bfaac1ff72cfb2 (DEFAULT_TABLE table=default.Table_110dd5476e984ca299e60f37066a8058 [id=ba21340acd344a59865d26d398ee1937]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:54.986570 9615 tablet_service.cc:1505] Processing CreateTablet for tablet e419e588b31d48c8bc19d7948081cc81 (DEFAULT_TABLE table=default.Table_c305a5155cce4754bcbc792f3900b690 [id=82b0b6e6d2e145a7870a13ed5b407b47]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:55.098423 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_728cbf4f936c4df79d77156e2d3ac59c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:54.994568 9598 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a58cd07e1bb4ffdbf39e7e16a46022a with delete_type TABLET_DATA_DELETED (replaced by tablet 4d88aacbf47144cb837b99d68f6331e2) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:54.999724 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet 27bd3b2a011648b2b8f2f92944513751 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:55.098861 9598 tablet_service.cc:1505] Processing CreateTablet for tablet a7d8d060916941a5b7e944e8b08a2882 (DEFAULT_TABLE table=default.Table_b547b70a001f404fb96dc4f8fa6d70e9 [id=e1f4a77dfc0a497080c3c0ad4fe2e370]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.098920 9606 tablet_service.cc:1505] Processing CreateTablet for tablet e477ca4b69c84327b04618d90406e19a (DEFAULT_TABLE table=default.Table_7392db44fc434fd1a795922799aaf639 [id=2b5a90a29b994f1bad6491e84865172d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.002175 9612 tablet_service.cc:1552] Processing DeleteTablet for tablet 533f34d4872444278477abcce89da9ec with delete_type TABLET_DATA_DELETED (replaced by tablet 5054bbc3e27243298b4df6b0a87af8be) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:55.099364 9612 tablet_service.cc:1505] Processing CreateTablet for tablet 815e12edcd3847ff8485ba768291a824 (DEFAULT_TABLE table=default.Table_2992b8e207024e06957ce7befb657acb [id=decffeb1481e4d798cffd8c9883be59b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.003250 9608 tablet_service.cc:1552] Processing DeleteTablet for tablet 61ba8d7df6fd45abb6f5231ee8c4ec38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:55.099757 9608 tablet_service.cc:1505] Processing CreateTablet for tablet 93bebcb636ec466b81453d90aa1bbd0f (DEFAULT_TABLE table=default.Table_2d1949f2a17c47748742fda3abdaaada [id=cd680fbc181a4836bd27a9d6338625f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.008173 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet d872cf3ef39042ff9a615b55e7195afa with delete_type TABLET_DATA_DELETED (replaced by tablet 6f901626832f4adb948ccb65f8573e86) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:55.100147 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 618f1657a10947fd98976820564c20b3 (DEFAULT_TABLE table=default.Table_7929c591af15477aa34ef5952495dfc6 [id=822f0869aaa946c69c92b5b27d715621]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.011791 9595 tablet_service.cc:1505] Processing CreateTablet for tablet e1385beb1dc14ee0b42a2f4e1f66f532 (DEFAULT_TABLE table=default.Table_92242f48713b48e5969d30e05bed6686 [id=3867dc60b7dd4012b55fc6e187d76b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.012943 9603 tablet_service.cc:1505] Processing CreateTablet for tablet 61ba8d7df6fd45abb6f5231ee8c4ec38 (DEFAULT_TABLE table=default.Table_3f462ba7e3ab4705a1daa311661c4f3d [id=39097e9cb16c449682ec4fa71836e4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.014290 9613 tablet_service.cc:1505] Processing CreateTablet for tablet cc4635c00e9b40c49ce04debc76a04bf (DEFAULT_TABLE table=default.Table_3c1a4867db0849ceb59d5900eb57eb73 [id=1af04a7db72b4f939ffae873e6e3e688]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.100584 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423047dfbf084bfcb881efeb0352f88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.100790 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc4635c00e9b40c49ce04debc76a04bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.100977 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c67c5cb3b244feb8478f9594b834fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.105669 10411 raft_consensus.cc:493] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:55.105796 10411 raft_consensus.cc:515] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:55.106359 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d2f7df14a74731aea84927f0b7ed45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.106437 10411 leader_election.cc:290] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:55.108129 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_acf5bf3b68d24276bfd6df1b4f2015fd" } new_table_name: "default.Table_a7632420526d409290924efb41972ee1" modify_external_catalogs: true
I20251027 21:07:55.109489 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 815e12edcd3847ff8485ba768291a824 (DEFAULT_TABLE table=default.Table_2992b8e207024e06957ce7befb657acb [id=decffeb1481e4d798cffd8c9883be59b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.109797 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 815e12edcd3847ff8485ba768291a824. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.110527 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8fc2d7d6601d4cc2849064f4cdddc90d" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:55.110963 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2752b86151a6428c88bfaac1ff72cfb2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:55.111016 9860 leader_election.cc:343] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:55.113674 9613 tablet_service.cc:1505] Processing CreateTablet for tablet d0c53b7b32c84a51a967e2a6a22258c6 (DEFAULT_TABLE table=default.Table_6845351c1f1949a9932e39a805cc4033 [id=d842fe6c9377442dbd25a4c103767aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.114022 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0c53b7b32c84a51a967e2a6a22258c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.114380 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e419e588b31d48c8bc19d7948081cc81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.010525 9609 tablet_service.cc:1505] Processing CreateTablet for tablet fb7fdc91156e41de80872b5c941c5d95 (DEFAULT_TABLE table=default.Table_f1103102a7544791874c67fa7cadbbbe [id=08bb0e89b4514dafb2758847b79e64c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.118425 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb7fdc91156e41de80872b5c941c5d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.119942 9900 tablet_service.cc:1505] Processing CreateTablet for tablet 93bebcb636ec466b81453d90aa1bbd0f (DEFAULT_TABLE table=default.Table_2d1949f2a17c47748742fda3abdaaada [id=cd680fbc181a4836bd27a9d6338625f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.120289 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93bebcb636ec466b81453d90aa1bbd0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.122481 9613 tablet_service.cc:1505] Processing CreateTablet for tablet 532ea47b15b544dfb05ddb2bdacf3953 (DEFAULT_TABLE table=default.Table_c9a81e9dabad4a7d9fbd427439a7587c [id=40ce2a0befe54d698759beff3bfdf0f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.122817 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 532ea47b15b544dfb05ddb2bdacf3953. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.123888 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 618f1657a10947fd98976820564c20b3 (DEFAULT_TABLE table=default.Table_7929c591af15477aa34ef5952495dfc6 [id=822f0869aaa946c69c92b5b27d715621]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.124244 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 618f1657a10947fd98976820564c20b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.126442 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7d8d060916941a5b7e944e8b08a2882. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.126626 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e477ca4b69c84327b04618d90406e19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.126955 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 815e12edcd3847ff8485ba768291a824. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.128832 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93bebcb636ec466b81453d90aa1bbd0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.129489 9903 tablet_service.cc:1505] Processing CreateTablet for tablet d0c53b7b32c84a51a967e2a6a22258c6 (DEFAULT_TABLE table=default.Table_6845351c1f1949a9932e39a805cc4033 [id=d842fe6c9377442dbd25a4c103767aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.129801 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0c53b7b32c84a51a967e2a6a22258c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.130363 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 618f1657a10947fd98976820564c20b3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:55.131860 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07a9c3b7ef1548ec9d5ae9f20545f263 consensus state has no leader
I20251027 21:07:55.133787 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 93a31187eca14b8797737936a80e7292 (DEFAULT_TABLE table=default.Table_b126a056ef2c41b28a178327ef69e524 [id=69a8a630d7ab4b2d8c7b580b41ebe3ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.134184 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93a31187eca14b8797737936a80e7292. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.134423 9595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1385beb1dc14ee0b42a2f4e1f66f532. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.136286 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61ba8d7df6fd45abb6f5231ee8c4ec38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.136843 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dafbb806923c4cf595abde7b7daa1637. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.137104 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 532ea47b15b544dfb05ddb2bdacf3953 (DEFAULT_TABLE table=default.Table_c9a81e9dabad4a7d9fbd427439a7587c [id=40ce2a0befe54d698759beff3bfdf0f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.137429 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 532ea47b15b544dfb05ddb2bdacf3953. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.137867 10411 raft_consensus.cc:493] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:55.138054 10411 raft_consensus.cc:515] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:55.139218 10411 leader_election.cc:290] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:55.139705 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b547b70a001f404fb96dc4f8fa6d70e9" } new_table_name: "default.Table_a4ae12c512d645e58eadc2daa1bb7b0a" modify_external_catalogs: true
I20251027 21:07:55.140839 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78884c2be75f45f881f5424d3b3f4e08" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:55.141347 9860 leader_election.cc:343] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:55.142066 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 93a31187eca14b8797737936a80e7292 (DEFAULT_TABLE table=default.Table_b126a056ef2c41b28a178327ef69e524 [id=69a8a630d7ab4b2d8c7b580b41ebe3ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.142387 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93a31187eca14b8797737936a80e7292. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.143040 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8840cffd9b454f6ba433d8bc28a3dd81" } new_table_name: "default.Table_eb44e5cb7bc24597a807d81c1207a162" modify_external_catalogs: true
I20251027 21:07:55.144091 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_fad068c83e3f4abeb0eefca5ed055111" } new_table_name: "default.Table_f20babe6e88047de940c58738a6d786c" modify_external_catalogs: true
I20251027 21:07:55.147351 9595 tablet_service.cc:1505] Processing CreateTablet for tablet e04f4c5cf724419ab0be0e84d30a9a57 (DEFAULT_TABLE table=default.Table_5315576a7f87464dbbfded0081ddef2d [id=ca2d675d800a46728d7b11ee50f2e056]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.147675 9595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e04f4c5cf724419ab0be0e84d30a9a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.151748 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_64efdeb42d0f4db68c0e13066cd8d38f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.152274 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64efdeb42d0f4db68c0e13066cd8d38f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.153621 9888 tablet_service.cc:1505] Processing CreateTablet for tablet e04f4c5cf724419ab0be0e84d30a9a57 (DEFAULT_TABLE table=default.Table_5315576a7f87464dbbfded0081ddef2d [id=ca2d675d800a46728d7b11ee50f2e056]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.154763 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e04f4c5cf724419ab0be0e84d30a9a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.155304 9611 tablet_service.cc:1505] Processing CreateTablet for tablet e9b8f76c33444bcb9b488af65e4d3e6b (DEFAULT_TABLE table=default.Table_3d031fe204804a95b5bc07e8b54573d8 [id=855df1b953454ddeac8efc6e5869db2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.154462 9898 tablet_service.cc:1505] Processing CreateTablet for tablet e9b8f76c33444bcb9b488af65e4d3e6b (DEFAULT_TABLE table=default.Table_3d031fe204804a95b5bc07e8b54573d8 [id=855df1b953454ddeac8efc6e5869db2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.155727 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9b8f76c33444bcb9b488af65e4d3e6b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:55.157569 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a7d8d060916941a5b7e944e8b08a2882 consensus state has no leader
I20251027 21:07:55.158581 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bbb69d62b9a41e3b88a5527f7549881 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:55.159824 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet 28a2e85021394f9d8a9ea524959b52a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:55.160132 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 4bc5fbdbbea0406a961e1a4df51b8762 (DEFAULT_TABLE table=default.Table_cc31649fd9f544e985ed00eb483164ea [id=4154657a736742b8a4f7dfbd254209c4]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:55.160545 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c97cba61e08d42a7bdb3f51db0a63ec8 consensus state has no leader
W20251027 21:07:55.160884 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2710f265ae9485dafec389168328067 consensus state has no leader
I20251027 21:07:55.162534 9614 tablet_service.cc:1505] Processing CreateTablet for tablet c27c699cc2744f40811f7b9395ca76be (DEFAULT_TABLE table=default.Table_691240a058cc4f879e5737f73a4c12e7 [id=697732b24e12465d8caac4e23de633bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.164928 9888 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bbb69d62b9a41e3b88a5527f7549881 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:55.165663 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 479d2cc8b8ce4c26841f50eea7c0e09a (DEFAULT_TABLE table=default.Table_9601abc37cc5482aa4b9a13b818e9074 [id=9a181744a9954b8388e6417fa7f8a630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.165802 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet ccaf5c2f8e544842880ba8ba27300ae6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:55.167825 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 4bc5fbdbbea0406a961e1a4df51b8762 (DEFAULT_TABLE table=default.Table_cc31649fd9f544e985ed00eb483164ea [id=4154657a736742b8a4f7dfbd254209c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.167845 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 67994a7cacd94b2ebb16fe154832c8fd (DEFAULT_TABLE table=default.Table_a328d75fa0f44ad9b92e05cd96556f06 [id=3a4883bc82ba48099b2e20589d7eef61]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.167932 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 0820648719a34f379507efced9cc434b (DEFAULT_TABLE table=default.Table_8291d411936d4f4e8d267fdc28efc533 [id=b0301c76f48f461c8ee32b925c91234b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.167526 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet 28a2e85021394f9d8a9ea524959b52a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:55.168399 9898 tablet_service.cc:1505] Processing CreateTablet for tablet c27c699cc2744f40811f7b9395ca76be (DEFAULT_TABLE table=default.Table_691240a058cc4f879e5737f73a4c12e7 [id=697732b24e12465d8caac4e23de633bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.168744 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bc5fbdbbea0406a961e1a4df51b8762. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.170568 9609 tablet_service.cc:1505] Processing CreateTablet for tablet f19b5e8e460a4c4ba7399b981c603b84 (DEFAULT_TABLE table=default.Table_7d20b20b1f274f4f94a9497e2d030235 [id=ab4e770ed197499ba9751f3061b2aeba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.170840 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c27c699cc2744f40811f7b9395ca76be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.171813 9613 tablet_service.cc:1505] Processing CreateTablet for tablet e3654aa289154b3ea613b6ff94cbd57e (DEFAULT_TABLE table=default.Table_c8cb97c3834e406ca7d8a2d047d7d249 [id=4c67737c2c734ad0ad33349fa69668bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.173475 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 95f9595784df434dbb0dd29d9a05511e (DEFAULT_TABLE table=default.Table_3ed917cd5e964f3b8f54abfcf705aba7 [id=9a42e4bcf97c4d45a5f6ec342844a805]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.174201 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9a6d7cca704541ca80a3cf047aea2914"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.174717 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a6d7cca704541ca80a3cf047aea2914 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.175937 9604 tablet_service.cc:1505] Processing CreateTablet for tablet 7c6c2292ad0e4066958897fff01370af (DEFAULT_TABLE table=default.Table_7a535e8483da496498c4c7f7e76d6670 [id=53f813baffa448c990d467d8267929ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.176301 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7453f892d0514227b015e34cf2461acd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.176739 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7453f892d0514227b015e34cf2461acd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.177904 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_fe40346a4d3641159b11c6b9ec566eca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:55.178305 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 72a6ca830c904373a4cca1169ae391ce (DEFAULT_TABLE table=default.Table_011f55ad6c164cdd8753c07e2359334c [id=6a6d9881d4b4433aa97e2eec32010e9e]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:55.178435 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe40346a4d3641159b11c6b9ec566eca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.060815 10029 tablet_bootstrap.cc:492] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:55.179839 10029 ts_tablet_manager.cc:1403] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.250s user 0.004s sys 0.000s
I20251027 21:07:55.180603 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 479d2cc8b8ce4c26841f50eea7c0e09a (DEFAULT_TABLE table=default.Table_9601abc37cc5482aa4b9a13b818e9074 [id=9a181744a9954b8388e6417fa7f8a630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.180596 10029 raft_consensus.cc:359] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:55.180775 10029 raft_consensus.cc:385] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:55.180850 10029 raft_consensus.cc:740] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:55.180922 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 479d2cc8b8ce4c26841f50eea7c0e09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.036559 10358 raft_consensus.cc:493] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:55.181099 10029 consensus_queue.cc:260] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:55.181200 10358 raft_consensus.cc:515] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:55.181596 10029 ts_tablet_manager.cc:1434] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:07:55.181876 10358 leader_election.cc:290] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:55.183979 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet ccaf5c2f8e544842880ba8ba27300ae6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:55.184106 9568 leader_election.cc:343] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 533f231e3ff64531ae2331409a961425
I20251027 21:07:55.184572 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 0820648719a34f379507efced9cc434b (DEFAULT_TABLE table=default.Table_8291d411936d4f4e8d267fdc28efc533 [id=b0301c76f48f461c8ee32b925c91234b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.200525 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0820648719a34f379507efced9cc434b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.208631 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67994a7cacd94b2ebb16fe154832c8fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.209250 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c27c699cc2744f40811f7b9395ca76be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.209682 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 479d2cc8b8ce4c26841f50eea7c0e09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.210041 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9b8f76c33444bcb9b488af65e4d3e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.214265 10029 tablet_bootstrap.cc:492] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:55.219666 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c6c2292ad0e4066958897fff01370af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.085029 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 8ae222307a374a91916f560a0fbe8d8f (DEFAULT_TABLE table=default.Table_6c9db36d368044e487f90ca694832333 [id=d89790f6beb646f8b2580feeddea18f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.225205 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 23673513189f47638dbbc737da0991d1 (DEFAULT_TABLE table=default.Table_bae8db615e424af398f1f9baec11cb76 [id=cc1da530aa95455e90889dd5ea3768c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.192605 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 67994a7cacd94b2ebb16fe154832c8fd (DEFAULT_TABLE table=default.Table_a328d75fa0f44ad9b92e05cd96556f06 [id=3a4883bc82ba48099b2e20589d7eef61]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.196508 9902 tablet_service.cc:1505] Processing CreateTablet for tablet e3654aa289154b3ea613b6ff94cbd57e (DEFAULT_TABLE table=default.Table_c8cb97c3834e406ca7d8a2d047d7d249 [id=4c67737c2c734ad0ad33349fa69668bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.196022 9891 tablet_service.cc:1505] Processing CreateTablet for tablet f19b5e8e460a4c4ba7399b981c603b84 (DEFAULT_TABLE table=default.Table_7d20b20b1f274f4f94a9497e2d030235 [id=ab4e770ed197499ba9751f3061b2aeba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.199471 9901 tablet_service.cc:1505] Processing CreateTablet for tablet 95f9595784df434dbb0dd29d9a05511e (DEFAULT_TABLE table=default.Table_3ed917cd5e964f3b8f54abfcf705aba7 [id=9a42e4bcf97c4d45a5f6ec342844a805]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.213303 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 7c6c2292ad0e4066958897fff01370af (DEFAULT_TABLE table=default.Table_7a535e8483da496498c4c7f7e76d6670 [id=53f813baffa448c990d467d8267929ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.214277 10032 tablet_bootstrap.cc:654] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:55.222007 10041 ts_tablet_manager.cc:1929] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:55.226562 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5181011f23634d8a91c3a50d531d951d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.227226 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5181011f23634d8a91c3a50d531d951d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.228578 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3e21b34c57774bd2843bbac07d8892ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:55.228710 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_84fa0625821f4deeb3422622babb5429" } new_table_name: "default.Table_32f38256207d4fa79ed533151449afa7" modify_external_catalogs: true
W20251027 21:07:55.229079 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e21b34c57774bd2843bbac07d8892ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.229894 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c90c85540a2142218c304c8da1a2df90" } modify_external_catalogs: true
I20251027 21:07:55.230093 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c90c85540a2142218c304c8da1a2df90" } modify_external_catalogs: true
I20251027 21:07:55.230170 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6f6b1a85f5f2410f8c36649b6a7b44a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:55.230387 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 27bd3b2a011648b2b8f2f92944513751 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:55.230618 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f6b1a85f5f2410f8c36649b6a7b44a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.230686 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "533f231e3ff64531ae2331409a961425" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:55.230882 9743 tablet_service.cc:1505] Processing CreateTablet for tablet e419e588b31d48c8bc19d7948081cc81 (DEFAULT_TABLE table=default.Table_c305a5155cce4754bcbc792f3900b690 [id=82b0b6e6d2e145a7870a13ed5b407b47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.231295 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e419e588b31d48c8bc19d7948081cc81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.231371 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_858fba8378424b5399351d9134fa7e60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.231788 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_858fba8378424b5399351d9134fa7e60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.231994 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 533f34d4872444278477abcce89da9ec with delete_type TABLET_DATA_DELETED (replaced by tablet 5054bbc3e27243298b4df6b0a87af8be) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:55.232095 10359 raft_consensus.cc:493] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:55.232021 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet 61ba8d7df6fd45abb6f5231ee8c4ec38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:55.232010 10031 tablet_bootstrap.cc:492] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:55.232043 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a58cd07e1bb4ffdbf39e7e16a46022a with delete_type TABLET_DATA_DELETED (replaced by tablet 4d88aacbf47144cb837b99d68f6331e2) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:55.233151 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9601abc37cc5482aa4b9a13b818e9074" } modify_external_catalogs: true
I20251027 21:07:55.233304 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9601abc37cc5482aa4b9a13b818e9074" } modify_external_catalogs: true
W20251027 21:07:55.234350 9566 leader_election.cc:343] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:55.231806 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_98ca10ad3609477aaf22dc563a68742e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.235113 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98ca10ad3609477aaf22dc563a68742e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.233385 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a6ca830c904373a4cca1169ae391ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.237490 9566 leader_election.cc:304] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:55.237932 10358 raft_consensus.cc:2749] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:55.237919 9711 leader_election.cc:304] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:55.238111 9612 tablet_service.cc:1505] Processing CreateTablet for tablet c548d2413a5940c8b114ced667b4dc41 (DEFAULT_TABLE table=default.Table_907ada415b694cf4817ef1e6f6f68e9d [id=ec3ae7f22bea4f9d8b7ba9c5c4a3b2c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.239106 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet d872cf3ef39042ff9a615b55e7195afa with delete_type TABLET_DATA_DELETED (replaced by tablet 6f901626832f4adb948ccb65f8573e86) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:55.240296 10031 ts_tablet_manager.cc:1403] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.401s user 0.002s sys 0.002s
I20251027 21:07:55.239241 9737 tablet_service.cc:1505] Processing CreateTablet for tablet dafbb806923c4cf595abde7b7daa1637 (DEFAULT_TABLE table=default.Table_4863860f9b4c419fb46dca7a45e5b3ff [id=759288d4d08247518fb011022363036f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.246037 10031 raft_consensus.cc:359] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:55.246264 10031 raft_consensus.cc:385] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:55.246356 10031 raft_consensus.cc:740] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:55.246552 10031 consensus_queue.cc:260] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:55.247918 10359 raft_consensus.cc:515] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:55.248551 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dafbb806923c4cf595abde7b7daa1637. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:55.253260 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed5619d295a246379b625f35692203e4 consensus state has no leader
I20251027 21:07:55.259318 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 874d5974698c448ab79f19812cb0e886
I20251027 21:07:55.263610 10359 leader_election.cc:290] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:55.267761 10359 raft_consensus.cc:2749] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:55.273348 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 479d2cc8b8ce4c26841f50eea7c0e09a
I20251027 21:07:55.279721 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4658a756ae8149cf9e248c65a102bdb6" } modify_external_catalogs: true
I20251027 21:07:55.279917 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4658a756ae8149cf9e248c65a102bdb6" } modify_external_catalogs: true
I20251027 21:07:55.281230 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ed23972576b74b5fbc7167cf7b541bfc" } modify_external_catalogs: true
I20251027 21:07:55.281389 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ed23972576b74b5fbc7167cf7b541bfc" } modify_external_catalogs: true
I20251027 21:07:55.281785 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 72a6ca830c904373a4cca1169ae391ce (DEFAULT_TABLE table=default.Table_011f55ad6c164cdd8753c07e2359334c [id=6a6d9881d4b4433aa97e2eec32010e9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.287557 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1d766c40686340d59cc4680fbf8a2dbb" } modify_external_catalogs: true
I20251027 21:07:55.287719 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1d766c40686340d59cc4680fbf8a2dbb" } modify_external_catalogs: true
I20251027 21:07:55.289762 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 631ff72613734913818a3cad24f20dbc
I20251027 21:07:55.290491 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 37ae3b6bf590473e8e09092fc5f7df1b
I20251027 21:07:55.295113 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 92a2393cff8044028c86c8cd2bccd930
I20251027 21:07:55.325899 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f846038ed8334d998a32d5775f2b0403"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.326757 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f846038ed8334d998a32d5775f2b0403 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.334261 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2994a62669894e3baf66db0e580ece68" } new_table_name: "default.Table_735a5a1d675c4c44a98e6793ab7f21fd" modify_external_catalogs: true
W20251027 21:07:55.344671 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 29693cf06bc24930870dc60457126519 consensus state has no leader
I20251027 21:07:55.357260 10043 ts_tablet_manager.cc:1916] T a9cc64ad85334daa9a09538e24ebd8a5 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:55.359125 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f8b57b86738a4f61b6a63a06b73016dc" } new_table_name: "default.Table_c37baeeee577492c99b99ee09a3781db" modify_external_catalogs: true
I20251027 21:07:55.363847 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5181011f23634d8a91c3a50d531d951d" } new_table_name: "default.Table_20d9ef0583ca492a85be3bf2cdf16b5b" modify_external_catalogs: true
I20251027 21:07:55.365062 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6c9db36d368044e487f90ca694832333" } new_table_name: "default.Table_e05ab59d2e0149c7b23dfba66d1bc2c2" modify_external_catalogs: true
I20251027 21:07:55.369731 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_24e18a6f669742d696b1929a9342ac9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:55.369983 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c548d2413a5940c8b114ced667b4dc41. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:55.370311 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24e18a6f669742d696b1929a9342ac9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:55.374890 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b83daa33ef0f45c6a78c70001e532ea2 has no consensus state
W20251027 21:07:55.375432 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ae222307a374a91916f560a0fbe8d8f consensus state has no leader
I20251027 21:07:55.375512 9595 tablet_service.cc:1505] Processing CreateTablet for tablet 4d2c40f983e743b0b53f987b6a7ba437 (DEFAULT_TABLE table=default.Table_73afc9d8e0d740d098eb77c4018c3cb2 [id=272264f0d3d347f0bd782edcebfec854]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.376681 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23673513189f47638dbbc737da0991d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.384364 10041 log.cc:1199] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/1e5f5b57f5d948f3b2352451ccaacb09
W20251027 21:07:55.384421 9710 leader_election.cc:343] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: c49775486ff24686b0f45c47e2757aba
I20251027 21:07:55.390933 9598 tablet_service.cc:1505] Processing CreateTablet for tablet 54c2b10e79e149c2a637475e1f518907 (DEFAULT_TABLE table=default.Table_602a5846b60c4860b33575da44b5015b [id=9f1e545ab3174e8a893b9183cf95768d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:55.391880 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe7b00e9cf984d12817c9ea860391199 consensus state has no leader
I20251027 21:07:55.395713 9606 tablet_service.cc:1505] Processing CreateTablet for tablet c8f0cd910b9d4daea287b3b192a81bd2 (DEFAULT_TABLE table=default.Table_7ed2c827a45d4c80a5416d12258c77c1 [id=f7d6376018f64583981d26e14762cbaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.395853 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3d352cd73ac64a89ade7b276936b52cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:55.396301 10041 ts_tablet_manager.cc:1950] T 1e5f5b57f5d948f3b2352451ccaacb09 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
W20251027 21:07:55.396375 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d352cd73ac64a89ade7b276936b52cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.397897 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9b0fd27b43c842008090999fe7da9233"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.398396 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b0fd27b43c842008090999fe7da9233 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.399825 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3654aa289154b3ea613b6ff94cbd57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.403273 9603 tablet_service.cc:1505] Processing CreateTablet for tablet fe7b00e9cf984d12817c9ea860391199 (DEFAULT_TABLE table=default.Table_f8b57b86738a4f61b6a63a06b73016dc [id=9192a31ca6324aa2a919311093d54ae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.404398 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_abc7f8be9f414322ad9064340647399f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.404910 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abc7f8be9f414322ad9064340647399f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.405102 9896 tablet_service.cc:1505] Processing CreateTablet for tablet c548d2413a5940c8b114ced667b4dc41 (DEFAULT_TABLE table=default.Table_907ada415b694cf4817ef1e6f6f68e9d [id=ec3ae7f22bea4f9d8b7ba9c5c4a3b2c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.406844 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 3ee02a05b5ae46e7bf5cf9fdd3e9a80f (DEFAULT_TABLE table=default.Table_f5b100cf12504aeaa64ef743bba49f18 [id=8c32f511b11d422fa461408a4e9d4e0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.408115 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f19b5e8e460a4c4ba7399b981c603b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.408967 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8f0cd910b9d4daea287b3b192a81bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.414911 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe7b00e9cf984d12817c9ea860391199. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.416707 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3654aa289154b3ea613b6ff94cbd57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.417240 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54c2b10e79e149c2a637475e1f518907. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.418076 9901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f9595784df434dbb0dd29d9a05511e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.422113 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a6ca830c904373a4cca1169ae391ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.422415 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67994a7cacd94b2ebb16fe154832c8fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.422873 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c6c2292ad0e4066958897fff01370af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.423350 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c548d2413a5940c8b114ced667b4dc41. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.431901 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0820648719a34f379507efced9cc434b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.432945 9608 tablet_service.cc:1505] Processing CreateTablet for tablet 674e624257f84c47acff5d61f1fd5b54 (DEFAULT_TABLE table=default.Table_b0eb812d94f1489892d520a5b4df323c [id=497a00b32737421bbfec0c6205350263]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.433349 9595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d2c40f983e743b0b53f987b6a7ba437. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.436833 10411 raft_consensus.cc:493] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:55.437012 10411 raft_consensus.cc:515] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:55.437726 10411 leader_election.cc:290] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:55.438393 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f19b5e8e460a4c4ba7399b981c603b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.440627 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 54c2b10e79e149c2a637475e1f518907 (DEFAULT_TABLE table=default.Table_602a5846b60c4860b33575da44b5015b [id=9f1e545ab3174e8a893b9183cf95768d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.440929 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54c2b10e79e149c2a637475e1f518907. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.444882 9886 tablet_service.cc:1505] Processing CreateTablet for tablet 4d2c40f983e743b0b53f987b6a7ba437 (DEFAULT_TABLE table=default.Table_73afc9d8e0d740d098eb77c4018c3cb2 [id=272264f0d3d347f0bd782edcebfec854]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.445348 9886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d2c40f983e743b0b53f987b6a7ba437. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.445725 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f0e8c68b9f1243bfb3584cca5b5f7002"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.446331 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0e8c68b9f1243bfb3584cca5b5f7002 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.452546 9905 tablet_service.cc:1505] Processing CreateTablet for tablet c8f0cd910b9d4daea287b3b192a81bd2 (DEFAULT_TABLE table=default.Table_7ed2c827a45d4c80a5416d12258c77c1 [id=f7d6376018f64583981d26e14762cbaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.452913 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8f0cd910b9d4daea287b3b192a81bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.454790 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a356e5bd7b1c40a087ef83c7078e0381" } new_table_name: "default.Table_6b471834498c4afeb57279278d45d9a9" modify_external_catalogs: true
I20251027 21:07:55.459726 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_cf9f3cddff8a486fbe76334c2fd833c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.460250 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf9f3cddff8a486fbe76334c2fd833c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.461966 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3ed917cd5e964f3b8f54abfcf705aba7" } modify_external_catalogs: true
I20251027 21:07:55.462057 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 23673513189f47638dbbc737da0991d1 (DEFAULT_TABLE table=default.Table_bae8db615e424af398f1f9baec11cb76 [id=cc1da530aa95455e90889dd5ea3768c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.462170 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3ed917cd5e964f3b8f54abfcf705aba7" } modify_external_catalogs: true
I20251027 21:07:55.464705 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_ccc943e7fbc54e609e33b3a131a07c74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.465178 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccc943e7fbc54e609e33b3a131a07c74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.465216 10411 raft_consensus.cc:493] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:55.466250 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23673513189f47638dbbc737da0991d1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:55.467267 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d13d00353b6740b5bed5a153936d2059 consensus state has no leader
I20251027 21:07:55.470813 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3c0c40b875fc45969c1a916c5545c4cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:55.470947 10411 raft_consensus.cc:515] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:55.471323 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c0c40b875fc45969c1a916c5545c4cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.471720 10411 leader_election.cc:290] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:55.474516 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 95f9595784df434dbb0dd29d9a05511e
I20251027 21:07:55.475349 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7b970942803b42f29feb36d6d845167a" } modify_external_catalogs: true
I20251027 21:07:55.475508 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7b970942803b42f29feb36d6d845167a" } modify_external_catalogs: true
I20251027 21:07:55.480165 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_1a51e839945b4912a5519bb1f21a6b75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.480608 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a51e839945b4912a5519bb1f21a6b75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.487040 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 3ee02a05b5ae46e7bf5cf9fdd3e9a80f (DEFAULT_TABLE table=default.Table_f5b100cf12504aeaa64ef743bba49f18 [id=8c32f511b11d422fa461408a4e9d4e0f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:55.488719 9904 consensus_meta.cc:287] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent flushing consensus metadata: real 0.613s user 0.003s sys 0.000s
I20251027 21:07:55.492856 9903 tablet_service.cc:1505] Processing CreateTablet for tablet fe7b00e9cf984d12817c9ea860391199 (DEFAULT_TABLE table=default.Table_f8b57b86738a4f61b6a63a06b73016dc [id=9192a31ca6324aa2a919311093d54ae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.493427 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 11627a84ff9b478c87c323617a09570b
I20251027 21:07:55.495220 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_61bbf6b5c1e740128b80465ef3b9a50e" } modify_external_catalogs: true
I20251027 21:07:55.495406 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_61bbf6b5c1e740128b80465ef3b9a50e" } modify_external_catalogs: true
I20251027 21:07:55.495529 9899 tablet_service.cc:1505] Processing CreateTablet for tablet ae0ee7cd2c1b406f84285f25420588d0 (DEFAULT_TABLE table=default.Table_e2cb137bea0d419b9791305f97bc0efd [id=251aef26011f4b728b1102e4570722b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.495819 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 674e624257f84c47acff5d61f1fd5b54 (DEFAULT_TABLE table=default.Table_b0eb812d94f1489892d520a5b4df323c [id=497a00b32737421bbfec0c6205350263]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.496922 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7ed2c827a45d4c80a5416d12258c77c1" } modify_external_catalogs: true
I20251027 21:07:55.497221 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 1e5f5b57f5d948f3b2352451ccaacb09 (table default.Table_f0792045d49041578f5ff6459c99f77e [id=7f83f7c0f9524147bb778a06dd233a8e]) successfully deleted
I20251027 21:07:55.497426 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_858fba8378424b5399351d9134fa7e60" } modify_external_catalogs: true
I20251027 21:07:55.497582 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_858fba8378424b5399351d9134fa7e60" } modify_external_catalogs: true
W20251027 21:07:55.497898 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 27bd3b2a011648b2b8f2f92944513751 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27bd3b2a011648b2b8f2f92944513751 already in progress: creating tablet
W20251027 21:07:55.498329 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 0a58cd07e1bb4ffdbf39e7e16a46022a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a58cd07e1bb4ffdbf39e7e16a46022a already in progress: creating tablet
W20251027 21:07:55.498688 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 533f34d4872444278477abcce89da9ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 533f34d4872444278477abcce89da9ec already in progress: creating tablet
I20251027 21:07:55.497171 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7ed2c827a45d4c80a5416d12258c77c1" } modify_external_catalogs: true
W20251027 21:07:55.499033 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 61ba8d7df6fd45abb6f5231ee8c4ec38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61ba8d7df6fd45abb6f5231ee8c4ec38 already in progress: creating tablet
I20251027 21:07:55.499276 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ee02a05b5ae46e7bf5cf9fdd3e9a80f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:55.500214 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet d872cf3ef39042ff9a615b55e7195afa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d872cf3ef39042ff9a615b55e7195afa already in progress: creating tablet
W20251027 21:07:55.501288 9299 catalog_manager.cc:4952] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 4bbb69d62b9a41e3b88a5527f7549881 because the tablet was not found. No further retry: Not found: Tablet not found: 4bbb69d62b9a41e3b88a5527f7549881
W20251027 21:07:55.501525 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 28a2e85021394f9d8a9ea524959b52a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28a2e85021394f9d8a9ea524959b52a1 already in progress: creating tablet
W20251027 21:07:55.501722 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet ccaf5c2f8e544842880ba8ba27300ae6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ccaf5c2f8e544842880ba8ba27300ae6 already in progress: creating tablet
I20251027 21:07:55.506333 10358 raft_consensus.cc:493] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:55.506526 10358 raft_consensus.cc:515] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:55.506538 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed5619d295a246379b625f35692203e4" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:55.507884 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "77797eea3e9c4f8099dc9d58f116200b" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:55.511220 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 5054bbc3e27243298b4df6b0a87af8be
I20251027 21:07:55.513031 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet bc9322398d024e629a3c2aae7b6a77be
W20251027 21:07:55.517367 9860 leader_election.cc:343] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:55.517810 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe7b00e9cf984d12817c9ea860391199. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.520212 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae0ee7cd2c1b406f84285f25420588d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.520366 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c8f0cd910b9d4daea287b3b192a81bd2
I20251027 21:07:55.530191 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 674e624257f84c47acff5d61f1fd5b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.534471 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 9ace4188d3494a05a5678381f8d8fd17 (DEFAULT_TABLE table=default.Table_aeaed62246864354b04616e1e4461644 [id=54ab773e30384ae98a0b0539284c85b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.535068 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_ea4ca66f1a144318a185c1324b2f3f61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.535699 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea4ca66f1a144318a185c1324b2f3f61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.539616 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ace4188d3494a05a5678381f8d8fd17. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.540207 10358 leader_election.cc:290] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:55.542752 10032 tablet_bootstrap.cc:492] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:55.543092 10032 ts_tablet_manager.cc:1403] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.530s user 0.010s sys 0.002s
I20251027 21:07:55.544066 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ee02a05b5ae46e7bf5cf9fdd3e9a80f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.544147 9893 tablet_service.cc:1505] Processing CreateTablet for tablet c3f4e692c08244afa2c79e0a075bbcdb (DEFAULT_TABLE table=default.Table_989e3ba3204f41f6b0d7f408eab2f4a7 [id=5525bf5e4c134750bdc1edf8d6b73969]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.544751 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3f4e692c08244afa2c79e0a075bbcdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.544713 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69da799c886841a19002d3c7c5084e73" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:55.546706 9566 leader_election.cc:343] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:55.547159 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7797741574f1487ba7211ddb3bb032dd" } new_table_name: "default.Table_773dc0a1742a4f9194e9b26ea3d07123" modify_external_catalogs: true
W20251027 21:07:55.550580 9860 leader_election.cc:343] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:55.555248 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 874d5974698c448ab79f19812cb0e886 (DEFAULT_TABLE table=default.Table_c90c85540a2142218c304c8da1a2df90 [id=1ed4b100af1d4ccdb13d108ba63cbd44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.555756 9886 tablet_service.cc:1505] Processing CreateTablet for tablet 720cfe66349f4a8896e433d79a2f25aa (DEFAULT_TABLE table=default.Table_235e62a9895345fdb8b0b428777ff0bd [id=7c40c4a6a45b41c9904fa08af8569bac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.556077 9894 tablet_service.cc:1505] Processing CreateTablet for tablet 11627a84ff9b478c87c323617a09570b (DEFAULT_TABLE table=default.Table_7b970942803b42f29feb36d6d845167a [id=20a729dd21c440a882083323ddb2561e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.558808 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 674e624257f84c47acff5d61f1fd5b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.560979 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 9ace4188d3494a05a5678381f8d8fd17 (DEFAULT_TABLE table=default.Table_aeaed62246864354b04616e1e4461644 [id=54ab773e30384ae98a0b0539284c85b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.563928 9900 tablet_service.cc:1505] Processing CreateTablet for tablet 37ae3b6bf590473e8e09092fc5f7df1b (DEFAULT_TABLE table=default.Table_4658a756ae8149cf9e248c65a102bdb6 [id=0a31c72c314e4e5ab0655d9b853c941c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.565665 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 37ae3b6bf590473e8e09092fc5f7df1b (DEFAULT_TABLE table=default.Table_4658a756ae8149cf9e248c65a102bdb6 [id=0a31c72c314e4e5ab0655d9b853c941c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:55.568465 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 842c1a1298234d9ea2719bab29c54ae3 consensus state has no leader
I20251027 21:07:55.568552 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bc5fbdbbea0406a961e1a4df51b8762. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.574055 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ace4188d3494a05a5678381f8d8fd17. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:55.580138 9568 leader_election.cc:343] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 77797eea3e9c4f8099dc9d58f116200b
I20251027 21:07:55.580286 9568 leader_election.cc:304] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:55.580621 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a2b19b07f5ec4668915f71b441eb1a37" } new_table_name: "default.Table_169ebf3b9cd4423aad9f0c459be30992" modify_external_catalogs: true
I20251027 21:07:55.581202 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_fa13b0f459f9443e965078c8867a629c" } new_table_name: "default.Table_bb9c3894cb5d40738ca8be284b1d1e25" modify_external_catalogs: true
I20251027 21:07:55.586726 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_1a2deac4d0444b85b51757688f89b202"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.587073 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fa39697360c46138ef4cd21e8c7a950 consensus state has no leader
W20251027 21:07:55.587249 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a2deac4d0444b85b51757688f89b202 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.597026 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_dac5ed8bed8045668aa636907042bf0a" } new_table_name: "default.Table_707e838734704c27a8dd97d0d58e6a33" modify_external_catalogs: true
W20251027 21:07:55.602527 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0e64d21abef342de86555e876f020787 consensus state has no leader
I20251027 21:07:55.605307 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0812ae8e053c4e6f913221ef573f618f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.605851 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0812ae8e053c4e6f913221ef573f618f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.608218 10358 raft_consensus.cc:2749] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:55.610348 9601 tablet_service.cc:1505] Processing CreateTablet for tablet ae0ee7cd2c1b406f84285f25420588d0 (DEFAULT_TABLE table=default.Table_e2cb137bea0d419b9791305f97bc0efd [id=251aef26011f4b728b1102e4570722b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.613165 10043 ts_tablet_manager.cc:1929] T a9cc64ad85334daa9a09538e24ebd8a5 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:55.613340 10043 log.cc:1199] T a9cc64ad85334daa9a09538e24ebd8a5 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/a9cc64ad85334daa9a09538e24ebd8a5
I20251027 21:07:55.613750 10043 ts_tablet_manager.cc:1950] T a9cc64ad85334daa9a09538e24ebd8a5 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:55.614207 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37ae3b6bf590473e8e09092fc5f7df1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.615312 9594 tablet_service.cc:1505] Processing CreateTablet for tablet c3f4e692c08244afa2c79e0a075bbcdb (DEFAULT_TABLE table=default.Table_989e3ba3204f41f6b0d7f408eab2f4a7 [id=5525bf5e4c134750bdc1edf8d6b73969]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.615447 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7d6df7f8ab8945a894c5984d320667b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:55.615660 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3f4e692c08244afa2c79e0a075bbcdb. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:55.615957 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d6df7f8ab8945a894c5984d320667b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.616554 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 11627a84ff9b478c87c323617a09570b (DEFAULT_TABLE table=default.Table_7b970942803b42f29feb36d6d845167a [id=20a729dd21c440a882083323ddb2561e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.616876 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11627a84ff9b478c87c323617a09570b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.617302 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_09602bd1d278472bb1ed05abe9d1def4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.617724 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09602bd1d278472bb1ed05abe9d1def4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.618326 10043 tablet_replica.cc:333] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:55.618554 10043 raft_consensus.cc:2243] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:55.618697 10043 raft_consensus.cc:2272] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:55.619207 10043 ts_tablet_manager.cc:1916] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:55.620253 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet a9cc64ad85334daa9a09538e24ebd8a5 (table default.Table_03046ffdd6ca47e8aa4cde13574dc119 [id=8a893ef2396748f08f737fddfb76fd65]) successfully deleted
I20251027 21:07:55.621631 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f9595784df434dbb0dd29d9a05511e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.622535 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ae222307a374a91916f560a0fbe8d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.626341 10358 consensus_queue.cc:1048] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:07:55.636157 10032 raft_consensus.cc:359] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:55.637372 9604 tablet_service.cc:1505] Processing CreateTablet for tablet 874d5974698c448ab79f19812cb0e886 (DEFAULT_TABLE table=default.Table_c90c85540a2142218c304c8da1a2df90 [id=1ed4b100af1d4ccdb13d108ba63cbd44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.639825 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 874d5974698c448ab79f19812cb0e886. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.644289 9886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 720cfe66349f4a8896e433d79a2f25aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.646482 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11627a84ff9b478c87c323617a09570b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.653541 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c58f3dadcca44b7d81025d6b5a7be405"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.654240 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c58f3dadcca44b7d81025d6b5a7be405 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.656970 10359 raft_consensus.cc:493] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251027 21:07:55.664819 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8cccd253ae2a46328dc9f767b8e9ee97 consensus state has no leader
I20251027 21:07:55.666477 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37ae3b6bf590473e8e09092fc5f7df1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.666911 9273 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu pid 10108
I20251027 21:07:55.668650 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f5b100cf12504aeaa64ef743bba49f18" } new_table_name: "default.Table_d0d99e11752e4faba5d28e5f7211c121" modify_external_catalogs: true
I20251027 21:07:55.672771 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 720cfe66349f4a8896e433d79a2f25aa (DEFAULT_TABLE table=default.Table_235e62a9895345fdb8b0b428777ff0bd [id=7c40c4a6a45b41c9904fa08af8569bac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.673913 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 4bfa70ded2f54e3fb2ca744c723395e2 (DEFAULT_TABLE table=default.Table_10a096bf3cc34dc89007e7abeb182f61 [id=46e6ec9244504c159ac7b9b2794e3fa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.674242 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bfa70ded2f54e3fb2ca744c723395e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.676455 9603 tablet_service.cc:1505] Processing CreateTablet for tablet 4bfa70ded2f54e3fb2ca744c723395e2 (DEFAULT_TABLE table=default.Table_10a096bf3cc34dc89007e7abeb182f61 [id=46e6ec9244504c159ac7b9b2794e3fa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.678644 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_fc2da109d1934a3fbe14ae51b20e771d" } modify_external_catalogs: true
I20251027 21:07:55.678894 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_fc2da109d1934a3fbe14ae51b20e771d" } modify_external_catalogs: true
I20251027 21:07:55.676936 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_fdf2451e0a2a48f9b902827e2a291e90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.680691 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdf2451e0a2a48f9b902827e2a291e90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.684247 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_85b289c2c1174393803662453ba6ac8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.684780 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85b289c2c1174393803662453ba6ac8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.688171 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7a9e545bc9374c8b986687268e421690"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.688652 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a9e545bc9374c8b986687268e421690 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.690620 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 82366f8bf4154ffd87ef45e4a4d5f54b (DEFAULT_TABLE table=default.Table_f0792045d49041578f5ff6459c99f77e [id=7f83f7c0f9524147bb778a06dd233a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.695159 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c814f5ad5b6f46f2aa8fc2ecf7ce2d73" } modify_external_catalogs: true
I20251027 21:07:55.695339 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c814f5ad5b6f46f2aa8fc2ecf7ce2d73" } modify_external_catalogs: true
I20251027 21:07:55.699388 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82366f8bf4154ffd87ef45e4a4d5f54b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.702823 9612 tablet_service.cc:1505] Processing CreateTablet for tablet 82366f8bf4154ffd87ef45e4a4d5f54b (DEFAULT_TABLE table=default.Table_f0792045d49041578f5ff6459c99f77e [id=7f83f7c0f9524147bb778a06dd233a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.703560 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e6ed9d7a700446edafcd0daa02aa7ef6
I20251027 21:07:55.698657 9890 tablet_service.cc:1505] Processing CreateTablet for tablet c97cba61e08d42a7bdb3f51db0a63ec8 (DEFAULT_TABLE table=default.Table_f20babe6e88047de940c58738a6d786c [id=96a64d67251d42558d9f1d0fb9663e26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.704684 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c97cba61e08d42a7bdb3f51db0a63ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.706853 10029 tablet_bootstrap.cc:654] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:55.706903 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5804b71f15924faf9b6c09ac13d09d3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.707427 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5804b71f15924faf9b6c09ac13d09d3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.708503 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c3b9a461567a48fcb4af514ecdf618b4
I20251027 21:07:55.712138 9614 tablet_service.cc:1505] Processing CreateTablet for tablet c97cba61e08d42a7bdb3f51db0a63ec8 (DEFAULT_TABLE table=default.Table_f20babe6e88047de940c58738a6d786c [id=96a64d67251d42558d9f1d0fb9663e26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.712853 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_97b01c1840c543838263ca4222695f54" } modify_external_catalogs: true
I20251027 21:07:55.713020 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_97b01c1840c543838263ca4222695f54" } modify_external_catalogs: true
W20251027 21:07:55.637923 9924 thread.cc:641] tablet-copy [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.067s sys 0.055s
I20251027 21:07:55.715992 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0eecf2c874144e7594c825f01fcc5770" } modify_external_catalogs: true
I20251027 21:07:55.716183 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0eecf2c874144e7594c825f01fcc5770" } modify_external_catalogs: true
I20251027 21:07:55.717705 9607 tablet_service.cc:1505] Processing CreateTablet for tablet d1f24d21bb954ac9b62ce458c6a22462 (DEFAULT_TABLE table=default.Table_7ae5d7111eac45a7b3061078d4fc83d8 [id=c89e827a550845a29952bc951b07dca1]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:55.718132 9924 thread.cc:608] tablet-copy [worker] (thread pool) Time spent starting thread: real 0.688s user 0.068s sys 0.055s
I20251027 21:07:55.718274 9613 tablet_service.cc:1505] Processing CreateTablet for tablet 19f3e1a9db434c519e26e864eb790706 (DEFAULT_TABLE table=default.Table_d5398f9a1dca4617be8062e32b583441 [id=704c1f1c3b964a55b95d7858445dd7e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.720916 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae0ee7cd2c1b406f84285f25420588d0. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:55.717747 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3ee02a05b5ae46e7bf5cf9fdd3e9a80f consensus state has no leader
I20251027 21:07:55.722339 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c9efffa9766e46aca8eb286fc2ad48d2
I20251027 21:07:55.725692 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_107ad52a34d44600aa08dbe6983ab081" } modify_external_catalogs: true
I20251027 21:07:55.726039 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_107ad52a34d44600aa08dbe6983ab081" } modify_external_catalogs: true
I20251027 21:07:55.726292 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f3e1a9db434c519e26e864eb790706. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.723342 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet aabaa34787cf4d179822269b7f323314
I20251027 21:07:55.732028 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e10ae34eaaf3476594813de8e96c7e77
I20251027 21:07:55.732743 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c97cba61e08d42a7bdb3f51db0a63ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.734603 9894 tablet_service.cc:1505] Processing CreateTablet for tablet d1f24d21bb954ac9b62ce458c6a22462 (DEFAULT_TABLE table=default.Table_7ae5d7111eac45a7b3061078d4fc83d8 [id=c89e827a550845a29952bc951b07dca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.735064 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f24d21bb954ac9b62ce458c6a22462. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.736649 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 720cfe66349f4a8896e433d79a2f25aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.738065 10032 raft_consensus.cc:385] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:55.740150 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bfa70ded2f54e3fb2ca744c723395e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.742671 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 19f3e1a9db434c519e26e864eb790706 (DEFAULT_TABLE table=default.Table_d5398f9a1dca4617be8062e32b583441 [id=704c1f1c3b964a55b95d7858445dd7e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.742969 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 874d5974698c448ab79f19812cb0e886. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.743124 10032 raft_consensus.cc:740] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:55.744048 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 7828f466a5ad4d7997721b07f815f56d (DEFAULT_TABLE table=default.Table_6a4f975247ee46deac07892f07bf19e9 [id=2cb2ec6a1ab440e582f75386b00ed7e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.744175 9905 tablet_service.cc:1505] Processing CreateTablet for tablet efbabe357ae947e09b1902b1fbc0ee40 (DEFAULT_TABLE table=default.Table_b5b5182f58364f46849e1af83b27c6d8 [id=c85b86cfe6494e919eb1eff998cc3ac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.746270 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82366f8bf4154ffd87ef45e4a4d5f54b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.746796 9886 tablet_service.cc:1505] Processing CreateTablet for tablet da23505b0fe742ddadfb829e37ca5499 (DEFAULT_TABLE table=default.Table_580942c54cc24c14b0849e1f7c209975 [id=a1ac88d6c244410ea3cbcd9a3e8b4827]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.749002 9903 tablet_service.cc:1505] Processing CreateTablet for tablet 849b3e9fc34045939a7a678fa8269d7f (DEFAULT_TABLE table=default.Table_b6905d2ba9e649178b7b4cb0a420a516 [id=84eab55ca4164dd5ba2441e447fb35ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.750782 9595 tablet_service.cc:1505] Processing CreateTablet for tablet 7828f466a5ad4d7997721b07f815f56d (DEFAULT_TABLE table=default.Table_6a4f975247ee46deac07892f07bf19e9 [id=2cb2ec6a1ab440e582f75386b00ed7e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.750941 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 6f901626832f4adb948ccb65f8573e86 (DEFAULT_TABLE table=default.Table_1e71059726554fec9b06fa78154e9ad5 [id=59de715802f547bca16f181917fcc418]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.754320 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f24d21bb954ac9b62ce458c6a22462. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.757088 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f3e1a9db434c519e26e864eb790706. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.757613 9606 tablet_service.cc:1505] Processing CreateTablet for tablet efbabe357ae947e09b1902b1fbc0ee40 (DEFAULT_TABLE table=default.Table_b5b5182f58364f46849e1af83b27c6d8 [id=c85b86cfe6494e919eb1eff998cc3ac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.754710 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 5054bbc3e27243298b4df6b0a87af8be (DEFAULT_TABLE table=default.Table_61bbf6b5c1e740128b80465ef3b9a50e [id=439f48256cbd43f785f57857e83ae25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.758339 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7828f466a5ad4d7997721b07f815f56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.758942 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efbabe357ae947e09b1902b1fbc0ee40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.761014 10029 tablet_bootstrap.cc:492] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:55.761240 10029 ts_tablet_manager.cc:1403] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.547s user 0.003s sys 0.007s
I20251027 21:07:55.762005 10029 raft_consensus.cc:359] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:55.762342 10029 raft_consensus.cc:385] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:55.762601 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 3e46c62b8a1144998eb34ac839563a9d (DEFAULT_TABLE table=default.Table_d00ba224ec354b72a605f17e11591b78 [id=f45509490f4540c698accc119d2ddb27]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.763209 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 849b3e9fc34045939a7a678fa8269d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.764024 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f901626832f4adb948ccb65f8573e86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.764761 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e46c62b8a1144998eb34ac839563a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.766335 9886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da23505b0fe742ddadfb829e37ca5499. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.770008 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c4cd97ce361b428799f703eec191abfc" } new_table_name: "default.Table_89e2320f599341e882a2d09c0985b220" modify_external_catalogs: true
I20251027 21:07:55.771024 10029 raft_consensus.cc:740] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:55.773701 10359 raft_consensus.cc:515] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:55.774413 10359 leader_election.cc:290] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:55.774406 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 5054bbc3e27243298b4df6b0a87af8be (DEFAULT_TABLE table=default.Table_61bbf6b5c1e740128b80465ef3b9a50e [id=439f48256cbd43f785f57857e83ae25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.774740 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5054bbc3e27243298b4df6b0a87af8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.778549 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet 77797eea3e9c4f8099dc9d58f116200b with delete_type TABLET_DATA_DELETED (replaced by tablet 631ff72613734913818a3cad24f20dbc) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:55.778977 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 849b3e9fc34045939a7a678fa8269d7f (DEFAULT_TABLE table=default.Table_b6905d2ba9e649178b7b4cb0a420a516 [id=84eab55ca4164dd5ba2441e447fb35ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.782755 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet 874d5974698c448ab79f19812cb0e886 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:55.782814 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4aae3c9852244881bf5286450425cf5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:55.782972 10032 consensus_queue.cc:260] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:55.783541 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4aae3c9852244881bf5286450425cf5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.787954 10032 ts_tablet_manager.cc:1434] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.244s user 0.001s sys 0.001s
W20251027 21:07:55.791043 9710 leader_election.cc:343] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 1e5f5b57f5d948f3b2352451ccaacb09
I20251027 21:07:55.792075 10032 tablet_bootstrap.cc:492] T e841c54a94d649c19b639f5021f150cd P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:55.794781 10032 tablet_bootstrap.cc:654] T e841c54a94d649c19b639f5021f150cd P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:55.794940 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2532bf7190064443a898e47e843fd9ac consensus state has no leader
I20251027 21:07:55.796476 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b78874d9b16b4c76bdff139266ed9f32" } new_table_name: "default.Table_cb55ac1fcbad4075b84ec473bf89aa2b" modify_external_catalogs: true
I20251027 21:07:55.797268 9888 tablet_service.cc:1505] Processing CreateTablet for tablet ed66f9eeeb2b4b4bb77062c241d2140f (DEFAULT_TABLE table=default.Table_79a3420c8b6542868231c740293ab1fb [id=f3babbbf131543d2813ed9d5882e3d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.797541 10040 ts_tablet_manager.cc:1929] T 601c74459d3a4a41b812fdcdf8b0a62c P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:55.797789 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed66f9eeeb2b4b4bb77062c241d2140f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.799288 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_aa03fc53fae14398b298f577a92d8b5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:55.799324 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 6f901626832f4adb948ccb65f8573e86 (DEFAULT_TABLE table=default.Table_1e71059726554fec9b06fa78154e9ad5 [id=59de715802f547bca16f181917fcc418]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:55.799816 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa03fc53fae14398b298f577a92d8b5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.803809 9604 tablet_service.cc:1505] Processing CreateTablet for tablet 3e46c62b8a1144998eb34ac839563a9d (DEFAULT_TABLE table=default.Table_d00ba224ec354b72a605f17e11591b78 [id=f45509490f4540c698accc119d2ddb27]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.805601 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3d352cd73ac64a89ade7b276936b52cc" } new_table_name: "default.Table_de3472a7a19e4cbdb32862388a574cc9" modify_external_catalogs: true
I20251027 21:07:55.805823 10043 ts_tablet_manager.cc:1929] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:55.805995 10043 log.cc:1199] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/533f231e3ff64531ae2331409a961425
I20251027 21:07:55.806450 10043 ts_tablet_manager.cc:1950] T 533f231e3ff64531ae2331409a961425 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:55.807601 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 533f231e3ff64531ae2331409a961425 (table default.Table_913666abca6044f9aacb48e76118aac5 [id=b45a015e747e4f0db54899a28ab97a7c]) successfully deleted
I20251027 21:07:55.809465 9887 tablet_service.cc:1505] Processing CreateTablet for tablet c25be263facd42c5bb6fa421160fbef0 (DEFAULT_TABLE table=default.Table_9d77e1ff90f047748d72fe89fbcc0c49 [id=e1a7e7b9b73746db840ba0594bca9239]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.809826 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c25be263facd42c5bb6fa421160fbef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.813122 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 64e39cc210664c3d83eaf1fc576f83ce (DEFAULT_TABLE table=default.Table_ef27c46c59a94abfbc41c240b0477362 [id=dc9d6d1f90084f18b22ec006f5161669]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.813721 9595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7828f466a5ad4d7997721b07f815f56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.814375 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5054bbc3e27243298b4df6b0a87af8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.814851 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efbabe357ae947e09b1902b1fbc0ee40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.815313 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 849b3e9fc34045939a7a678fa8269d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.815551 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet 479d2cc8b8ce4c26841f50eea7c0e09a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:55.816247 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 123e6e3748574ea581f67fff49037762 (DEFAULT_TABLE table=default.Table_1a08b666f7f4426eb19757e15584dd53 [id=2537aadc803d44cdbd43de4694661e79]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.817101 9603 tablet_service.cc:1505] Processing CreateTablet for tablet ed66f9eeeb2b4b4bb77062c241d2140f (DEFAULT_TABLE table=default.Table_79a3420c8b6542868231c740293ab1fb [id=f3babbbf131543d2813ed9d5882e3d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.817813 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet 874d5974698c448ab79f19812cb0e886 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:55.817950 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b913b021bfa84a6a8858434231e958fc consensus state has no leader
I20251027 21:07:55.818244 9613 tablet_service.cc:1505] Processing CreateTablet for tablet c25be263facd42c5bb6fa421160fbef0 (DEFAULT_TABLE table=default.Table_9d77e1ff90f047748d72fe89fbcc0c49 [id=e1a7e7b9b73746db840ba0594bca9239]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.818456 9900 tablet_service.cc:1505] Processing CreateTablet for tablet 4d88aacbf47144cb837b99d68f6331e2 (DEFAULT_TABLE table=default.Table_bc5ec745ebe54017b6bc9635581c813d [id=31172964ef544c5f99cade30a42b340d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:55.820142 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 22dbfa50996b47269983c107cc399000 consensus state has no leader
I20251027 21:07:55.820242 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet 631ff72613734913818a3cad24f20dbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:55.820672 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet 37ae3b6bf590473e8e09092fc5f7df1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:55.820690 9608 tablet_service.cc:1505] Processing CreateTablet for tablet 64e39cc210664c3d83eaf1fc576f83ce (DEFAULT_TABLE table=default.Table_ef27c46c59a94abfbc41c240b0477362 [id=dc9d6d1f90084f18b22ec006f5161669]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.820968 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2a9fa2fa9615471691f07230a08e1399"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:55.821264 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 4bbb69d62b9a41e3b88a5527f7549881 (DEFAULT_TABLE table=default.Table_ff202033c9ec4fba9cc9525ea6767cac [id=506208e23e3a410fbd15adc68b4c471a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:55.821581 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a9fa2fa9615471691f07230a08e1399 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.824754 9901 tablet_service.cc:1552] Processing DeleteTablet for tablet 92a2393cff8044028c86c8cd2bccd930 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:55.825145 9612 tablet_service.cc:1552] Processing DeleteTablet for tablet 479d2cc8b8ce4c26841f50eea7c0e09a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:55.825529 9901 tablet_service.cc:1505] Processing CreateTablet for tablet 2f641bc4dbed4c0abd0a55b253352507 (DEFAULT_TABLE table=default.Table_12da35feb05c42a79377049e12fe73d6 [id=8a26e744abae4c5eb1d18dd994cd6bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.825529 9612 tablet_service.cc:1505] Processing CreateTablet for tablet 4d88aacbf47144cb837b99d68f6331e2 (DEFAULT_TABLE table=default.Table_bc5ec745ebe54017b6bc9635581c813d [id=31172964ef544c5f99cade30a42b340d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.826314 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f901626832f4adb948ccb65f8573e86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.827313 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 631ff72613734913818a3cad24f20dbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:55.827739 9598 tablet_service.cc:1505] Processing CreateTablet for tablet 123e6e3748574ea581f67fff49037762 (DEFAULT_TABLE table=default.Table_1a08b666f7f4426eb19757e15584dd53 [id=2537aadc803d44cdbd43de4694661e79]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.828856 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet 37ae3b6bf590473e8e09092fc5f7df1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:55.829229 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 2f641bc4dbed4c0abd0a55b253352507 (DEFAULT_TABLE table=default.Table_12da35feb05c42a79377049e12fe73d6 [id=8a26e744abae4c5eb1d18dd994cd6bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.830722 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 4bbb69d62b9a41e3b88a5527f7549881 (DEFAULT_TABLE table=default.Table_ff202033c9ec4fba9cc9525ea6767cac [id=506208e23e3a410fbd15adc68b4c471a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.836030 10358 raft_consensus.cc:493] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:55.836278 10358 raft_consensus.cc:515] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:55.836723 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b230ee0b39fd44e0b9a956f480073963"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:55.837106 10358 leader_election.cc:290] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:55.837556 9923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f7ad06a15c04efbb701634e14904351" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
W20251027 21:07:55.837805 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b230ee0b39fd44e0b9a956f480073963 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.837792 9923 raft_consensus.cc:2468] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:55.838389 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f7ad06a15c04efbb701634e14904351" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:55.838898 9566 leader_election.cc:343] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:07:55.840291 9711 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.002s sys 0.016s
I20251027 21:07:55.840541 9568 leader_election.cc:304] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:55.841090 10358 raft_consensus.cc:2804] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:55.841069 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_851f66cf4b1047879eed5fb768c07156"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:55.841248 10358 raft_consensus.cc:493] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:55.841339 10358 raft_consensus.cc:3060] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:07:55.841722 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_851f66cf4b1047879eed5fb768c07156 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:55.844025 9711 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.602s user 0.002s sys 0.016s
W20251027 21:07:55.844254 9711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 749461us
I20251027 21:07:55.844748 10040 log.cc:1199] T 601c74459d3a4a41b812fdcdf8b0a62c P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/601c74459d3a4a41b812fdcdf8b0a62c
I20251027 21:07:55.845438 10358 raft_consensus.cc:515] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:55.846479 10358 leader_election.cc:290] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:55.846561 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f7ad06a15c04efbb701634e14904351" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:07:55.846993 9566 leader_election.cc:343] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:55.847244 9923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f7ad06a15c04efbb701634e14904351" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:07:55.847446 9923 raft_consensus.cc:3060] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:55.850838 9923 raft_consensus.cc:2468] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:55.851222 9568 leader_election.cc:304] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:55.854293 9902 tablet_service.cc:1505] Processing CreateTablet for tablet c0e2772d66404d74891e0893c49a66e0 (DEFAULT_TABLE table=default.Table_94d4f5b9325a4c27a6305962fe774808 [id=c92fda1282d7419f94b0004ab8b8403d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.855002 10497 raft_consensus.cc:2804] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:55.855374 10497 raft_consensus.cc:697] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:07:55.855742 10497 consensus_queue.cc:237] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:55.856871 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 6c61c360653346fabff0718f0990d100 (DEFAULT_TABLE table=default.Table_6f6b1a85f5f2410f8c36649b6a7b44a9 [id=6a0bd72e2b44485bac23d34cddf8a57e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.857753 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 92a2393cff8044028c86c8cd2bccd930 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:55.858244 9614 tablet_service.cc:1505] Processing CreateTablet for tablet c0e2772d66404d74891e0893c49a66e0 (DEFAULT_TABLE table=default.Table_94d4f5b9325a4c27a6305962fe774808 [id=c92fda1282d7419f94b0004ab8b8403d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.861379 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2cd2c8e57ab742eba0e7840ced528f74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.861999 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cd2c8e57ab742eba0e7840ced528f74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.866212 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c28df5bcb6c0432791ff79f424fcdca2" } new_table_name: "default.Table_4a21d4b2faee414e88ba4533b814cce2" modify_external_catalogs: true
I20251027 21:07:55.874615 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d88aacbf47144cb837b99d68f6331e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.875891 9898 tablet_service.cc:1505] Processing CreateTablet for tablet b83daa33ef0f45c6a78c70001e532ea2 (DEFAULT_TABLE table=default.Table_20d9ef0583ca492a85be3bf2cdf16b5b [id=5ce64cf618c248cba652043b26376cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.876013 9310 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet cb1686fc7c3a4996aa4462884d30c41f (table default.Table_4a21d4b2faee414e88ba4533b814cce2 [id=3f2856bb858142989104b3f1d9e7f510]). Expected version 1 got 0
I20251027 21:07:55.882313 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bbb69d62b9a41e3b88a5527f7549881. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.882864 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64e39cc210664c3d83eaf1fc576f83ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.883882 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 123e6e3748574ea581f67fff49037762. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.884330 9901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f641bc4dbed4c0abd0a55b253352507. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:55.887557 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 479d2cc8b8ce4c26841f50eea7c0e09a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 479d2cc8b8ce4c26841f50eea7c0e09a already in progress: creating tablet
I20251027 21:07:55.887565 9890 tablet_service.cc:1505] Processing CreateTablet for tablet bbf92e29993b46389a69efd9e9a2341f (DEFAULT_TABLE table=default.Table_9a6d7cca704541ca80a3cf047aea2914 [id=fc420109cc2e43ada2c4ca5123fa0998]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.887878 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbf92e29993b46389a69efd9e9a2341f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:55.888013 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 631ff72613734913818a3cad24f20dbc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 631ff72613734913818a3cad24f20dbc already in progress: creating tablet
W20251027 21:07:55.888216 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 37ae3b6bf590473e8e09092fc5f7df1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37ae3b6bf590473e8e09092fc5f7df1b already in progress: creating tablet
W20251027 21:07:55.888480 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 92a2393cff8044028c86c8cd2bccd930 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92a2393cff8044028c86c8cd2bccd930 already in progress: creating tablet
W20251027 21:07:55.888857 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 874d5974698c448ab79f19812cb0e886 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 874d5974698c448ab79f19812cb0e886 already in progress: creating tablet
I20251027 21:07:55.896538 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_cd4849448781434f836c682ba134ce00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.897058 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd4849448781434f836c682ba134ce00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.897274 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 699c6a8dd8d44d729df463b20cdcfdda (DEFAULT_TABLE table=default.Table_fe40346a4d3641159b11c6b9ec566eca [id=6e17bac6ca06444aa464cb19795ed793]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.897702 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 699c6a8dd8d44d729df463b20cdcfdda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.900904 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d5b9049d02b8474aafbf1e27d9f103ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.901546 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5b9049d02b8474aafbf1e27d9f103ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:55.904395 10031 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.658s user 0.008s sys 0.010s
W20251027 21:07:55.904503 10031 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.658s user 0.008s sys 0.010s
I20251027 21:07:55.904919 10031 ts_tablet_manager.cc:1434] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.664s user 0.009s sys 0.010s
W20251027 21:07:55.905042 10031 ts_tablet_manager.cc:1467] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218: Tablet startup took 664ms
W20251027 21:07:55.905125 10031 ts_tablet_manager.cc:1469] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218: Trace:
1027 21:07:48.600800 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:48.600846 (+ 46us) service_pool.cc:225] Handling call
1027 21:07:48.713402 (+112556us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:48.713429 (+ 27us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:48.720892 (+ 7463us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:48.774820 (+ 53928us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:48.776879 (+ 2059us) inbound_call.cc:173] Queueing success response
1027 21:07:54.831803 (+6054924us) ts_tablet_manager.cc:1379] Bootstrapping tablet
1027 21:07:55.240406 (+408603us) ts_tablet_manager.cc:1435] Starting tablet replica
1027 21:07:55.245420 (+ 5014us) tablet_replica.cc:261] Starting instrumentation
1027 21:07:55.245551 (+ 131us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":21231,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"tablet-open.queue_time_us":6054983,"thread_start_us":657811,"threads_started":1}
I20251027 21:07:55.905375 9905 tablet_service.cc:1505] Processing CreateTablet for tablet edbf5f0db6fa40c7b24f1cd20ee80dd4 (DEFAULT_TABLE table=default.Table_728cbf4f936c4df79d77156e2d3ac59c [id=5c378e8c32d34067bfa42a4b05d6ea7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.905722 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edbf5f0db6fa40c7b24f1cd20ee80dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.907073 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_cd4bceadd9e4409c9ed9dc8365e4ed8e" } modify_external_catalogs: true
I20251027 21:07:55.907310 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_cd4bceadd9e4409c9ed9dc8365e4ed8e" } modify_external_catalogs: true
I20251027 21:07:55.909744 10031 tablet_bootstrap.cc:492] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:55.910497 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7929c591af15477aa34ef5952495dfc6" } modify_external_catalogs: true
I20251027 21:07:55.910653 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7929c591af15477aa34ef5952495dfc6" } modify_external_catalogs: true
I20251027 21:07:55.912928 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b83daa33ef0f45c6a78c70001e532ea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.913455 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c61c360653346fabff0718f0990d100. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.874310 10029 consensus_queue.cc:260] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:55.914849 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0e8bd764e80e46a5b665da0e3e2b677e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.915371 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e8bd764e80e46a5b665da0e3e2b677e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.916177 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 699c6a8dd8d44d729df463b20cdcfdda (DEFAULT_TABLE table=default.Table_fe40346a4d3641159b11c6b9ec566eca [id=6e17bac6ca06444aa464cb19795ed793]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.917754 9753 tablet_service.cc:1505] Processing CreateTablet for tablet fb7fdc91156e41de80872b5c941c5d95 (DEFAULT_TABLE table=default.Table_f1103102a7544791874c67fa7cadbbbe [id=08bb0e89b4514dafb2758847b79e64c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.918491 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb7fdc91156e41de80872b5c941c5d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.919937 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_35621e756a6b4ccd8f4fae865ec4766e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:55.920467 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35621e756a6b4ccd8f4fae865ec4766e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:55.874675 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 6c61c360653346fabff0718f0990d100 (DEFAULT_TABLE table=default.Table_6f6b1a85f5f2410f8c36649b6a7b44a9 [id=6a0bd72e2b44485bac23d34cddf8a57e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.925438 10032 tablet_bootstrap.cc:492] T e841c54a94d649c19b639f5021f150cd P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:55.925614 10032 ts_tablet_manager.cc:1403] T e841c54a94d649c19b639f5021f150cd P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.134s user 0.002s sys 0.001s
I20251027 21:07:55.926651 10032 raft_consensus.cc:359] T e841c54a94d649c19b639f5021f150cd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:55.926895 10032 raft_consensus.cc:385] T e841c54a94d649c19b639f5021f150cd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:55.927000 10032 raft_consensus.cc:740] T e841c54a94d649c19b639f5021f150cd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:55.927258 10032 consensus_queue.cc:260] T e841c54a94d649c19b639f5021f150cd P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:55.879468 9599 tablet_service.cc:1505] Processing CreateTablet for tablet b83daa33ef0f45c6a78c70001e532ea2 (DEFAULT_TABLE table=default.Table_20d9ef0583ca492a85be3bf2cdf16b5b [id=5ce64cf618c248cba652043b26376cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.927783 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_32f38256207d4fa79ed533151449afa7" } modify_external_catalogs: true
I20251027 21:07:55.928442 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_32f38256207d4fa79ed533151449afa7" } modify_external_catalogs: true
I20251027 21:07:55.929497 10032 ts_tablet_manager.cc:1434] T e841c54a94d649c19b639f5021f150cd P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251027 21:07:55.929903 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ef27c46c59a94abfbc41c240b0477362" } modify_external_catalogs: true
I20251027 21:07:55.930027 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f20babe6e88047de940c58738a6d786c" } new_table_name: "default.Table_637357195ab4452dbc808d4d6a8d3c6c" modify_external_catalogs: true
I20251027 21:07:55.930066 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ef27c46c59a94abfbc41c240b0477362" } modify_external_catalogs: true
I20251027 21:07:55.933678 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_73afc9d8e0d740d098eb77c4018c3cb2" } modify_external_catalogs: true
I20251027 21:07:55.933868 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_73afc9d8e0d740d098eb77c4018c3cb2" } modify_external_catalogs: true
I20251027 21:07:55.933905 9896 tablet_service.cc:1505] Processing CreateTablet for tablet e8fe63ea8aba4614900ac254189464de (DEFAULT_TABLE table=default.Table_64efdeb42d0f4db68c0e13066cd8d38f [id=439b38e1dcd84d7d9fd40dd9cbe3a429]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:55.938210 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c97cba61e08d42a7bdb3f51db0a63ec8 consensus state has no leader
I20251027 21:07:55.938396 9899 tablet_service.cc:1505] Processing CreateTablet for tablet bc9322398d024e629a3c2aae7b6a77be (DEFAULT_TABLE table=default.Table_858fba8378424b5399351d9134fa7e60 [id=4239b383dcb841468abe1c9fbb0b0b70]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:55.941666 9568 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 72168.6us
I20251027 21:07:55.944767 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet d89c3c588a1747d7875ebf45924fb890 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:55.944871 9601 tablet_service.cc:1505] Processing CreateTablet for tablet e8fe63ea8aba4614900ac254189464de (DEFAULT_TABLE table=default.Table_64efdeb42d0f4db68c0e13066cd8d38f [id=439b38e1dcd84d7d9fd40dd9cbe3a429]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.946420 9611 tablet_service.cc:1505] Processing CreateTablet for tablet bc9322398d024e629a3c2aae7b6a77be (DEFAULT_TABLE table=default.Table_858fba8378424b5399351d9134fa7e60 [id=4239b383dcb841468abe1c9fbb0b0b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.946439 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 7414744bd9bd40cebe126925af9f7a31 (DEFAULT_TABLE table=default.Table_3e21b34c57774bd2843bbac07d8892ac [id=76c0dad4c78644fc9107e3c8caed66a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.953987 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 61ba8d7df6fd45abb6f5231ee8c4ec38 (DEFAULT_TABLE table=default.Table_3f462ba7e3ab4705a1daa311661c4f3d [id=39097e9cb16c449682ec4fa71836e4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.954377 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61ba8d7df6fd45abb6f5231ee8c4ec38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.958065 10040 ts_tablet_manager.cc:1950] T 601c74459d3a4a41b812fdcdf8b0a62c P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:55.960072 9742 tablet_service.cc:1505] Processing CreateTablet for tablet e477ca4b69c84327b04618d90406e19a (DEFAULT_TABLE table=default.Table_7392db44fc434fd1a795922799aaf639 [id=2b5a90a29b994f1bad6491e84865172d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.960722 9741 tablet_service.cc:1505] Processing CreateTablet for tablet 815e12edcd3847ff8485ba768291a824 (DEFAULT_TABLE table=default.Table_2992b8e207024e06957ce7befb657acb [id=decffeb1481e4d798cffd8c9883be59b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.961781 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 618f1657a10947fd98976820564c20b3 (DEFAULT_TABLE table=default.Table_7929c591af15477aa34ef5952495dfc6 [id=822f0869aaa946c69c92b5b27d715621]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.962550 9747 tablet_service.cc:1505] Processing CreateTablet for tablet d0c53b7b32c84a51a967e2a6a22258c6 (DEFAULT_TABLE table=default.Table_6845351c1f1949a9932e39a805cc4033 [id=d842fe6c9377442dbd25a4c103767aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.967831 9753 tablet_service.cc:1505] Processing CreateTablet for tablet e1385beb1dc14ee0b42a2f4e1f66f532 (DEFAULT_TABLE table=default.Table_92242f48713b48e5969d30e05bed6686 [id=3867dc60b7dd4012b55fc6e187d76b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.973176 9743 tablet_service.cc:1505] Processing CreateTablet for tablet a7d8d060916941a5b7e944e8b08a2882 (DEFAULT_TABLE table=default.Table_b547b70a001f404fb96dc4f8fa6d70e9 [id=e1f4a77dfc0a497080c3c0ad4fe2e370]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.973601 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e477ca4b69c84327b04618d90406e19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.974298 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0e2772d66404d74891e0893c49a66e0. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:55.975874 9566 consensus_peers.cc:597] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:07:55.978228 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 93bebcb636ec466b81453d90aa1bbd0f (DEFAULT_TABLE table=default.Table_2d1949f2a17c47748742fda3abdaaada [id=cd680fbc181a4836bd27a9d6338625f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.978689 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 325) took 3258 ms. Trace:
I20251027 21:07:55.978971 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 532ea47b15b544dfb05ddb2bdacf3953 (DEFAULT_TABLE table=default.Table_c9a81e9dabad4a7d9fbd427439a7587c [id=40ce2a0befe54d698759beff3bfdf0f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.979673 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 93a31187eca14b8797737936a80e7292 (DEFAULT_TABLE table=default.Table_b126a056ef2c41b28a178327ef69e524 [id=69a8a630d7ab4b2d8c7b580b41ebe3ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.980249 9744 tablet_service.cc:1505] Processing CreateTablet for tablet cc4635c00e9b40c49ce04debc76a04bf (DEFAULT_TABLE table=default.Table_3c1a4867db0849ceb59d5900eb57eb73 [id=1af04a7db72b4f939ffae873e6e3e688]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.980515 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7d8d060916941a5b7e944e8b08a2882. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.982939 10040 rpcz_store.cc:276] 1027 21:07:52.699935 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.703159 (+ 3224us) service_pool.cc:225] Handling call
1027 21:07:54.966241 (+2263082us) ts_tablet_manager.cc:1151] Deleting tablet 601c74459d3a4a41b812fdcdf8b0a62c
1027 21:07:54.966250 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:55.465955 (+499705us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:55.788932 (+322977us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:55.958394 (+169462us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2262867}
I20251027 21:07:55.987886 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc4635c00e9b40c49ce04debc76a04bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.988638 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93a31187eca14b8797737936a80e7292. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.994514 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93bebcb636ec466b81453d90aa1bbd0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.995178 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 532ea47b15b544dfb05ddb2bdacf3953. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.996444 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_12da35feb05c42a79377049e12fe73d6" } new_table_name: "default.Table_5cb117352e754df2a40d8e5cfed465b5" modify_external_catalogs: true
I20251027 21:07:55.996608 10032 tablet_bootstrap.cc:492] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:55.998409 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0c53b7b32c84a51a967e2a6a22258c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.998487 10032 tablet_bootstrap.cc:654] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:56.000145 9894 tablet_service.cc:1505] Processing CreateTablet for tablet 953b137f14a44955ae41ab503d04b43c (DEFAULT_TABLE table=default.Table_98ca10ad3609477aaf22dc563a68742e [id=8090dd5b916d49719ded3b0209a80b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.000569 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 953b137f14a44955ae41ab503d04b43c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.000917 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 815e12edcd3847ff8485ba768291a824. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.002455 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 618f1657a10947fd98976820564c20b3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:56.003590 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f641bc4dbed4c0abd0a55b253352507 consensus state has no leader
I20251027 21:07:56.008935 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 338) took 3004 ms. Trace:
I20251027 21:07:56.009367 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2ef7853cbd8b4f59a59cec83c91857c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:56.012442 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 371aa8a57aee466ab01652d6061db4c2
W20251027 21:07:56.013901 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ef7853cbd8b4f59a59cec83c91857c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.016301 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8fe63ea8aba4614900ac254189464de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.016978 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_919a2713d2c54825aa74493e188181e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:56.017711 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 4d2c40f983e743b0b53f987b6a7ba437
I20251027 21:07:56.022562 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_318a68ed05144ed9aeb2db53570ab9c3" } new_table_name: "default.Table_526d1cbb28784221b43a6719685c5e2a" modify_external_catalogs: true
W20251027 21:07:56.023078 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_919a2713d2c54825aa74493e188181e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.021128 10040 rpcz_store.cc:276] 1027 21:07:52.985278 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.985534 (+ 256us) service_pool.cc:225] Handling call
1027 21:07:55.984022 (+2998488us) ts_tablet_manager.cc:1151] Deleting tablet 2a4aa1fe05114c1287ca580c1f411d3f
1027 21:07:55.984031 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:55.989778 (+ 5747us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2998270}
I20251027 21:07:56.021382 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 618f1657a10947fd98976820564c20b3
I20251027 21:07:56.024210 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc9322398d024e629a3c2aae7b6a77be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.027609 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1385beb1dc14ee0b42a2f4e1f66f532. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.032051 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 4c023428437f45b4bae1374bd3dd36a7 (DEFAULT_TABLE table=default.Table_7453f892d0514227b015e34cf2461acd [id=f79bde1a71234b53b5a960c983481cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.032439 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c023428437f45b4bae1374bd3dd36a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.034374 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_09f7ae54c9544043afb9f2baef3ab51d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.034894 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09f7ae54c9544043afb9f2baef3ab51d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.036731 9901 tablet_service.cc:1552] Processing DeleteTablet for tablet 95f9595784df434dbb0dd29d9a05511e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:56.037307 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 95f9595784df434dbb0dd29d9a05511e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95f9595784df434dbb0dd29d9a05511e already in progress: creating tablet
I20251027 21:07:56.037320 9901 tablet_service.cc:1552] Processing DeleteTablet for tablet 11627a84ff9b478c87c323617a09570b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:56.038002 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 11627a84ff9b478c87c323617a09570b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11627a84ff9b478c87c323617a09570b already in progress: creating tablet
I20251027 21:07:56.038260 9901 tablet_service.cc:1505] Processing CreateTablet for tablet b913b021bfa84a6a8858434231e958fc (DEFAULT_TABLE table=default.Table_3d352cd73ac64a89ade7b276936b52cc [id=3f5ef05eb0bf4cf5a11ee0d405c0b23e]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:56.039199 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8328318a4f14b38a4ce36d183e99dfb consensus state has no leader
I20251027 21:07:56.045984 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 337) took 3047 ms. Trace:
I20251027 21:07:56.047752 10040 rpcz_store.cc:276] 1027 21:07:52.985153 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.985732 (+ 579us) service_pool.cc:225] Handling call
1027 21:07:56.023489 (+3037757us) ts_tablet_manager.cc:1151] Deleting tablet 94f2dbf47a5c4458bee54b26f272156e
1027 21:07:56.023498 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.033083 (+ 9585us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3037588}
I20251027 21:07:56.054600 10029 ts_tablet_manager.cc:1434] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.293s user 0.004s sys 0.000s
I20251027 21:07:56.058398 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 336) took 3063 ms. Trace:
I20251027 21:07:56.058506 10040 rpcz_store.cc:276] 1027 21:07:52.985009 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.985911 (+ 902us) service_pool.cc:225] Handling call
1027 21:07:56.047987 (+3062076us) ts_tablet_manager.cc:1151] Deleting tablet 30ce703650c94c06b93bce976cb5d1fd
1027 21:07:56.047994 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.048074 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3061938}
I20251027 21:07:56.058888 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 339) took 3069 ms. Trace:
I20251027 21:07:56.058998 10040 rpcz_store.cc:276] 1027 21:07:52.989339 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.989731 (+ 392us) service_pool.cc:225] Handling call
1027 21:07:56.058737 (+3069006us) ts_tablet_manager.cc:1151] Deleting tablet 30ce703650c94c06b93bce976cb5d1fd
1027 21:07:56.058747 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.058838 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3068822}
I20251027 21:07:56.059880 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 335) took 3098 ms. Trace:
I20251027 21:07:56.059990 10040 rpcz_store.cc:276] 1027 21:07:52.960395 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:52.973019 (+ 12624us) service_pool.cc:225] Handling call
1027 21:07:56.059209 (+3086190us) ts_tablet_manager.cc:1151] Deleting tablet 24ec2e0ba1994ab59c53977583bd2058
1027 21:07:56.059216 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.059298 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3052350}
I20251027 21:07:56.060325 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 342) took 3051 ms. Trace:
I20251027 21:07:56.060417 10040 rpcz_store.cc:276] 1027 21:07:53.008655 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.008716 (+ 61us) service_pool.cc:225] Handling call
1027 21:07:56.060186 (+3051470us) ts_tablet_manager.cc:1151] Deleting tablet f34512059f0c47fc9af7db2645f86b40
1027 21:07:56.060193 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.060280 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3050408}
I20251027 21:07:56.060770 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 343) took 3051 ms. Trace:
I20251027 21:07:56.060864 10040 rpcz_store.cc:276] 1027 21:07:53.008801 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.008843 (+ 42us) service_pool.cc:225] Handling call
1027 21:07:56.060622 (+3051779us) ts_tablet_manager.cc:1151] Deleting tablet f6789dc2f7cb4a3c826ac507d5c9a239
1027 21:07:56.060629 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.060727 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3050497}
I20251027 21:07:56.061192 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 341) took 3052 ms. Trace:
I20251027 21:07:56.061278 10040 rpcz_store.cc:276] 1027 21:07:53.008447 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.008869 (+ 422us) service_pool.cc:225] Handling call
1027 21:07:56.061060 (+3052191us) ts_tablet_manager.cc:1151] Deleting tablet e841c54a94d649c19b639f5021f150cd
1027 21:07:56.061067 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.061145 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3048509}
I20251027 21:07:56.061654 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 348) took 2953 ms. Trace:
I20251027 21:07:56.061748 10040 rpcz_store.cc:276] 1027 21:07:53.108511 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.108851 (+ 340us) service_pool.cc:225] Handling call
1027 21:07:56.061488 (+2952637us) ts_tablet_manager.cc:1151] Deleting tablet 32f80614e30d41128f9fb2e0ea165223
1027 21:07:56.061513 (+ 25us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.061608 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2952416}
I20251027 21:07:56.062032 10040 tablet_replica.cc:333] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:56.062669 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2cd2c8e57ab742eba0e7840ced528f74" } new_table_name: "default.Table_45436efc9fd041c0a813930bd895b3dd" modify_external_catalogs: true
I20251027 21:07:56.063501 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bbb69d62b9a41e3b88a5527f7549881 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.063958 9740 tablet_service.cc:1505] Processing CreateTablet for tablet c27c699cc2744f40811f7b9395ca76be (DEFAULT_TABLE table=default.Table_691240a058cc4f879e5737f73a4c12e7 [id=697732b24e12465d8caac4e23de633bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.892823 9595 tablet_service.cc:1505] Processing CreateTablet for tablet bbf92e29993b46389a69efd9e9a2341f (DEFAULT_TABLE table=default.Table_9a6d7cca704541ca80a3cf047aea2914 [id=fc420109cc2e43ada2c4ca5123fa0998]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.064401 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c27c699cc2744f40811f7b9395ca76be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.903667 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c25be263facd42c5bb6fa421160fbef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:55.768682 9594 tablet_service.cc:1505] Processing CreateTablet for tablet da23505b0fe742ddadfb829e37ca5499 (DEFAULT_TABLE table=default.Table_580942c54cc24c14b0849e1f7c209975 [id=a1ac88d6c244410ea3cbcd9a3e8b4827]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:55.912513 10043 tablet_replica.cc:333] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:56.067508 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet ccaf5c2f8e544842880ba8ba27300ae6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.068048 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 479d2cc8b8ce4c26841f50eea7c0e09a (DEFAULT_TABLE table=default.Table_9601abc37cc5482aa4b9a13b818e9074 [id=9a181744a9954b8388e6417fa7f8a630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.068437 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 479d2cc8b8ce4c26841f50eea7c0e09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.069926 10043 raft_consensus.cc:2243] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:56.070128 10043 raft_consensus.cc:2272] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:56.071292 10043 ts_tablet_manager.cc:1916] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:56.071635 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 0820648719a34f379507efced9cc434b (DEFAULT_TABLE table=default.Table_8291d411936d4f4e8d267fdc28efc533 [id=b0301c76f48f461c8ee32b925c91234b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.072067 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0820648719a34f379507efced9cc434b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.075426 10029 tablet_bootstrap.cc:492] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:56.076011 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet 28a2e85021394f9d8a9ea524959b52a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.076467 9741 tablet_service.cc:1505] Processing CreateTablet for tablet e9b8f76c33444bcb9b488af65e4d3e6b (DEFAULT_TABLE table=default.Table_3d031fe204804a95b5bc07e8b54573d8 [id=855df1b953454ddeac8efc6e5869db2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.076844 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9b8f76c33444bcb9b488af65e4d3e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.077821 9606 tablet_service.cc:1505] Processing CreateTablet for tablet edbf5f0db6fa40c7b24f1cd20ee80dd4 (DEFAULT_TABLE table=default.Table_728cbf4f936c4df79d77156e2d3ac59c [id=5c378e8c32d34067bfa42a4b05d6ea7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.078308 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da23505b0fe742ddadfb829e37ca5499. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.081975 9595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbf92e29993b46389a69efd9e9a2341f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.082756 9747 tablet_service.cc:1505] Processing CreateTablet for tablet 4bc5fbdbbea0406a961e1a4df51b8762 (DEFAULT_TABLE table=default.Table_cc31649fd9f544e985ed00eb483164ea [id=4154657a736742b8a4f7dfbd254209c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.077057 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0c6076a969244f39ac709114449efb98" } new_table_name: "default.Table_5d4d055be5ca4beb9199ff43126de81c" modify_external_catalogs: true
I20251027 21:07:56.083256 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0e2772d66404d74891e0893c49a66e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.077118 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b64e1c03b47345e2adc8d87940f6280e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.084789 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b64e1c03b47345e2adc8d87940f6280e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.078310 9303 catalog_manager.cc:5649] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } }
I20251027 21:07:56.079545 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet ed5619d295a246379b625f35692203e4
I20251027 21:07:56.079772 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 64e39cc210664c3d83eaf1fc576f83ce
I20251027 21:07:56.086166 9303 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 7f7ad06a15c04efbb701634e14904351 (table default.Table_318d41ccfb2c4d2c99bafcea875143a1 [id=2bc39b5bc02645ef830eee14e92707db]). Expected version 1 got 0
I20251027 21:07:56.087242 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed66f9eeeb2b4b4bb77062c241d2140f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.090267 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 123e6e3748574ea581f67fff49037762. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:56.090605 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9558d53008640739ac6fb45f1352fc0 consensus state has no leader
I20251027 21:07:56.091830 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_db53bd5823b84458b7cd9564d5e9676e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.092320 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db53bd5823b84458b7cd9564d5e9676e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.093355 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 7414744bd9bd40cebe126925af9f7a31 (DEFAULT_TABLE table=default.Table_3e21b34c57774bd2843bbac07d8892ac [id=76c0dad4c78644fc9107e3c8caed66a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.093797 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7414744bd9bd40cebe126925af9f7a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.082918 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c59fd341300a4f2f90f1bbd06750d943"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:56.094616 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e46c62b8a1144998eb34ac839563a9d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:56.094623 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c59fd341300a4f2f90f1bbd06750d943 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.039364 9900 tablet_service.cc:1505] Processing CreateTablet for tablet 191009830c30467b9c5ab39e13b7c267 (DEFAULT_TABLE table=default.Table_f846038ed8334d998a32d5775f2b0403 [id=40835fcfaf9b47bc9cea87f9b8785652]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.039851 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7414744bd9bd40cebe126925af9f7a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.040340 9886 tablet_service.cc:1505] Processing CreateTablet for tablet 1329ff0f8ebe4da8a6462a574ede98d4 (DEFAULT_TABLE table=default.Table_24e18a6f669742d696b1929a9342ac9f [id=973c679f79994603b7335f87e4f815ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.040741 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet 5054bbc3e27243298b4df6b0a87af8be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.041009 9903 tablet_service.cc:1552] Processing DeleteTablet for tablet bc9322398d024e629a3c2aae7b6a77be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.041625 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet c8f0cd910b9d4daea287b3b192a81bd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.044003 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet e6ed9d7a700446edafcd0daa02aa7ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.050292 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet c3b9a461567a48fcb4af514ecdf618b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.050757 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet c9efffa9766e46aca8eb286fc2ad48d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.051146 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet aabaa34787cf4d179822269b7f323314 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:56.100088 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 5054bbc3e27243298b4df6b0a87af8be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5054bbc3e27243298b4df6b0a87af8be already in progress: creating tablet
I20251027 21:07:56.104189 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bc5fbdbbea0406a961e1a4df51b8762. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:56.104463 9296 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 345d3795a37b4a40b40967cfacf0f893 (table default.Table_b2fe93c466204947aa406fbece12b682 [id=93d6307da3a34f87a67018a06825cd15], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 345d3795a37b4a40b40967cfacf0f893 consensus state has no leader
I20251027 21:07:56.110234 9743 tablet_service.cc:1505] Processing CreateTablet for tablet 67994a7cacd94b2ebb16fe154832c8fd (DEFAULT_TABLE table=default.Table_a328d75fa0f44ad9b92e05cd96556f06 [id=3a4883bc82ba48099b2e20589d7eef61]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:56.111215 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet c8f0cd910b9d4daea287b3b192a81bd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8f0cd910b9d4daea287b3b192a81bd2 already in progress: creating tablet
I20251027 21:07:56.111901 10031 tablet_bootstrap.cc:654] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:56.112164 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67994a7cacd94b2ebb16fe154832c8fd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:56.114064 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet e6ed9d7a700446edafcd0daa02aa7ef6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6ed9d7a700446edafcd0daa02aa7ef6 already in progress: creating tablet
W20251027 21:07:56.114635 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet c3b9a461567a48fcb4af514ecdf618b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3b9a461567a48fcb4af514ecdf618b4 already in progress: creating tablet
W20251027 21:07:56.116730 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet c9efffa9766e46aca8eb286fc2ad48d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9efffa9766e46aca8eb286fc2ad48d2 already in progress: creating tablet
W20251027 21:07:56.116932 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet aabaa34787cf4d179822269b7f323314 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aabaa34787cf4d179822269b7f323314 already in progress: creating tablet
I20251027 21:07:56.117058 9744 tablet_service.cc:1505] Processing CreateTablet for tablet f19b5e8e460a4c4ba7399b981c603b84 (DEFAULT_TABLE table=default.Table_7d20b20b1f274f4f94a9497e2d030235 [id=ab4e770ed197499ba9751f3061b2aeba]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:56.117705 9299 catalog_manager.cc:4952] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet bc9322398d024e629a3c2aae7b6a77be because the tablet was not found. No further retry: Not found: Tablet not found: bc9322398d024e629a3c2aae7b6a77be
I20251027 21:07:56.118640 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f19b5e8e460a4c4ba7399b981c603b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.119313 9903 tablet_service.cc:1552] Processing DeleteTablet for tablet 64e39cc210664c3d83eaf1fc576f83ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.119513 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d2c40f983e743b0b53f987b6a7ba437 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.119656 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 371aa8a57aee466ab01652d6061db4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.120204 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 191009830c30467b9c5ab39e13b7c267. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.121601 10411 raft_consensus.cc:493] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:56.121776 10411 raft_consensus.cc:515] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:56.124718 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 95f9595784df434dbb0dd29d9a05511e (DEFAULT_TABLE table=default.Table_3ed917cd5e964f3b8f54abfcf705aba7 [id=9a42e4bcf97c4d45a5f6ec342844a805]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.125687 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f9595784df434dbb0dd29d9a05511e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:56.126036 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 64e39cc210664c3d83eaf1fc576f83ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64e39cc210664c3d83eaf1fc576f83ce already in progress: creating tablet
W20251027 21:07:56.126351 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 4d2c40f983e743b0b53f987b6a7ba437 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d2c40f983e743b0b53f987b6a7ba437 already in progress: creating tablet
I20251027 21:07:56.126420 10411 leader_election.cc:290] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
W20251027 21:07:56.126593 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 371aa8a57aee466ab01652d6061db4c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 371aa8a57aee466ab01652d6061db4c2 already in progress: creating tablet
I20251027 21:07:56.127312 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_8b0df27ef0ab4ed6afe3765a736f8f20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:56.127442 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet e0968c2f25b64f6685f38a999ee9d60e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:56.127846 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b0df27ef0ab4ed6afe3765a736f8f20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:56.127928 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 87612b8bd8a04fd9b32f2df3f9e0d997 has no consensus state
I20251027 21:07:56.127861 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 618f1657a10947fd98976820564c20b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.130235 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 113a5d30d4b3490187681383d4fef325 (DEFAULT_TABLE table=default.Table_1a51e839945b4912a5519bb1f21a6b75 [id=740df97b965b4a8facf0e11e2c38f020]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.130419 10041 tablet_replica.cc:333] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:56.131013 9897 tablet_service.cc:1505] Processing CreateTablet for tablet a8c46c2a01f946aea8b85875115f7f37 (DEFAULT_TABLE table=default.Table_9b0fd27b43c842008090999fe7da9233 [id=62434317788348feb003dfac99359cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.131871 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet e10ae34eaaf3476594813de8e96c7e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.131973 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 7c6c2292ad0e4066958897fff01370af (DEFAULT_TABLE table=default.Table_7a535e8483da496498c4c7f7e76d6670 [id=53f813baffa448c990d467d8267929ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.132151 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e58c8bb0dfc540ba81698903abdfdbb0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:56.132490 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c6c2292ad0e4066958897fff01370af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.132485 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet e841c54a94d649c19b639f5021f150cd with delete_type TABLET_DATA_DELETED (replaced by tablet 666842b8e54c4561989fd9401210464b) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:56.132612 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e58c8bb0dfc540ba81698903abdfdbb0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.133540 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet ed5619d295a246379b625f35692203e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.134105 10041 raft_consensus.cc:2243] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:56.134351 10041 raft_consensus.cc:2272] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:56.136277 9886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1329ff0f8ebe4da8a6462a574ede98d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.137894 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 113a5d30d4b3490187681383d4fef325. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.137996 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 72a6ca830c904373a4cca1169ae391ce (DEFAULT_TABLE table=default.Table_011f55ad6c164cdd8753c07e2359334c [id=6a6d9881d4b4433aa97e2eec32010e9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.138432 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a6ca830c904373a4cca1169ae391ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.121910 9742 tablet_service.cc:1505] Processing CreateTablet for tablet e3654aa289154b3ea613b6ff94cbd57e (DEFAULT_TABLE table=default.Table_c8cb97c3834e406ca7d8a2d047d7d249 [id=4c67737c2c734ad0ad33349fa69668bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.142071 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3654aa289154b3ea613b6ff94cbd57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.142724 9306 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet cb1686fc7c3a4996aa4462884d30c41f (table default.Table_4a21d4b2faee414e88ba4533b814cce2 [id=3f2856bb858142989104b3f1d9e7f510]). Expected version 1 got 0
I20251027 21:07:56.143810 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b2fe93c466204947aa406fbece12b682" } new_table_name: "default.Table_ef79e4d09fb741429811f81e1efd90cb" modify_external_catalogs: true
I20251027 21:07:56.144381 9739 tablet_service.cc:1505] Processing CreateTablet for tablet 23673513189f47638dbbc737da0991d1 (DEFAULT_TABLE table=default.Table_bae8db615e424af398f1f9baec11cb76 [id=cc1da530aa95455e90889dd5ea3768c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.144841 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23673513189f47638dbbc737da0991d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.148195 10040 raft_consensus.cc:2243] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:07:56.151278 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_cff1f8d7488047e3b7f4608d376e6f5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.151995 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cff1f8d7488047e3b7f4608d376e6f5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.154480 10032 tablet_bootstrap.cc:492] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:56.154817 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8fe63ea8aba4614900ac254189464de. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:56.156340 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 345d3795a37b4a40b40967cfacf0f893 consensus state has no leader
I20251027 21:07:56.156366 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0968c2f25b64f6685f38a999ee9d60e" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:56.157292 10041 ts_tablet_manager.cc:1916] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:56.157548 9860 leader_election.cc:343] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:56.159865 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c46c2a01f946aea8b85875115f7f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.160691 9736 tablet_service.cc:1505] Processing CreateTablet for tablet c548d2413a5940c8b114ced667b4dc41 (DEFAULT_TABLE table=default.Table_907ada415b694cf4817ef1e6f6f68e9d [id=ec3ae7f22bea4f9d8b7ba9c5c4a3b2c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.161091 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c548d2413a5940c8b114ced667b4dc41. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.163146 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f5602dbcd14b4fb5ae7190b6a0bd65bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.163817 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5602dbcd14b4fb5ae7190b6a0bd65bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.164719 10032 ts_tablet_manager.cc:1403] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.158s user 0.001s sys 0.003s
I20251027 21:07:56.167374 9891 tablet_service.cc:1505] Processing CreateTablet for tablet b99232a6018a4b1a850da954c551d38d (DEFAULT_TABLE table=default.Table_cf9f3cddff8a486fbe76334c2fd833c5 [id=983747413ebc4e3ea8ccb0eac5ffff08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.169896 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 54c2b10e79e149c2a637475e1f518907 (DEFAULT_TABLE table=default.Table_602a5846b60c4860b33575da44b5015b [id=9f1e545ab3174e8a893b9183cf95768d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.171154 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54c2b10e79e149c2a637475e1f518907. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.176612 9751 tablet_service.cc:1505] Processing CreateTablet for tablet c8f0cd910b9d4daea287b3b192a81bd2 (DEFAULT_TABLE table=default.Table_7ed2c827a45d4c80a5416d12258c77c1 [id=f7d6376018f64583981d26e14762cbaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.177016 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8f0cd910b9d4daea287b3b192a81bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.177277 9306 master_service.cc:954] ReplaceTablet: received request to replace tablet 0e4d35f229bc46e1859ded7c8196ab9d from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:56.177907 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 4d2c40f983e743b0b53f987b6a7ba437 (DEFAULT_TABLE table=default.Table_73afc9d8e0d740d098eb77c4018c3cb2 [id=272264f0d3d347f0bd782edcebfec854]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.186465 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 0b4bf2b20e7c416aad123c6f96355cac (DEFAULT_TABLE table=default.Table_3c0c40b875fc45969c1a916c5545c4cb [id=699b0f7383ef48aeb27575efd44e5658]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.186940 9896 tablet_service.cc:1505] Processing CreateTablet for tablet c6a189aa22d048858e9f30bf4276859d (DEFAULT_TABLE table=default.Table_abc7f8be9f414322ad9064340647399f [id=93edbbc1222b4ce990f6dc4e9a8cbe25]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.188450 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 674e624257f84c47acff5d61f1fd5b54 (DEFAULT_TABLE table=default.Table_b0eb812d94f1489892d520a5b4df323c [id=497a00b32737421bbfec0c6205350263]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.192339 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 0e4d35f229bc46e1859ded7c8196ab9d
I20251027 21:07:56.192767 10032 raft_consensus.cc:359] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:56.193141 9306 catalog_manager.cc:6719] ReplaceTablet: tablet 0e4d35f229bc46e1859ded7c8196ab9d deleted and replaced by tablet 280ee23d80d64587b18811ff06340eee
I20251027 21:07:56.193266 10032 raft_consensus.cc:385] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:56.194116 10032 raft_consensus.cc:740] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:56.195957 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 9b9ffa18afa54f2c9ffd7b5a03a3a324 (DEFAULT_TABLE table=default.Table_ccc943e7fbc54e609e33b3a131a07c74 [id=1cc502193c3443898bc7eafa5ed15652]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.206997 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_917a7cf5e19547888798a43ded99f4b8" } new_table_name: "default.Table_df5f4de40f7f46728f38580d23124544" modify_external_catalogs: true
I20251027 21:07:56.207142 9310 master_service.cc:954] ReplaceTablet: received request to replace tablet 9351a369dc1e4ac49f6ad419eab3a4c5 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:56.215839 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7392db44fc434fd1a795922799aaf639" } modify_external_catalogs: true
I20251027 21:07:56.216032 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7392db44fc434fd1a795922799aaf639" } modify_external_catalogs: true
I20251027 21:07:56.217259 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 9351a369dc1e4ac49f6ad419eab3a4c5
I20251027 21:07:56.217721 9310 catalog_manager.cc:6719] ReplaceTablet: tablet 9351a369dc1e4ac49f6ad419eab3a4c5 deleted and replaced by tablet d5f4f3990da34f5e8f0385595f457427
I20251027 21:07:56.218616 9901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b913b021bfa84a6a8858434231e958fc. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:56.221124 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 952583fd7fde4b4baccdb98c01bdf9e1 consensus state has no leader
I20251027 21:07:56.222405 9902 tablet_service.cc:1505] Processing CreateTablet for tablet a6edd6a5c4434f0497667b1a2771f885 (DEFAULT_TABLE table=default.Table_f0e8c68b9f1243bfb3584cca5b5f7002 [id=1d7f0c839ec546178f6cd39b81db07ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.221572 10032 consensus_queue.cc:260] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:56.226090 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e477ca4b69c84327b04618d90406e19a
I20251027 21:07:56.226070 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9a6d7cca704541ca80a3cf047aea2914" } modify_external_catalogs: true
I20251027 21:07:56.226274 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9a6d7cca704541ca80a3cf047aea2914" } modify_external_catalogs: true
I20251027 21:07:56.228221 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_db53bd5823b84458b7cd9564d5e9676e" } modify_external_catalogs: true
I20251027 21:07:56.228385 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_db53bd5823b84458b7cd9564d5e9676e" } modify_external_catalogs: true
I20251027 21:07:56.233700 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 3ee02a05b5ae46e7bf5cf9fdd3e9a80f (DEFAULT_TABLE table=default.Table_f5b100cf12504aeaa64ef743bba49f18 [id=8c32f511b11d422fa461408a4e9d4e0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.234288 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet bbf92e29993b46389a69efd9e9a2341f
I20251027 21:07:56.234309 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b4bf2b20e7c416aad123c6f96355cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.234972 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d2c40f983e743b0b53f987b6a7ba437. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.235507 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a189aa22d048858e9f30bf4276859d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.236527 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 9ace4188d3494a05a5678381f8d8fd17 (DEFAULT_TABLE table=default.Table_aeaed62246864354b04616e1e4461644 [id=54ab773e30384ae98a0b0539284c85b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.236878 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ee02a05b5ae46e7bf5cf9fdd3e9a80f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.238348 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 874d5974698c448ab79f19812cb0e886 (DEFAULT_TABLE table=default.Table_c90c85540a2142218c304c8da1a2df90 [id=1ed4b100af1d4ccdb13d108ba63cbd44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.238760 9739 tablet_service.cc:1505] Processing CreateTablet for tablet fe7b00e9cf984d12817c9ea860391199 (DEFAULT_TABLE table=default.Table_f8b57b86738a4f61b6a63a06b73016dc [id=9192a31ca6324aa2a919311093d54ae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.240542 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_94d4f5b9325a4c27a6305962fe774808" } new_table_name: "default.Table_349a03f3bc284c719a39a8bb95bbb526" modify_external_catalogs: true
I20251027 21:07:56.241402 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_fdca4795c6d042ec80cf84a378acb979"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.241923 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdca4795c6d042ec80cf84a378acb979 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.242060 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ace4188d3494a05a5678381f8d8fd17. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.244196 9749 tablet_service.cc:1505] Processing CreateTablet for tablet ae0ee7cd2c1b406f84285f25420588d0 (DEFAULT_TABLE table=default.Table_e2cb137bea0d419b9791305f97bc0efd [id=251aef26011f4b728b1102e4570722b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.247731 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet f1aa5f7f76cb4a1ba15f016942f17067 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.248137 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_41ae08dcac5e4355b09bd4d15a35c229"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.248637 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41ae08dcac5e4355b09bd4d15a35c229 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.250281 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet bbf92e29993b46389a69efd9e9a2341f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.250463 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet e477ca4b69c84327b04618d90406e19a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.250937 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b99232a6018a4b1a850da954c551d38d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.252239 10032 ts_tablet_manager.cc:1434] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.086s user 0.000s sys 0.002s
I20251027 21:07:56.253324 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 720cfe66349f4a8896e433d79a2f25aa (DEFAULT_TABLE table=default.Table_235e62a9895345fdb8b0b428777ff0bd [id=7c40c4a6a45b41c9904fa08af8569bac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.255424 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 4bfa70ded2f54e3fb2ca744c723395e2 (DEFAULT_TABLE table=default.Table_10a096bf3cc34dc89007e7abeb182f61 [id=46e6ec9244504c159ac7b9b2794e3fa6]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:56.256395 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c0e2772d66404d74891e0893c49a66e0 consensus state has no leader
I20251027 21:07:56.257591 9747 tablet_service.cc:1505] Processing CreateTablet for tablet 82366f8bf4154ffd87ef45e4a4d5f54b (DEFAULT_TABLE table=default.Table_f0792045d49041578f5ff6459c99f77e [id=7f83f7c0f9524147bb778a06dd233a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.259411 10032 tablet_bootstrap.cc:492] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:56.269655 9753 tablet_service.cc:1505] Processing CreateTablet for tablet e04f4c5cf724419ab0be0e84d30a9a57 (DEFAULT_TABLE table=default.Table_5315576a7f87464dbbfded0081ddef2d [id=ca2d675d800a46728d7b11ee50f2e056]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.271096 9751 tablet_service.cc:1505] Processing CreateTablet for tablet 37ae3b6bf590473e8e09092fc5f7df1b (DEFAULT_TABLE table=default.Table_4658a756ae8149cf9e248c65a102bdb6 [id=0a31c72c314e4e5ab0655d9b853c941c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.275810 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 9351a369dc1e4ac49f6ad419eab3a4c5 with delete_type TABLET_DATA_DELETED (replaced by tablet d5f4f3990da34f5e8f0385595f457427) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.276050 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b9ffa18afa54f2c9ffd7b5a03a3a324. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.276784 9736 tablet_service.cc:1505] Processing CreateTablet for tablet c3f4e692c08244afa2c79e0a075bbcdb (DEFAULT_TABLE table=default.Table_989e3ba3204f41f6b0d7f408eab2f4a7 [id=5525bf5e4c134750bdc1edf8d6b73969]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.281791 9304 master_service.cc:954] ReplaceTablet: received request to replace tablet 78884c2be75f45f881f5424d3b3f4e08 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:56.283974 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_fe401505cb114943863339485e5ff2f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.284550 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe401505cb114943863339485e5ff2f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.286128 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e4d35f229bc46e1859ded7c8196ab9d with delete_type TABLET_DATA_DELETED (replaced by tablet 280ee23d80d64587b18811ff06340eee) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.289351 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a4ae12c512d645e58eadc2daa1bb7b0a" } modify_external_catalogs: true
I20251027 21:07:56.289568 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a4ae12c512d645e58eadc2daa1bb7b0a" } modify_external_catalogs: true
I20251027 21:07:56.289595 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4a21d4b2faee414e88ba4533b814cce2" } modify_external_catalogs: true
I20251027 21:07:56.289773 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4a21d4b2faee414e88ba4533b814cce2" } modify_external_catalogs: true
I20251027 21:07:56.293552 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 19f3e1a9db434c519e26e864eb790706 (DEFAULT_TABLE table=default.Table_d5398f9a1dca4617be8062e32b583441 [id=704c1f1c3b964a55b95d7858445dd7e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.294437 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d395c65b39c04ef2a3ce8490e8487f47" } new_table_name: "default.Table_92cfd00027b045f3b151deec3558ece6" modify_external_catalogs: true
I20251027 21:07:56.299971 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d88aacbf47144cb837b99d68f6331e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.301668 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet cb1686fc7c3a4996aa4462884d30c41f
I20251027 21:07:56.303165 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f641bc4dbed4c0abd0a55b253352507. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:56.303973 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7425b6b0184e4d0b871777b25d904ac4 consensus state has no leader
I20251027 21:07:56.304903 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edbf5f0db6fa40c7b24f1cd20ee80dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.305421 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5639561ca4aa45e5bc117ecc76d943d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.305966 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5639561ca4aa45e5bc117ecc76d943d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.306357 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet cb1686fc7c3a4996aa4462884d30c41f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.308267 10029 tablet_bootstrap.cc:654] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:56.309252 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_cc55a31e64194c35a8e42c5cdf11600d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.309798 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc55a31e64194c35a8e42c5cdf11600d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.316111 9595 tablet_service.cc:1505] Processing CreateTablet for tablet b913b021bfa84a6a8858434231e958fc (DEFAULT_TABLE table=default.Table_3d352cd73ac64a89ade7b276936b52cc [id=3f5ef05eb0bf4cf5a11ee0d405c0b23e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.330068 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_052e3f023dfd462884130780e8c57936" } new_table_name: "default.Table_e12fdbe9bbc0492f85e48ceed92def03" modify_external_catalogs: true
I20251027 21:07:56.335116 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_45692264ec9640ecbb170bfb95297d43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.335690 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45692264ec9640ecbb170bfb95297d43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.340524 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e13c97413ddf417ea96834ba933b537b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.341035 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e13c97413ddf417ea96834ba933b537b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:56.347783 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 72c1dc3b9eff47ee81df821deb789307 consensus state has no leader
I20251027 21:07:56.347903 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c9d359a62fe14203ae1d3b96f4b1bbf1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.348369 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9d359a62fe14203ae1d3b96f4b1bbf1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.348730 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b83daa33ef0f45c6a78c70001e532ea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.354118 9595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b913b021bfa84a6a8858434231e958fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.357558 10043 ts_tablet_manager.cc:1929] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:56.357676 10043 log.cc:1199] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/7d8825b80cec4ba68da077ca3b8d89db
I20251027 21:07:56.357702 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 699c6a8dd8d44d729df463b20cdcfdda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.358064 10043 ts_tablet_manager.cc:1950] T 7d8825b80cec4ba68da077ca3b8d89db P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:56.358309 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc9322398d024e629a3c2aae7b6a77be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.361048 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_110dd5476e984ca299e60f37066a8058" } new_table_name: "default.Table_400ccee830744304b52118f36ab774ff" modify_external_catalogs: true
I20251027 21:07:56.361469 9595 tablet_service.cc:1552] Processing DeleteTablet for tablet bc9322398d024e629a3c2aae7b6a77be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.363786 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0f287ac2ad254338bd5151726cc40821"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.364465 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f287ac2ad254338bd5151726cc40821 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.365988 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 674e624257f84c47acff5d61f1fd5b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.367254 9740 tablet_service.cc:1505] Processing CreateTablet for tablet c97cba61e08d42a7bdb3f51db0a63ec8 (DEFAULT_TABLE table=default.Table_f20babe6e88047de940c58738a6d786c [id=96a64d67251d42558d9f1d0fb9663e26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.368471 10040 pending_rounds.cc:70] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
I20251027 21:07:56.368671 10040 pending_rounds.cc:77] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510623307816960 op_type: NO_OP noop_request { }
W20251027 21:07:56.368788 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2752b86151a6428c88bfaac1ff72cfb2 consensus state has no leader
I20251027 21:07:56.374357 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6edd6a5c4434f0497667b1a2771f885. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.376247 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae0ee7cd2c1b406f84285f25420588d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.379781 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e6e6313ef68948c79fc2f7d96afae64b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:56.379961 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 11627a84ff9b478c87c323617a09570b (DEFAULT_TABLE table=default.Table_7b970942803b42f29feb36d6d845167a [id=20a729dd21c440a882083323ddb2561e]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:56.380406 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6e6313ef68948c79fc2f7d96afae64b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.381498 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c97cba61e08d42a7bdb3f51db0a63ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.382448 10040 raft_consensus.cc:2889] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:07:56.384164 10032 tablet_bootstrap.cc:654] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:56.388262 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 849b3e9fc34045939a7a678fa8269d7f (DEFAULT_TABLE table=default.Table_b6905d2ba9e649178b7b4cb0a420a516 [id=84eab55ca4164dd5ba2441e447fb35ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.389961 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11627a84ff9b478c87c323617a09570b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.393584 10040 raft_consensus.cc:2272] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:56.394264 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 720cfe66349f4a8896e433d79a2f25aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.396051 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet c3b9a461567a48fcb4af514ecdf618b4 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:56.397238 10291 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 21:07:56.404230 10041 ts_tablet_manager.cc:1929] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:56.404922 10041 log.cc:1199] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/e0968c2f25b64f6685f38a999ee9d60e
I20251027 21:07:56.405436 10041 ts_tablet_manager.cc:1950] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:56.408432 9737 tablet_service.cc:1505] Processing CreateTablet for tablet d1f24d21bb954ac9b62ce458c6a22462 (DEFAULT_TABLE table=default.Table_7ae5d7111eac45a7b3061078d4fc83d8 [id=c89e827a550845a29952bc951b07dca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.361717 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c61c360653346fabff0718f0990d100. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.407716 9752 tablet_service.cc:1505] Processing CreateTablet for tablet efbabe357ae947e09b1902b1fbc0ee40 (DEFAULT_TABLE table=default.Table_b5b5182f58364f46849e1af83b27c6d8 [id=c85b86cfe6494e919eb1eff998cc3ac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.416101 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37ae3b6bf590473e8e09092fc5f7df1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.414490 9741 tablet_service.cc:1505] Processing CreateTablet for tablet da23505b0fe742ddadfb829e37ca5499 (DEFAULT_TABLE table=default.Table_580942c54cc24c14b0849e1f7c209975 [id=a1ac88d6c244410ea3cbcd9a3e8b4827]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.415827 9744 tablet_service.cc:1505] Processing CreateTablet for tablet 6f901626832f4adb948ccb65f8573e86 (DEFAULT_TABLE table=default.Table_1e71059726554fec9b06fa78154e9ad5 [id=59de715802f547bca16f181917fcc418]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.418828 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a
I20251027 21:07:56.420886 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 7828f466a5ad4d7997721b07f815f56d (DEFAULT_TABLE table=default.Table_6a4f975247ee46deac07892f07bf19e9 [id=2cb2ec6a1ab440e582f75386b00ed7e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.421712 10040 ts_tablet_manager.cc:1916] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:56.421980 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efbabe357ae947e09b1902b1fbc0ee40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.366614 9598 tablet_service.cc:1505] Processing CreateTablet for tablet 4c023428437f45b4bae1374bd3dd36a7 (DEFAULT_TABLE table=default.Table_7453f892d0514227b015e34cf2461acd [id=f79bde1a71234b53b5a960c983481cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.367071 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 953b137f14a44955ae41ab503d04b43c (DEFAULT_TABLE table=default.Table_98ca10ad3609477aaf22dc563a68742e [id=8090dd5b916d49719ded3b0209a80b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.367239 9594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 506) took 1094 ms. Trace:
I20251027 21:07:56.367405 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 95f9595784df434dbb0dd29d9a05511e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.423493 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f343a39b880b4c7a827bff89fd3fe411" } new_table_name: "default.Table_162a38d7fe14491d8dae6d2e097d3e3a" modify_external_catalogs: true
I20251027 21:07:56.367583 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 11627a84ff9b478c87c323617a09570b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.369413 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 191009830c30467b9c5ab39e13b7c267 (DEFAULT_TABLE table=default.Table_f846038ed8334d998a32d5775f2b0403 [id=40835fcfaf9b47bc9cea87f9b8785652]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.369863 9604 tablet_service.cc:1505] Processing CreateTablet for tablet 1329ff0f8ebe4da8a6462a574ede98d4 (DEFAULT_TABLE table=default.Table_24e18a6f669742d696b1929a9342ac9f [id=973c679f79994603b7335f87e4f815ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.370177 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet 5054bbc3e27243298b4df6b0a87af8be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.410953 10029 tablet_bootstrap.cc:492] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:56.430908 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f0792045d49041578f5ff6459c99f77e" } modify_external_catalogs: true
I20251027 21:07:56.431108 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f0792045d49041578f5ff6459c99f77e" } modify_external_catalogs: true
W20251027 21:07:56.432309 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8fc2d7d6601d4cc2849064f4cdddc90d consensus state has no leader
I20251027 21:07:56.433476 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 78884c2be75f45f881f5424d3b3f4e08
I20251027 21:07:56.433971 9304 catalog_manager.cc:6719] ReplaceTablet: tablet 78884c2be75f45f881f5424d3b3f4e08 deleted and replaced by tablet 4996016747cd484cbda5f11e40e269b8
I20251027 21:07:56.423599 9594 rpcz_store.cc:276] 1027 21:07:55.224113 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.768454 (+544341us) service_pool.cc:225] Handling call
1027 21:07:56.078067 (+309613us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:56.078112 (+ 45us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.081404 (+ 3292us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:56.318455 (+237051us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:56.318592 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:56.435247 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet aabaa34787cf4d179822269b7f323314 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.435437 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet e10ae34eaaf3476594813de8e96c7e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.422466 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64e39cc210664c3d83eaf1fc576f83ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.437078 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_185df62a04014137809bfc1bc3e5ecad" } modify_external_catalogs: true
I20251027 21:07:56.423871 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet c8f0cd910b9d4daea287b3b192a81bd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.437239 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_185df62a04014137809bfc1bc3e5ecad" } modify_external_catalogs: true
I20251027 21:07:56.437353 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 371aa8a57aee466ab01652d6061db4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.437530 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d2c40f983e743b0b53f987b6a7ba437 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.437697 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 618f1657a10947fd98976820564c20b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.424099 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet e6ed9d7a700446edafcd0daa02aa7ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.438580 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 113a5d30d4b3490187681383d4fef325 (DEFAULT_TABLE table=default.Table_1a51e839945b4912a5519bb1f21a6b75 [id=740df97b965b4a8facf0e11e2c38f020]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.439025 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 113a5d30d4b3490187681383d4fef325. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.439738 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 82366f8bf4154ffd87ef45e4a4d5f54b
I20251027 21:07:56.438099 9603 tablet_service.cc:1505] Processing CreateTablet for tablet a8c46c2a01f946aea8b85875115f7f37 (DEFAULT_TABLE table=default.Table_9b0fd27b43c842008090999fe7da9233 [id=62434317788348feb003dfac99359cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.424876 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet c3b9a461567a48fcb4af514ecdf618b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.425254 10029 ts_tablet_manager.cc:1403] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.350s user 0.002s sys 0.003s
I20251027 21:07:56.445286 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 465be1b24d16457e834023a2b2194049
I20251027 21:07:56.428432 9595 tablet_service.cc:1552] Processing DeleteTablet for tablet c9efffa9766e46aca8eb286fc2ad48d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.433280 9606 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 528) took 1037 ms. Trace:
I20251027 21:07:56.419926 9615 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 524) took 1031 ms. Trace:
I20251027 21:07:56.452736 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet a7d8d060916941a5b7e944e8b08a2882
I20251027 21:07:56.456751 9523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a
I20251027 21:07:56.457535 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e31d4ff2c48e4eea9fe1f13da4fdf218"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.458068 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e31d4ff2c48e4eea9fe1f13da4fdf218 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.458282 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e05ab59d2e0149c7b23dfba66d1bc2c2" } new_table_name: "default.Table_8ec40deea5fd4803a9ee3f6c295a7439" modify_external_catalogs: true
I20251027 21:07:56.458698 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da23505b0fe742ddadfb829e37ca5499. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.458815 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6b452ea1c8f14d038a841d02b164c04a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.459255 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b452ea1c8f14d038a841d02b164c04a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.459468 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f901626832f4adb948ccb65f8573e86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.461694 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f3e1a9db434c519e26e864eb790706. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.463878 9743 tablet_service.cc:1505] Processing CreateTablet for tablet 5054bbc3e27243298b4df6b0a87af8be (DEFAULT_TABLE table=default.Table_61bbf6b5c1e740128b80465ef3b9a50e [id=439f48256cbd43f785f57857e83ae25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.465649 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3f4e692c08244afa2c79e0a075bbcdb. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:56.466931 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ae222307a374a91916f560a0fbe8d8f consensus state has no leader
I20251027 21:07:56.471498 10031 tablet_bootstrap.cc:492] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:56.471665 10031 ts_tablet_manager.cc:1403] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.562s user 0.003s sys 0.007s
I20251027 21:07:56.472451 10031 raft_consensus.cc:359] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:56.472640 10031 raft_consensus.cc:385] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:56.472710 10031 raft_consensus.cc:740] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:56.472924 10031 consensus_queue.cc:260] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:56.475018 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.482396 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5054bbc3e27243298b4df6b0a87af8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.482946 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 82366f8bf4154ffd87ef45e4a4d5f54b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.483134 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 78884c2be75f45f881f5424d3b3f4e08 with delete_type TABLET_DATA_DELETED (replaced by tablet 4996016747cd484cbda5f11e40e269b8) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.484769 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 465be1b24d16457e834023a2b2194049 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.484854 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f24d21bb954ac9b62ce458c6a22462. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.485054 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet a7d8d060916941a5b7e944e8b08a2882 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.486097 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 874d5974698c448ab79f19812cb0e886. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.487128 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82366f8bf4154ffd87ef45e4a4d5f54b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.488128 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bfa70ded2f54e3fb2ca744c723395e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.488837 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe7b00e9cf984d12817c9ea860391199. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.489655 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e04f4c5cf724419ab0be0e84d30a9a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.490296 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7828f466a5ad4d7997721b07f815f56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.491292 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.491591 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 53948dc94c334329bdf81f1eba94863c (DEFAULT_TABLE table=default.Table_09f7ae54c9544043afb9f2baef3ab51d [id=487916f3aa3d49a093bbbac97ebb2fe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.491757 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 622fa1b3ff134f2d8a087ba13a85e597 (DEFAULT_TABLE table=default.Table_8b0df27ef0ab4ed6afe3765a736f8f20 [id=dbc939496da54ddea766d18462c76342]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.491935 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 849b3e9fc34045939a7a678fa8269d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.492136 9904 tablet_service.cc:1505] Processing CreateTablet for tablet d03918b19787449a88e0f17a925ce599 (DEFAULT_TABLE table=default.Table_b64e1c03b47345e2adc8d87940f6280e [id=c0d010fc3b2e4f87a2c5409ce2473cab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.492303 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 0a86c3cc20fc4d149d3847ca281a4ec8 (DEFAULT_TABLE table=default.Table_e58c8bb0dfc540ba81698903abdfdbb0 [id=a2b99ffda2e34bbcb4547836a2524c6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.492434 9905 tablet_service.cc:1505] Processing CreateTablet for tablet a68bd9243b7541bbbc77014113e0106a (DEFAULT_TABLE table=default.Table_f5602dbcd14b4fb5ae7190b6a0bd65bc [id=078add1b91ad4cbfbe2215fe89b2334e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.492797 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 25fbfcc28776483da266bb23c221f882 (DEFAULT_TABLE table=default.Table_cff1f8d7488047e3b7f4608d376e6f5b [id=b105395953eb4a9eaa6931668721d255]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.495046 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_563c35f040434ab5bd6eee6981ddb400"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.495591 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_563c35f040434ab5bd6eee6981ddb400 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.495990 9903 tablet_service.cc:1505] Processing CreateTablet for tablet e24312671b4b4106a1790c61a5548a76 (DEFAULT_TABLE table=default.Table_aa03fc53fae14398b298f577a92d8b5b [id=f1e7ee19e2af41858faf94994bd6144b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.501099 9746 tablet_service.cc:1505] Processing CreateTablet for tablet ed66f9eeeb2b4b4bb77062c241d2140f (DEFAULT_TABLE table=default.Table_79a3420c8b6542868231c740293ab1fb [id=f3babbbf131543d2813ed9d5882e3d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.501400 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed66f9eeeb2b4b4bb77062c241d2140f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.501605 9894 tablet_service.cc:1505] Processing CreateTablet for tablet fdd0198d5047458aab8bf3ce08ccc0de (DEFAULT_TABLE table=default.Table_c59fd341300a4f2f90f1bbd06750d943 [id=83a5151706a64ece82d758b4ef1e2d9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.504045 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_096b7cf08dbf49279d8d5c11b3b687ab" } modify_external_catalogs: true
I20251027 21:07:56.504206 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_096b7cf08dbf49279d8d5c11b3b687ab" } modify_external_catalogs: true
I20251027 21:07:56.507084 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6d0035eeace14307bbb048e1f584b0c5" } new_table_name: "default.Table_4b2294dc7f3745288d9f26232351ebd2" modify_external_catalogs: true
I20251027 21:07:56.509521 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet fde0980aa64c4496861a662f7ed3ef75
I20251027 21:07:56.516397 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_82f669b5326f44efa26969b7d9c7c3f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.516892 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82f669b5326f44efa26969b7d9c7c3f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:56.517366 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d672118ecd6f4094855adf106e5f7115 consensus state has no leader
I20251027 21:07:56.526176 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b4fa791919944d64850f1344cb8f980f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.526633 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4fa791919944d64850f1344cb8f980f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.531307 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 3e46c62b8a1144998eb34ac839563a9d (DEFAULT_TABLE table=default.Table_d00ba224ec354b72a605f17e11591b78 [id=f45509490f4540c698accc119d2ddb27]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.546660 10032 tablet_bootstrap.cc:492] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:56.547456 10032 ts_tablet_manager.cc:1403] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.288s user 0.004s sys 0.007s
I20251027 21:07:56.547932 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9797869b2b084612acc50e375cd116f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.548611 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9797869b2b084612acc50e375cd116f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.549577 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3895a18e2b7a4456a3681679d7c8ba4f" } new_table_name: "default.Table_140424c161ea4581bed312cf5deb5ab5" modify_external_catalogs: true
I20251027 21:07:56.551210 10032 raft_consensus.cc:359] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:56.552037 10032 raft_consensus.cc:385] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:56.552420 10032 raft_consensus.cc:740] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:56.553500 10032 consensus_queue.cc:260] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:56.557034 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6acc073c9e6b45958ee3033e7f0695f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.557461 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6acc073c9e6b45958ee3033e7f0695f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.558387 9615 rpcz_store.cc:276] 1027 21:07:55.388856 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.874414 (+485558us) service_pool.cc:225] Handling call
1027 21:07:55.923040 (+ 48626us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:55.923063 (+ 23us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.415085 (+492022us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:56.419719 (+ 4634us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:56.419869 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:56.559330 9595 tablet_service.cc:1552] Processing DeleteTablet for tablet ed5619d295a246379b625f35692203e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.561528 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4e25bcf24847404786773903009ac9f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.561992 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e25bcf24847404786773903009ac9f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.561926 10032 ts_tablet_manager.cc:1434] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
W20251027 21:07:56.566221 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 995284883d834bbb9ef583b736ca8402 consensus state has no leader
I20251027 21:07:56.566428 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c46c2a01f946aea8b85875115f7f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.566989 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet 64e39cc210664c3d83eaf1fc576f83ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.567363 9613 tablet_service.cc:1505] Processing CreateTablet for tablet b99232a6018a4b1a850da954c551d38d (DEFAULT_TABLE table=default.Table_cf9f3cddff8a486fbe76334c2fd833c5 [id=983747413ebc4e3ea8ccb0eac5ffff08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.570308 9606 rpcz_store.cc:276] 1027 21:07:55.394973 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.943266 (+548293us) service_pool.cc:225] Handling call
1027 21:07:56.096142 (+152876us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:56.096176 (+ 34us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.352506 (+256330us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:56.430645 (+ 78139us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:56.432835 (+ 2190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:56.572642 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e74355d0f90e4c50abfb229be0c398c8" } new_table_name: "default.Table_203191e192be4e3a9a7477b4313ef4f4" modify_external_catalogs: true
I20251027 21:07:56.574100 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d03918b19787449a88e0f17a925ce599. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.576445 9307 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 7aa8c538b6c849a28a28ffe706306891 (table default.Table_b779f2756a924ab6ba1bc1702a412e06 [id=8ed572891f9b41cba8ed984911fa6b11]). Expected version 1 got 0
I20251027 21:07:56.539945 9743 tablet_service.cc:1552] Processing DeleteTablet for tablet 874d5974698c448ab79f19812cb0e886 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.578289 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet e0968c2f25b64f6685f38a999ee9d60e (table default.Table_68f478946b1543a68842ffad9e14864f [id=830444c79e80486aa3e7668a0ec91023]) successfully deleted
W20251027 21:07:56.579903 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8856c5dc8f144358445165324bfb477 consensus state has no leader
I20251027 21:07:56.580953 10041 tablet_replica.cc:333] T e841c54a94d649c19b639f5021f150cd P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:56.581120 10041 raft_consensus.cc:2243] T e841c54a94d649c19b639f5021f150cd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:56.581305 10041 raft_consensus.cc:2272] T e841c54a94d649c19b639f5021f150cd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:56.581837 10041 ts_tablet_manager.cc:1916] T e841c54a94d649c19b639f5021f150cd P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:56.582477 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 618f1657a10947fd98976820564c20b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 618f1657a10947fd98976820564c20b3 already in progress: creating tablet
I20251027 21:07:56.583045 9900 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 550) took 1056 ms. Trace:
I20251027 21:07:56.583163 9900 rpcz_store.cc:276] 1027 21:07:55.526374 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.039153 (+512779us) service_pool.cc:225] Handling call
1027 21:07:56.097578 (+ 58425us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:56.097597 (+ 19us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.283111 (+185514us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:56.508753 (+225642us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:56.582983 (+ 74230us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251027 21:07:56.583364 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet e10ae34eaaf3476594813de8e96c7e77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e10ae34eaaf3476594813de8e96c7e77 already in progress: creating tablet
I20251027 21:07:56.583300 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 9239f74d725649cab5ea1b0077bccef3 (DEFAULT_TABLE table=default.Table_d5b9049d02b8474aafbf1e27d9f103ec [id=553cbebcfbca40f596b31399ef218181]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.583634 9900 tablet_service.cc:1505] Processing CreateTablet for tablet 959c48ed8a8e4d2189aa6546fdab4e7d (DEFAULT_TABLE table=default.Table_c58f3dadcca44b7d81025d6b5a7be405 [id=8eacacc3c28941baa04ebff88d8928c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.583992 9893 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 546) took 1090 ms. Trace:
I20251027 21:07:56.584076 9893 rpcz_store.cc:276] 1027 21:07:55.493493 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.031796 (+538303us) service_pool.cc:225] Handling call
1027 21:07:56.032245 (+ 449us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:56.032269 (+ 24us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.219742 (+187473us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:56.510000 (+290258us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:56.583948 (+ 73948us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:56.584457 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 38f2133010b14aad88b550fb12c966a5 (DEFAULT_TABLE table=default.Table_0812ae8e053c4e6f913221ef573f618f [id=aaad36e806d140bdbbdd90ed29d945e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.584813 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38f2133010b14aad88b550fb12c966a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.588250 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 04a50ad9927e441faa3c57579c9cd215 (DEFAULT_TABLE table=default.Table_2ef7853cbd8b4f59a59cec83c91857c2 [id=8da438c490124663982f76f5b075126b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.588809 9891 tablet_service.cc:1505] Processing CreateTablet for tablet a12bf9c3d0b445dc92d54aa3059b834c (DEFAULT_TABLE table=default.Table_1a2deac4d0444b85b51757688f89b202 [id=71db9ec3df6a4f71aaaefb47e219929c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.589300 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 959c48ed8a8e4d2189aa6546fdab4e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.589331 9897 tablet_service.cc:1505] Processing CreateTablet for tablet a37ff8650247492a9c20a3b55300c5f6 (DEFAULT_TABLE table=default.Table_b230ee0b39fd44e0b9a956f480073963 [id=72addfdb5cfb4964a1eccbd639c04039]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.589787 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9239f74d725649cab5ea1b0077bccef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.589864 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a (DEFAULT_TABLE table=default.Table_db53bd5823b84458b7cd9564d5e9676e [id=3e816b6785f8495fa468922a28feee04]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.591408 9901 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 549) took 1072 ms. Trace:
I20251027 21:07:56.591531 9901 rpcz_store.cc:276] 1027 21:07:55.518839 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.037894 (+519055us) service_pool.cc:225] Handling call
1027 21:07:56.097297 (+ 59403us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:56.097328 (+ 31us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.481757 (+384429us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:56.522623 (+ 40866us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:56.591357 (+ 68734us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:56.591883 9901 tablet_service.cc:1505] Processing CreateTablet for tablet 33c5bc33130a424fb63e3f92a2cc49fc (DEFAULT_TABLE table=default.Table_fdf2451e0a2a48f9b902827e2a291e90 [id=6eae55ab35bf40bca718266926b1187f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.592167 9901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c5bc33130a424fb63e3f92a2cc49fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.594067 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet 84b9de82d55447bba2a897edb683e5c5 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:56.594985 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_cf9164135a4b4ccc91f15ca37555a7c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.595654 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf9164135a4b4ccc91f15ca37555a7c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.598269 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c6736558d45d4ba291581bb4b39a2208"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.598711 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6736558d45d4ba291581bb4b39a2208 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.600198 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a37ff8650247492a9c20a3b55300c5f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.601038 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04a50ad9927e441faa3c57579c9cd215. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.602896 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 64e39cc210664c3d83eaf1fc576f83ce (DEFAULT_TABLE table=default.Table_ef27c46c59a94abfbc41c240b0477362 [id=dc9d6d1f90084f18b22ec006f5161669]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.603443 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet 94e94cd50c91429d8da9804149501c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.603611 10358 raft_consensus.cc:493] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:56.606247 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a12bf9c3d0b445dc92d54aa3059b834c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.607630 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1329ff0f8ebe4da8a6462a574ede98d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.606066 9743 tablet_service.cc:1505] Processing CreateTablet for tablet c25be263facd42c5bb6fa421160fbef0 (DEFAULT_TABLE table=default.Table_9d77e1ff90f047748d72fe89fbcc0c49 [id=e1a7e7b9b73746db840ba0594bca9239]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.610087 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 84b9de82d55447bba2a897edb683e5c5
I20251027 21:07:56.612481 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e46c62b8a1144998eb34ac839563a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.607654 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 479d2cc8b8ce4c26841f50eea7c0e09a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.618702 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64e39cc210664c3d83eaf1fc576f83ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.618475 9307 catalog_manager.cc:6719] ReplaceTablet: tablet 84b9de82d55447bba2a897edb683e5c5 deleted and replaced by tablet befdd3a485a1426b8a3adf340d4eae4f
I20251027 21:07:56.624559 10029 raft_consensus.cc:359] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:56.625761 10029 raft_consensus.cc:385] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:56.625860 10029 raft_consensus.cc:740] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:56.627095 9595 tablet_service.cc:1505] Processing CreateTablet for tablet a6edd6a5c4434f0497667b1a2771f885 (DEFAULT_TABLE table=default.Table_f0e8c68b9f1243bfb3584cca5b5f7002 [id=1d7f0c839ec546178f6cd39b81db07ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.623819 9615 tablet_service.cc:1505] Processing CreateTablet for tablet c6a189aa22d048858e9f30bf4276859d (DEFAULT_TABLE table=default.Table_abc7f8be9f414322ad9064340647399f [id=93edbbc1222b4ce990f6dc4e9a8cbe25]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.627604 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 953b137f14a44955ae41ab503d04b43c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.628177 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 0b4bf2b20e7c416aad123c6f96355cac (DEFAULT_TABLE table=default.Table_3c0c40b875fc45969c1a916c5545c4cb [id=699b0f7383ef48aeb27575efd44e5658]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.630260 10358 raft_consensus.cc:515] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:56.630815 10358 leader_election.cc:290] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:56.631372 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94e94cd50c91429d8da9804149501c86" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:56.632232 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 123e6e3748574ea581f67fff49037762 (DEFAULT_TABLE table=default.Table_1a08b666f7f4426eb19757e15584dd53 [id=2537aadc803d44cdbd43de4694661e79]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.630846 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_24e18a6f669742d696b1929a9342ac9f" } modify_external_catalogs: true
I20251027 21:07:56.633603 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a3f6fbc547e4462097650dbde068ef25" } new_table_name: "default.Table_6327cacf4ed54d56b522291ad9594784" modify_external_catalogs: true
I20251027 21:07:56.639374 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 191009830c30467b9c5ab39e13b7c267. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:56.641065 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67cb71ce43244d8ab9e71ead6bb315c9 consensus state has no leader
I20251027 21:07:56.641873 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 631ff72613734913818a3cad24f20dbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:56.642057 9566 leader_election.cc:343] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:56.644253 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a7632420526d409290924efb41972ee1" } modify_external_catalogs: true
I20251027 21:07:56.647003 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ccc943e7fbc54e609e33b3a131a07c74" } modify_external_catalogs: true
I20251027 21:07:56.647198 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ccc943e7fbc54e609e33b3a131a07c74" } modify_external_catalogs: true
I20251027 21:07:56.648129 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c023428437f45b4bae1374bd3dd36a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.647797 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a7632420526d409290924efb41972ee1" } modify_external_catalogs: true
I20251027 21:07:56.648751 10029 consensus_queue.cc:260] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:56.634449 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_24e18a6f669742d696b1929a9342ac9f" } modify_external_catalogs: true
I20251027 21:07:56.649782 9609 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 527) took 1249 ms. Trace:
I20251027 21:07:56.651399 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 436) took 1718 ms. Trace:
I20251027 21:07:56.655501 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c2e56e5918cb4007b681c71977325233" } modify_external_catalogs: true
I20251027 21:07:56.655685 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c2e56e5918cb4007b681c71977325233" } modify_external_catalogs: true
I20251027 21:07:56.658246 9609 rpcz_store.cc:276] 1027 21:07:55.393924 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.915921 (+521997us) service_pool.cc:225] Handling call
1027 21:07:55.916361 (+ 440us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:55.916382 (+ 21us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.425614 (+509232us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:56.474337 (+ 48723us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:56.643537 (+169200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:56.658754 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 9b9ffa18afa54f2c9ffd7b5a03a3a324 (DEFAULT_TABLE table=default.Table_ccc943e7fbc54e609e33b3a131a07c74 [id=1cc502193c3443898bc7eafa5ed15652]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.659377 10029 ts_tablet_manager.cc:1434] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.216s user 0.001s sys 0.001s
I20251027 21:07:56.659709 10029 tablet_bootstrap.cc:492] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:56.661000 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bbb69d62b9a41e3b88a5527f7549881. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.664157 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 37ae3b6bf590473e8e09092fc5f7df1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.664597 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 2f641bc4dbed4c0abd0a55b253352507 (DEFAULT_TABLE table=default.Table_12da35feb05c42a79377049e12fe73d6 [id=8a26e744abae4c5eb1d18dd994cd6bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.668509 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6df39c16cd96415fa08e5ad69b4b4504" } new_table_name: "default.Table_090fa7c40bdb40d09777fd8458acd6b2" modify_external_catalogs: true
I20251027 21:07:56.668551 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f21fe641a0574b80a1fe5551e551e96c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.669039 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f21fe641a0574b80a1fe5551e551e96c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:56.670327 9273 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu pid 10108: Timed out: CheckLeaks RPC to 127.9.14.124:45071 timed out after 1.000s (SENT)
I20251027 21:07:56.670533 9273 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu with pid 10108
I20251027 21:07:56.671963 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b9ffa18afa54f2c9ffd7b5a03a3a324. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.672206 9595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6edd6a5c4434f0497667b1a2771f885. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.673072 10043 rpcz_store.cc:276] 1027 21:07:54.929991 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.931447 (+ 1456us) service_pool.cc:225] Handling call
1027 21:07:55.912422 (+980975us) ts_tablet_manager.cc:1151] Deleting tablet 7d8825b80cec4ba68da077ca3b8d89db
1027 21:07:55.912434 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.137577 (+225143us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:56.357441 (+219864us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:56.648477 (+291036us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2497,"tablet-delete.queue_time_us":980736}
I20251027 21:07:56.673528 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b4bf2b20e7c416aad123c6f96355cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.673756 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 7d8825b80cec4ba68da077ca3b8d89db (table default.Table_943417713b3b456ab4ca691d2fb168fd [id=1cb32eea25114c6dbcb3b28044036c08]) successfully deleted
I20251027 21:07:56.673849 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3afa18a27b0b442780b1a3182be6b779"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.674347 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3afa18a27b0b442780b1a3182be6b779 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.674578 9751 tablet_service.cc:1505] Processing CreateTablet for tablet 4d88aacbf47144cb837b99d68f6331e2 (DEFAULT_TABLE table=default.Table_bc5ec745ebe54017b6bc9635581c813d [id=31172964ef544c5f99cade30a42b340d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.678678 9888 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 544) took 1120 ms. Trace:
I20251027 21:07:56.679935 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 123e6e3748574ea581f67fff49037762. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:56.679996 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 94f2819dd0394530ae93754bdd1e482f consensus state has no leader
I20251027 21:07:56.680423 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 1b7cf52a2ac249a0ac786ddc24acdae5 (DEFAULT_TABLE table=default.Table_919a2713d2c54825aa74493e188181e6 [id=af408a20521f4e2fbba065b502422579]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.681406 9888 rpcz_store.cc:276] 1027 21:07:55.481338 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.946095 (+464757us) service_pool.cc:225] Handling call
1027 21:07:56.039664 (+ 93569us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:56.039697 (+ 33us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.372966 (+333269us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:56.530050 (+157084us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:56.602126 (+ 72076us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:56.681813 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 6e06ae559ca14b8bb0630e1571ca4852 (DEFAULT_TABLE table=default.Table_ea4ca66f1a144318a185c1324b2f3f61 [id=10a848ac0e3d4c1288be7fc15c1ce4b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.682173 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53948dc94c334329bdf81f1eba94863c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.683039 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e24312671b4b4106a1790c61a5548a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.686951 9886 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 551) took 1158 ms. Trace:
I20251027 21:07:56.687145 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b7cf52a2ac249a0ac786ddc24acdae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.694464 10032 tablet_bootstrap.cc:492] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:56.694468 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdd0198d5047458aab8bf3ce08ccc0de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.701382 9308 master_service.cc:954] ReplaceTablet: received request to replace tablet b3abc5f78c4e41e29128e2d790319c75 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:56.701980 9736 tablet_service.cc:1505] Processing CreateTablet for tablet c0e2772d66404d74891e0893c49a66e0 (DEFAULT_TABLE table=default.Table_94d4f5b9325a4c27a6305962fe774808 [id=c92fda1282d7419f94b0004ab8b8403d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.704059 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4cbc59c030d044d8a097385ed57a0b5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.704594 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4cbc59c030d044d8a097385ed57a0b5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.706605 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet cfce0e5c837743ea969374244163be3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.706995 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 6c61c360653346fabff0718f0990d100 (DEFAULT_TABLE table=default.Table_6f6b1a85f5f2410f8c36649b6a7b44a9 [id=6a0bd72e2b44485bac23d34cddf8a57e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.707367 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c25be263facd42c5bb6fa421160fbef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.708101 9889 tablet_service.cc:1505] Processing CreateTablet for tablet deb8b54e5e0f42bca4330d2160e5524a (DEFAULT_TABLE table=default.Table_851f66cf4b1047879eed5fb768c07156 [id=743b2e05c3674d1fabb93aa073d0df40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.709479 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a68bd9243b7541bbbc77014113e0106a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.710661 9745 tablet_service.cc:1505] Processing CreateTablet for tablet bc9322398d024e629a3c2aae7b6a77be (DEFAULT_TABLE table=default.Table_858fba8378424b5399351d9134fa7e60 [id=4239b383dcb841468abe1c9fbb0b0b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.710727 9886 rpcz_store.cc:276] 1027 21:07:55.528359 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.040145 (+511786us) service_pool.cc:225] Handling call
1027 21:07:56.097995 (+ 57850us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:56.098014 (+ 19us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.249102 (+151088us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:56.497902 (+248800us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:56.686914 (+189012us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:56.711889 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_162a38d7fe14491d8dae6d2e097d3e3a" } modify_external_catalogs: true
I20251027 21:07:56.712056 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_162a38d7fe14491d8dae6d2e097d3e3a" } modify_external_catalogs: true
I20251027 21:07:56.694235 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 4bbb69d62b9a41e3b88a5527f7549881 (DEFAULT_TABLE table=default.Table_ff202033c9ec4fba9cc9525ea6767cac [id=506208e23e3a410fbd15adc68b4c471a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.716086 9312 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 7aa8c538b6c849a28a28ffe706306891 (table default.Table_b779f2756a924ab6ba1bc1702a412e06 [id=8ed572891f9b41cba8ed984911fa6b11]). Expected version 1 got 0
I20251027 21:07:56.717453 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a86c3cc20fc4d149d3847ca281a4ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.719264 9886 tablet_service.cc:1552] Processing DeleteTablet for tablet fde0980aa64c4496861a662f7ed3ef75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.720140 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.720999 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 07a9c3b7ef1548ec9d5ae9f20545f263
I20251027 21:07:56.722489 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8a1e00e5debb481cbc1fdef16d74527b" } new_table_name: "default.Table_bb56319b6fe0405991a4d171477621e7" modify_external_catalogs: true
I20251027 21:07:56.725461 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc9322398d024e629a3c2aae7b6a77be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.727420 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet 92a2393cff8044028c86c8cd2bccd930 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.727865 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 439) took 1792 ms. Trace:
I20251027 21:07:56.729122 10043 rpcz_store.cc:276] 1027 21:07:54.935774 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.935977 (+ 203us) service_pool.cc:225] Handling call
1027 21:07:56.674223 (+1738246us) ts_tablet_manager.cc:1151] Deleting tablet 7292bc6c22ca4ab5b8f53a69520833ba
1027 21:07:56.674235 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.727810 (+ 53575us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1737983}
I20251027 21:07:56.729144 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c9c1ea88906a4ee6bbd763eced368b50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.729615 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9c1ea88906a4ee6bbd763eced368b50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.729691 10040 ts_tablet_manager.cc:1929] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:07:56.733296 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2a89c7b51adb4448b74b750f3ec00c53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:56.733831 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a89c7b51adb4448b74b750f3ec00c53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:56.734438 10031 ts_tablet_manager.cc:1434] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.255s user 0.002s sys 0.002s
I20251027 21:07:56.735086 9886 tablet_service.cc:1505] Processing CreateTablet for tablet eeb667b3b10746dbad351c4c123ffd97 (DEFAULT_TABLE table=default.Table_4aae3c9852244881bf5286450425cf5d [id=f500277871dc4f78ab54c95ad34ef1c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.735311 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bbb69d62b9a41e3b88a5527f7549881. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.736327 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25fbfcc28776483da266bb23c221f882. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.738694 9904 tablet_service.cc:1505] Processing CreateTablet for tablet f543e8bcf70c4acab9880ee01af85562 (DEFAULT_TABLE table=default.Table_7d6df7f8ab8945a894c5984d320667b0 [id=4b12cf2789744c5b8d02c1a82b89e118]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.741528 9739 tablet_service.cc:1505] Processing CreateTablet for tablet b83daa33ef0f45c6a78c70001e532ea2 (DEFAULT_TABLE table=default.Table_20d9ef0583ca492a85be3bf2cdf16b5b [id=5ce64cf618c248cba652043b26376cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.742913 10040 log.cc:1199] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/f6249fc5c6174e8e81b9d02d598d5ebd
I20251027 21:07:56.745108 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 1329ff0f8ebe4da8a6462a574ede98d4
I20251027 21:07:56.745858 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 69da799c886841a19002d3c7c5084e73
I20251027 21:07:56.746635 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 8fc2d7d6601d4cc2849064f4cdddc90d
I20251027 21:07:56.753700 9744 tablet_service.cc:1505] Processing CreateTablet for tablet bbf92e29993b46389a69efd9e9a2341f (DEFAULT_TABLE table=default.Table_9a6d7cca704541ca80a3cf047aea2914 [id=fc420109cc2e43ada2c4ca5123fa0998]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.754081 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 699c6a8dd8d44d729df463b20cdcfdda (DEFAULT_TABLE table=default.Table_fe40346a4d3641159b11c6b9ec566eca [id=6e17bac6ca06444aa464cb19795ed793]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.757822 9747 tablet_service.cc:1505] Processing CreateTablet for tablet 953b137f14a44955ae41ab503d04b43c (DEFAULT_TABLE table=default.Table_98ca10ad3609477aaf22dc563a68742e [id=8090dd5b916d49719ded3b0209a80b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.759920 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 4c023428437f45b4bae1374bd3dd36a7 (DEFAULT_TABLE table=default.Table_7453f892d0514227b015e34cf2461acd [id=f79bde1a71234b53b5a960c983481cc1]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:56.703706 9568 leader_election.cc:343] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 94e94cd50c91429d8da9804149501c86
I20251027 21:07:56.760223 9612 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 517) took 1314 ms. Trace:
I20251027 21:07:56.761245 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b99232a6018a4b1a850da954c551d38d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.762321 9568 leader_election.cc:304] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
W20251027 21:07:56.762737 9568 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 76328us
W20251027 21:07:56.764797 9568 service_pool.cc:137] DeleteTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:59712 dropped due to backpressure: service queue is full with 50 items
I20251027 21:07:56.764899 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a189aa22d048858e9f30bf4276859d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.765054 10358 raft_consensus.cc:2749] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:07:56.766547 9566 consensus_peers.cc:597] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:07:56.766757 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 9351a369dc1e4ac49f6ad419eab3a4c5 with delete_type TABLET_DATA_DELETED (replaced by tablet d5f4f3990da34f5e8f0385595f457427) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.766934 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet e477ca4b69c84327b04618d90406e19a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.767089 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet bbf92e29993b46389a69efd9e9a2341f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.767241 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet cb1686fc7c3a4996aa4462884d30c41f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.767375 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.767513 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 78884c2be75f45f881f5424d3b3f4e08 with delete_type TABLET_DATA_DELETED (replaced by tablet 4996016747cd484cbda5f11e40e269b8) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.767688 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 82366f8bf4154ffd87ef45e4a4d5f54b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.767863 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 465be1b24d16457e834023a2b2194049 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.767901 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 95f9595784df434dbb0dd29d9a05511e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.768066 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet a7d8d060916941a5b7e944e8b08a2882 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.768247 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.768793 10029 tablet_bootstrap.cc:654] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:56.769516 9754 tablet_service.cc:1505] Processing CreateTablet for tablet e8fe63ea8aba4614900ac254189464de (DEFAULT_TABLE table=default.Table_64efdeb42d0f4db68c0e13066cd8d38f [id=439b38e1dcd84d7d9fd40dd9cbe3a429]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.769016 9741 tablet_service.cc:1505] Processing CreateTablet for tablet 7414744bd9bd40cebe126925af9f7a31 (DEFAULT_TABLE table=default.Table_3e21b34c57774bd2843bbac07d8892ac [id=76c0dad4c78644fc9107e3c8caed66a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.773109 9601 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 529) took 1273 ms. Trace:
I20251027 21:07:56.774627 9601 rpcz_store.cc:276] 1027 21:07:55.395119 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.944655 (+549536us) service_pool.cc:225] Handling call
1027 21:07:55.945104 (+ 449us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:55.945134 (+ 30us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.626522 (+681388us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:56.667099 (+ 40577us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:56.668784 (+ 1685us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:56.775097 10040 ts_tablet_manager.cc:1950] T f6249fc5c6174e8e81b9d02d598d5ebd P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:56.775208 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 11627a84ff9b478c87c323617a09570b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.776260 10031 tablet_bootstrap.cc:492] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:56.776348 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f641bc4dbed4c0abd0a55b253352507. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.776780 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0e2772d66404d74891e0893c49a66e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.777007 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b83daa33ef0f45c6a78c70001e532ea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.777216 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c023428437f45b4bae1374bd3dd36a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.777707 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7414744bd9bd40cebe126925af9f7a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.780840 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 699c6a8dd8d44d729df463b20cdcfdda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.781024 9742 tablet_service.cc:1505] Processing CreateTablet for tablet edbf5f0db6fa40c7b24f1cd20ee80dd4 (DEFAULT_TABLE table=default.Table_728cbf4f936c4df79d77156e2d3ac59c [id=5c378e8c32d34067bfa42a4b05d6ea7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.781339 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edbf5f0db6fa40c7b24f1cd20ee80dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.783264 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 953b137f14a44955ae41ab503d04b43c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.784047 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8fe63ea8aba4614900ac254189464de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.776261 9600 tablet_service.cc:1505] Processing CreateTablet for tablet a68bd9243b7541bbbc77014113e0106a (DEFAULT_TABLE table=default.Table_f5602dbcd14b4fb5ae7190b6a0bd65bc [id=078add1b91ad4cbfbe2215fe89b2334e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.784744 9612 rpcz_store.cc:276] 1027 21:07:55.358358 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.825339 (+466981us) service_pool.cc:225] Handling call
1027 21:07:55.825781 (+ 442us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:55.825802 (+ 21us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.559555 (+733753us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:56.672557 (+113002us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:56.672671 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:56.785003 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbf92e29993b46389a69efd9e9a2341f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.785712 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e4d35f229bc46e1859ded7c8196ab9d with delete_type TABLET_DATA_DELETED (replaced by tablet 280ee23d80d64587b18811ff06340eee) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:56.786556 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 451) took 1782 ms. Trace:
I20251027 21:07:56.787609 10043 rpcz_store.cc:276] 1027 21:07:54.994454 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.994507 (+ 53us) service_pool.cc:225] Handling call
1027 21:07:56.760642 (+1766135us) ts_tablet_manager.cc:1151] Deleting tablet 0a58cd07e1bb4ffdbf39e7e16a46022a
1027 21:07:56.760656 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.776408 (+ 15752us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1662061}
I20251027 21:07:56.788251 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a68bd9243b7541bbbc77014113e0106a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.788580 9893 tablet_service.cc:1505] Processing CreateTablet for tablet de2c033a064643b386ee84085cbdad64 (DEFAULT_TABLE table=default.Table_09602bd1d278472bb1ed05abe9d1def4 [id=9a86a04b859446a685bcc946e7331dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.793429 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 352) took 3602 ms. Trace:
I20251027 21:07:56.793547 10040 rpcz_store.cc:276] 1027 21:07:53.185156 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.186902 (+ 1746us) service_pool.cc:225] Handling call
1027 21:07:56.061964 (+2875062us) ts_tablet_manager.cc:1151] Deleting tablet f6249fc5c6174e8e81b9d02d598d5ebd
1027 21:07:56.061971 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.516560 (+454589us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:56.679591 (+163031us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:56.787058 (+107467us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2866961}
I20251027 21:07:56.797649 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 25fbfcc28776483da266bb23c221f882 (DEFAULT_TABLE table=default.Table_cff1f8d7488047e3b7f4608d376e6f5b [id=b105395953eb4a9eaa6931668721d255]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.798074 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25fbfcc28776483da266bb23c221f882. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.798295 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 933b33aac5354778a5866d340fd34db4 (DEFAULT_TABLE table=default.Table_7a9e545bc9374c8b986687268e421690 [id=a97617c2cd71489aaf45385d3cb1ca9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.798427 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 191009830c30467b9c5ab39e13b7c267 (DEFAULT_TABLE table=default.Table_f846038ed8334d998a32d5775f2b0403 [id=40835fcfaf9b47bc9cea87f9b8785652]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.798951 9753 tablet_service.cc:1505] Processing CreateTablet for tablet b913b021bfa84a6a8858434231e958fc (DEFAULT_TABLE table=default.Table_3d352cd73ac64a89ade7b276936b52cc [id=3f5ef05eb0bf4cf5a11ee0d405c0b23e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.764921 9568 service_pool.cc:138] DeleteTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:59712 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 573)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 574)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 575)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 576)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 577)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 578)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 579)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 580)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 581)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 582)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 583)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 584)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 585)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 586)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 587)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 588)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 589)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 590)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 591)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 592)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 593)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 594)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 595)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 596)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 597)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 598)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 599)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 600)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 601)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 602)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 603)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 604)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 606)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 605)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 607)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 608)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 609)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 610)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 611)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 612)
W20251027 21:07:56.800335 9298 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: DeleteTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:59712 dropped due to backpressure: service queue is full with 50 items [suppressed 4 similar messages]
I20251027 21:07:56.800494 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e06ae559ca14b8bb0630e1571ca4852. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.802299 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 353) took 3607 ms. Trace:
I20251027 21:07:56.802402 10040 rpcz_store.cc:276] 1027 21:07:53.194027 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.194173 (+ 146us) service_pool.cc:225] Handling call
1027 21:07:56.800826 (+3606653us) ts_tablet_manager.cc:1151] Deleting tablet 588e2f58f07b4a32a78dffd5b92ceb12
1027 21:07:56.800837 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.801311 (+ 474us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3605088}
I20251027 21:07:56.802793 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 354) took 3607 ms. Trace:
I20251027 21:07:56.802870 10040 rpcz_store.cc:276] 1027 21:07:53.194961 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.195062 (+ 101us) service_pool.cc:225] Handling call
1027 21:07:56.802655 (+3607593us) ts_tablet_manager.cc:1151] Deleting tablet f4187b2f66f24055b1df170f1062cc88
1027 21:07:56.802662 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.802745 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3606759}
I20251027 21:07:56.803225 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 355) took 3607 ms. Trace:
I20251027 21:07:56.803321 10040 rpcz_store.cc:276] 1027 21:07:53.196094 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.196152 (+ 58us) service_pool.cc:225] Handling call
1027 21:07:56.803098 (+3606946us) ts_tablet_manager.cc:1151] Deleting tablet bbb4116d1edd492893adfbf5fb3ddeef
1027 21:07:56.803105 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.803174 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3606117}
I20251027 21:07:56.803655 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 357) took 3569 ms. Trace:
I20251027 21:07:56.803649 10411 raft_consensus.cc:493] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:56.803747 10040 rpcz_store.cc:276] 1027 21:07:53.234303 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.234359 (+ 56us) service_pool.cc:225] Handling call
1027 21:07:56.803513 (+3569154us) ts_tablet_manager.cc:1151] Deleting tablet b719dc40d7e348608cdfc78401c19824
1027 21:07:56.803520 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.803605 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3568969}
I20251027 21:07:56.804092 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 356) took 3574 ms. Trace:
I20251027 21:07:56.804172 10040 rpcz_store.cc:276] 1027 21:07:53.229939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.294699 (+ 64760us) service_pool.cc:225] Handling call
1027 21:07:56.803956 (+3509257us) ts_tablet_manager.cc:1151] Deleting tablet 1d64a13667194c12864fa388eb88a380
1027 21:07:56.803964 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.804037 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3508968}
I20251027 21:07:56.804435 10040 tablet_replica.cc:333] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:56.804627 10040 raft_consensus.cc:2243] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:56.804811 10040 raft_consensus.cc:2272] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:56.804885 10411 raft_consensus.cc:515] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:56.805383 10040 ts_tablet_manager.cc:1916] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:56.805593 10411 leader_election.cc:290] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:56.811822 9897 tablet_service.cc:1505] Processing CreateTablet for tablet aef964f116ca4f98a5fcfa4d9f44437a (DEFAULT_TABLE table=default.Table_85b289c2c1174393803662453ba6ac8f [id=5f0e3f844ce7452f92e3bdda3be5c58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.815812 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deb8b54e5e0f42bca4330d2160e5524a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.818913 9739 tablet_service.cc:1505] Processing CreateTablet for tablet 1329ff0f8ebe4da8a6462a574ede98d4 (DEFAULT_TABLE table=default.Table_24e18a6f669742d696b1929a9342ac9f [id=973c679f79994603b7335f87e4f815ea]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:56.820269 9711 service_pool.cc:137] CreateTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:40732 dropped due to backpressure: service queue is full with 50 items
I20251027 21:07:56.820602 9901 tablet_service.cc:1505] Processing CreateTablet for tablet 37cf73d154e24dfeb85a335d8a2d6f10 (DEFAULT_TABLE table=default.Table_5804b71f15924faf9b6c09ac13d09d3c [id=17af96dd9b7f4daabc64b44aaedfef5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.822851 9711 service_pool.cc:138] CreateTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:40732 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 542)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 543)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 544)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 545)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 546)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 547)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 548)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 549)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 550)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 551)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 552)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 553)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 554)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 555)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 556)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 557)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 558)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 559)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 560)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 561)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 562)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 563)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 564)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 565)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 566)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 567)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 568)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 569)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 570)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 571)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 572)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 573)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 574)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 575)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 576)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 577)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 578)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 579)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 580)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 581)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 582)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 583)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 584)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 585)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 586)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 587)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 588)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 589)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 590)
I20251027 21:07:56.824196 9711 service_pool.cc:138] DeleteTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:40732 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 542)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 543)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 544)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 545)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 546)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 547)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 548)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 549)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 550)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 551)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 552)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 553)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 554)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 555)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 556)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 557)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 558)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 559)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 560)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 561)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 562)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 563)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 564)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 565)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 566)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 567)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 568)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 569)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 570)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 571)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 572)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 573)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 574)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 575)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 576)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 577)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 578)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 579)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 580)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 581)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 582)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 593)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 583)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 584)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 585)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 586)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 587)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 588)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 589)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 590)
I20251027 21:07:56.825119 9711 service_pool.cc:138] CreateTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:40732 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 542)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 543)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 544)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 545)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 546)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 547)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 548)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 549)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 550)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 551)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 552)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 553)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 554)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 555)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 556)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 557)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 558)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 559)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 560)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 561)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 562)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 563)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 564)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 565)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 566)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 567)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 568)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 569)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 570)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 571)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 572)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 573)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 574)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 575)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 576)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 577)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 578)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 579)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 580)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 581)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 582)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 593)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 583)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 584)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 585)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 586)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 587)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 588)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 589)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 590)
I20251027 21:07:56.826517 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c61c360653346fabff0718f0990d100. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.826814 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b913b021bfa84a6a8858434231e958fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.827083 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d88aacbf47144cb837b99d68f6331e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.828006 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1329ff0f8ebe4da8a6462a574ede98d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.828675 10411 raft_consensus.cc:493] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:56.828832 10411 raft_consensus.cc:515] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:56.829921 10041 ts_tablet_manager.cc:1929] T e841c54a94d649c19b639f5021f150cd P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:56.831084 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 191009830c30467b9c5ab39e13b7c267. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.831243 10411 leader_election.cc:290] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:56.840140 9900 tablet_service.cc:1505] Processing CreateTablet for tablet 87612b8bd8a04fd9b32f2df3f9e0d997 (DEFAULT_TABLE table=default.Table_45436efc9fd041c0a813930bd895b3dd [id=25ffc58bf0a54adfa648f55c76f3df44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.844461 10041 log.cc:1199] T e841c54a94d649c19b639f5021f150cd P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/e841c54a94d649c19b639f5021f150cd
I20251027 21:07:56.846868 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 57de49dacafa497dad79ab96540b5888 (DEFAULT_TABLE table=default.Table_2a9fa2fa9615471691f07230a08e1399 [id=876a8959f347456a977578a8dd9e0645]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.851987 10041 ts_tablet_manager.cc:1950] T e841c54a94d649c19b639f5021f150cd P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:56.853749 10041 tablet_replica.cc:333] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:56.856092 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 0ac1aed5f56844639b848f90c8eb5b6c (DEFAULT_TABLE table=default.Table_0e8bd764e80e46a5b665da0e3e2b677e [id=288ecea80da64655ae6246da518b77b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.858843 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet e841c54a94d649c19b639f5021f150cd (table default.Table_5cdd921f5bdf47bab7ec19ac00c6292f [id=d392262a11e24cf0991bc200475b0ed6]) successfully deleted
I20251027 21:07:56.860708 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 84b9de82d55447bba2a897edb683e5c5 with delete_type TABLET_DATA_DELETED (replaced by tablet befdd3a485a1426b8a3adf340d4eae4f) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.867648 9903 tablet_service.cc:1505] Processing CreateTablet for tablet 0d2aa6eb9d6541d5a19b30aa4b93cd1b (DEFAULT_TABLE table=default.Table_35621e756a6b4ccd8f4fae865ec4766e [id=3d392943433944a0952668a2e0d016dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.876981 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 217fbfb1f58f420f8f258b4178901370 (DEFAULT_TABLE table=default.Table_cd4849448781434f836c682ba134ce00 [id=c24b425ab9eb4bbcabcff356465c3127]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.878086 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 452) took 1795 ms. Trace:
I20251027 21:07:56.878732 10041 raft_consensus.cc:2243] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:56.886593 10041 raft_consensus.cc:2272] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:56.887065 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 07a9c3b7ef1548ec9d5ae9f20545f263 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:56.888523 9599 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 525) took 1499 ms. Trace:
I20251027 21:07:56.889717 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 622fa1b3ff134f2d8a087ba13a85e597 (DEFAULT_TABLE table=default.Table_8b0df27ef0ab4ed6afe3765a736f8f20 [id=dbc939496da54ddea766d18462c76342]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.889741 9609 tablet_service.cc:1505] Processing CreateTablet for tablet fdd0198d5047458aab8bf3ce08ccc0de (DEFAULT_TABLE table=default.Table_c59fd341300a4f2f90f1bbd06750d943 [id=83a5151706a64ece82d758b4ef1e2d9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.890213 10043 rpcz_store.cc:276] 1027 21:07:54.999534 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.999636 (+ 102us) service_pool.cc:225] Handling call
1027 21:07:56.787966 (+1788330us) ts_tablet_manager.cc:1151] Deleting tablet 27bd3b2a011648b2b8f2f92944513751
1027 21:07:56.787979 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.795133 (+ 7154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1689352}
W20251027 21:07:56.891033 9610 consensus_meta.cc:287] T 2f641bc4dbed4c0abd0a55b253352507 P 51902b6b61ff4d5584ff21896e847e14: Time spent flushing consensus metadata: real 0.538s user 0.003s sys 0.003s
I20251027 21:07:56.891798 9610 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 521) took 1514 ms. Trace:
I20251027 21:07:56.892243 9610 rpcz_store.cc:276] 1027 21:07:55.377389 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.829039 (+451650us) service_pool.cc:225] Handling call
1027 21:07:55.829426 (+ 387us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:55.829446 (+ 20us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.352526 (+523080us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:56.891392 (+538866us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:56.891720 (+ 328us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:56.892740 9611 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 530) took 1490 ms. Trace:
I20251027 21:07:56.892839 9611 rpcz_store.cc:276] 1027 21:07:55.402260 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.946226 (+543966us) service_pool.cc:225] Handling call
1027 21:07:55.946607 (+ 381us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:55.946640 (+ 33us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.640571 (+693931us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:56.892565 (+251994us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:56.892671 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:56.894507 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 453) took 1892 ms. Trace:
I20251027 21:07:56.894512 9612 tablet_service.cc:1505] Processing CreateTablet for tablet 0a86c3cc20fc4d149d3847ca281a4ec8 (DEFAULT_TABLE table=default.Table_e58c8bb0dfc540ba81698903abdfdbb0 [id=a2b99ffda2e34bbcb4547836a2524c6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.895717 9599 rpcz_store.cc:276] 1027 21:07:55.389468 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.879196 (+489728us) service_pool.cc:225] Handling call
1027 21:07:55.927881 (+ 48685us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:55.927900 (+ 19us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.626640 (+698740us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:56.888311 (+261671us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:56.888480 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:56.898820 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 622fa1b3ff134f2d8a087ba13a85e597. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.903731 10043 rpcz_store.cc:276] 1027 21:07:55.002011 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.002071 (+ 60us) service_pool.cc:225] Handling call
1027 21:07:56.890517 (+1888446us) ts_tablet_manager.cc:1151] Deleting tablet 533f34d4872444278477abcce89da9ec
1027 21:07:56.893591 (+ 3074us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.893954 (+ 363us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1791415}
I20251027 21:07:56.904394 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a86c3cc20fc4d149d3847ca281a4ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.905164 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdd0198d5047458aab8bf3ce08ccc0de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.905305 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 454) took 1902 ms. Trace:
I20251027 21:07:56.905424 10043 rpcz_store.cc:276] 1027 21:07:55.003112 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.003166 (+ 54us) service_pool.cc:225] Handling call
1027 21:07:56.905157 (+1901991us) ts_tablet_manager.cc:1151] Deleting tablet 61ba8d7df6fd45abb6f5231ee8c4ec38
1027 21:07:56.905166 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.905268 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1805585}
I20251027 21:07:56.912189 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 455) took 1900 ms. Trace:
I20251027 21:07:56.914451 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1aa5f7f76cb4a1ba15f016942f17067" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:56.915829 10043 rpcz_store.cc:276] 1027 21:07:55.008047 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.008096 (+ 49us) service_pool.cc:225] Handling call
1027 21:07:56.906333 (+1898237us) ts_tablet_manager.cc:1151] Deleting tablet d872cf3ef39042ff9a615b55e7195afa
1027 21:07:56.906341 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.908433 (+ 2092us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1806366}
I20251027 21:07:56.919381 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 470) took 1840 ms. Trace:
I20251027 21:07:56.924054 10043 rpcz_store.cc:276] 1027 21:07:55.076708 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.158500 (+ 81792us) service_pool.cc:225] Handling call
1027 21:07:56.916099 (+1757599us) ts_tablet_manager.cc:1151] Deleting tablet 4bbb69d62b9a41e3b88a5527f7549881
1027 21:07:56.916108 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.916956 (+ 848us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1757376}
I20251027 21:07:56.929138 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7aa8c538b6c849a28a28ffe706306891" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:56.929832 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 471) took 1850 ms. Trace:
I20251027 21:07:56.930516 10043 rpcz_store.cc:276] 1027 21:07:55.077666 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.159765 (+ 82099us) service_pool.cc:225] Handling call
1027 21:07:56.924346 (+1764581us) ts_tablet_manager.cc:1151] Deleting tablet 28a2e85021394f9d8a9ea524959b52a1
1027 21:07:56.924358 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.928113 (+ 3755us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1764410}
I20251027 21:07:56.930995 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 475) took 1848 ms. Trace:
I20251027 21:07:56.931116 10043 rpcz_store.cc:276] 1027 21:07:55.082590 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.165711 (+ 83121us) service_pool.cc:225] Handling call
1027 21:07:56.930800 (+1765089us) ts_tablet_manager.cc:1151] Deleting tablet ccaf5c2f8e544842880ba8ba27300ae6
1027 21:07:56.930812 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.930960 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1764875}
I20251027 21:07:56.931452 10043 tablet_replica.cc:333] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
W20251027 21:07:56.935447 9708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251027 21:07:56.931643 10043 raft_consensus.cc:2243] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:56.945372 10043 raft_consensus.cc:2272] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:56.958281 9607 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 531) took 1547 ms. Trace:
I20251027 21:07:56.959010 9607 rpcz_store.cc:276] 1027 21:07:55.402417 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.093116 (+690699us) service_pool.cc:225] Handling call
1027 21:07:56.093577 (+ 461us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:56.093601 (+ 24us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.667043 (+573442us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:56.950041 (+282998us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:56.950164 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:56.960223 9608 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 514) took 1600 ms. Trace:
I20251027 21:07:56.960572 10043 ts_tablet_manager.cc:1916] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:56.960930 9608 rpcz_store.cc:276] 1027 21:07:55.354025 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.820493 (+466468us) service_pool.cc:225] Handling call
1027 21:07:55.820872 (+ 379us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:55.820888 (+ 16us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.665969 (+845081us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:56.953906 (+287937us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:56.954016 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:56.971024 10041 ts_tablet_manager.cc:1916] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:56.973603 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 431313b4c80940239536aea40721b635 (DEFAULT_TABLE table=default.Table_6b452ea1c8f14d038a841d02b164c04a [id=37af64eacdfc43bfada9fc9097ca7555]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:56.974473 9860 leader_election.cc:343] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:56.825876 9711 service_pool.cc:138] CreateTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:40732 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 542)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 543)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 544)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 545)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 546)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 547)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 548)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 549)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 550)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 551)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 552)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 553)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 554)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 555)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 556)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 557)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 558)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 559)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 560)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 561)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 562)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 563)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 564)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 565)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 566)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 567)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 568)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 569)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 570)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 571)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 572)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 573)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 574)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 575)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 576)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 577)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 578)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 579)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 580)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 581)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 582)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 593)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 583)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 584)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 585)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 586)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 587)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 588)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 589)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 590)
I20251027 21:07:56.975407 9747 tablet_service.cc:1552] Processing DeleteTablet for tablet 5054bbc3e27243298b4df6b0a87af8be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.975648 9747 tablet_service.cc:1552] Processing DeleteTablet for tablet c8f0cd910b9d4daea287b3b192a81bd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.975836 9747 tablet_service.cc:1552] Processing DeleteTablet for tablet e6ed9d7a700446edafcd0daa02aa7ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.975525 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 9a2432f5c24b4bbca744d5f4806f2b58 (DEFAULT_TABLE table=default.Table_e31d4ff2c48e4eea9fe1f13da4fdf218 [id=ac996f557a39497e8f633b22be565e9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.976018 9747 tablet_service.cc:1552] Processing DeleteTablet for tablet c3b9a461567a48fcb4af514ecdf618b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.976171 9747 tablet_service.cc:1552] Processing DeleteTablet for tablet c9efffa9766e46aca8eb286fc2ad48d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.976349 9747 tablet_service.cc:1552] Processing DeleteTablet for tablet aabaa34787cf4d179822269b7f323314 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.976504 9747 tablet_service.cc:1552] Processing DeleteTablet for tablet e10ae34eaaf3476594813de8e96c7e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.976719 9747 tablet_service.cc:1552] Processing DeleteTablet for tablet 371aa8a57aee466ab01652d6061db4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:56.976863 9860 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.65:44531, user_credentials={real_user=slave}} blocked reactor thread for 59022.3us
I20251027 21:07:56.976888 9747 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d2c40f983e743b0b53f987b6a7ba437 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.977075 9747 tablet_service.cc:1552] Processing DeleteTablet for tablet 618f1657a10947fd98976820564c20b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:56.977362 9860 leader_election.cc:343] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:56.977423 9747 tablet_service.cc:1505] Processing CreateTablet for tablet a8c46c2a01f946aea8b85875115f7f37 (DEFAULT_TABLE table=default.Table_9b0fd27b43c842008090999fe7da9233 [id=62434317788348feb003dfac99359cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.977819 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c46c2a01f946aea8b85875115f7f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.978325 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet bc9322398d024e629a3c2aae7b6a77be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.978545 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 64e39cc210664c3d83eaf1fc576f83ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.978818 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet ed5619d295a246379b625f35692203e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.978933 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 113a5d30d4b3490187681383d4fef325 (DEFAULT_TABLE table=default.Table_1a51e839945b4912a5519bb1f21a6b75 [id=740df97b965b4a8facf0e11e2c38f020]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.979216 9753 tablet_service.cc:1505] Processing CreateTablet for tablet c6a189aa22d048858e9f30bf4276859d (DEFAULT_TABLE table=default.Table_abc7f8be9f414322ad9064340647399f [id=93edbbc1222b4ce990f6dc4e9a8cbe25]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.979305 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 113a5d30d4b3490187681383d4fef325. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.979730 9739 tablet_service.cc:1505] Processing CreateTablet for tablet a6edd6a5c4434f0497667b1a2771f885 (DEFAULT_TABLE table=default.Table_f0e8c68b9f1243bfb3584cca5b5f7002 [id=1d7f0c839ec546178f6cd39b81db07ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.980157 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6edd6a5c4434f0497667b1a2771f885. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.981240 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 622fa1b3ff134f2d8a087ba13a85e597. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.982400 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de2c033a064643b386ee84085cbdad64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.982460 9749 tablet_service.cc:1505] Processing CreateTablet for tablet b99232a6018a4b1a850da954c551d38d (DEFAULT_TABLE table=default.Table_cf9f3cddff8a486fbe76334c2fd833c5 [id=983747413ebc4e3ea8ccb0eac5ffff08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.982841 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b99232a6018a4b1a850da954c551d38d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.983721 10032 tablet_bootstrap.cc:654] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:56.986079 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a2432f5c24b4bbca744d5f4806f2b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.986296 10513 raft_consensus.cc:493] T 40bd202c1b834c949f9a5cee23b3486d P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fbd567f929a4f6d8e4d78feef7ed678)
I20251027 21:07:56.986446 10513 raft_consensus.cc:515] T 40bd202c1b834c949f9a5cee23b3486d P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:56.981338 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 55a545185c0140e1aa1ddf3fb6ceba9a (DEFAULT_TABLE table=default.Table_0f287ac2ad254338bd5151726cc40821 [id=c78c21c0d3fe49cfaf9b8c9c9eff3926]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.987107 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 0b4bf2b20e7c416aad123c6f96355cac (DEFAULT_TABLE table=default.Table_3c0c40b875fc45969c1a916c5545c4cb [id=699b0f7383ef48aeb27575efd44e5658]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.987236 10513 leader_election.cc:290] T 40bd202c1b834c949f9a5cee23b3486d P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:56.987471 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b4bf2b20e7c416aad123c6f96355cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.987488 9886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeb667b3b10746dbad351c4c123ffd97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.987668 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55a545185c0140e1aa1ddf3fb6ceba9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.990454 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a189aa22d048858e9f30bf4276859d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.995945 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 217fbfb1f58f420f8f258b4178901370. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.996485 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 9b9ffa18afa54f2c9ffd7b5a03a3a324 (DEFAULT_TABLE table=default.Table_ccc943e7fbc54e609e33b3a131a07c74 [id=1cc502193c3443898bc7eafa5ed15652]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:56.997624 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 431313b4c80940239536aea40721b635. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.997725 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b9ffa18afa54f2c9ffd7b5a03a3a324. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.998445 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e4d35f229bc46e1859ded7c8196ab9d with delete_type TABLET_DATA_DELETED (replaced by tablet 280ee23d80d64587b18811ff06340eee) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.998648 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet 9351a369dc1e4ac49f6ad419eab3a4c5 with delete_type TABLET_DATA_DELETED (replaced by tablet d5f4f3990da34f5e8f0385595f457427) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.998847 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet e477ca4b69c84327b04618d90406e19a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.999018 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet bbf92e29993b46389a69efd9e9a2341f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.999202 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet cb1686fc7c3a4996aa4462884d30c41f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.999337 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.999430 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 933b33aac5354778a5866d340fd34db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:56.999531 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet 78884c2be75f45f881f5424d3b3f4e08 with delete_type TABLET_DATA_DELETED (replaced by tablet 4996016747cd484cbda5f11e40e269b8) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.999668 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet 82366f8bf4154ffd87ef45e4a4d5f54b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:56.999843 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet 465be1b24d16457e834023a2b2194049 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:57.000010 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet a7d8d060916941a5b7e944e8b08a2882 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:57.000169 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:57.000464 9751 tablet_service.cc:1505] Processing CreateTablet for tablet a68bd9243b7541bbbc77014113e0106a (DEFAULT_TABLE table=default.Table_f5602dbcd14b4fb5ae7190b6a0bd65bc [id=078add1b91ad4cbfbe2215fe89b2334e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.000788 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a68bd9243b7541bbbc77014113e0106a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.002262 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f543e8bcf70c4acab9880ee01af85562. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.004307 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57de49dacafa497dad79ab96540b5888. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.009943 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ac1aed5f56844639b848f90c8eb5b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.010309 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87612b8bd8a04fd9b32f2df3f9e0d997. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.013302 10032 tablet_bootstrap.cc:492] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:57.013473 10032 ts_tablet_manager.cc:1403] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.330s user 0.018s sys 0.013s
I20251027 21:07:57.014286 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aef964f116ca4f98a5fcfa4d9f44437a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.014433 10032 raft_consensus.cc:359] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.014619 10032 raft_consensus.cc:385] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:57.014710 10032 raft_consensus.cc:740] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:57.015048 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d2aa6eb9d6541d5a19b30aa4b93cd1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.014950 10032 consensus_queue.cc:260] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.015586 10032 ts_tablet_manager.cc:1434] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:07:57.015990 10032 tablet_bootstrap.cc:492] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:57.017879 10032 tablet_bootstrap.cc:654] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:57.022534 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 25fbfcc28776483da266bb23c221f882 (DEFAULT_TABLE table=default.Table_cff1f8d7488047e3b7f4608d376e6f5b [id=b105395953eb4a9eaa6931668721d255]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.023072 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25fbfcc28776483da266bb23c221f882. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.026098 10031 tablet_bootstrap.cc:654] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:57.026237 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 622fa1b3ff134f2d8a087ba13a85e597 (DEFAULT_TABLE table=default.Table_8b0df27ef0ab4ed6afe3765a736f8f20 [id=dbc939496da54ddea766d18462c76342]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.026611 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 622fa1b3ff134f2d8a087ba13a85e597. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.024785 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 0a86c3cc20fc4d149d3847ca281a4ec8 (DEFAULT_TABLE table=default.Table_e58c8bb0dfc540ba81698903abdfdbb0 [id=a2b99ffda2e34bbcb4547836a2524c6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.033541 10032 tablet_bootstrap.cc:492] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:57.033703 10032 ts_tablet_manager.cc:1403] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20251027 21:07:57.034447 10032 raft_consensus.cc:359] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.034662 10032 raft_consensus.cc:385] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:57.034755 10032 raft_consensus.cc:740] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:57.035043 10032 consensus_queue.cc:260] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.035636 10032 ts_tablet_manager.cc:1434] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:57.037202 10032 tablet_bootstrap.cc:492] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:57.040730 9901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37cf73d154e24dfeb85a335d8a2d6f10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.045975 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40bd202c1b834c949f9a5cee23b3486d" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:57.046089 10032 tablet_bootstrap.cc:654] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:57.048164 9905 tablet_service.cc:1505] Processing CreateTablet for tablet eb2d1454d44243559964440cd49a44f5 (DEFAULT_TABLE table=default.Table_c9d359a62fe14203ae1d3b96f4b1bbf1 [id=2ee2f70ca45441fe95780652446e4713]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.048538 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb2d1454d44243559964440cd49a44f5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:57.051512 9568 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 67434.8us
I20251027 21:07:57.052769 9891 tablet_service.cc:1505] Processing CreateTablet for tablet be5a6d7c7c1747ce89d4e4d2359e7ecc (DEFAULT_TABLE table=default.Table_45692264ec9640ecbb170bfb95297d43 [id=cc855d4d4ef641f5a78dd5ba1a47fde4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.053534 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be5a6d7c7c1747ce89d4e4d2359e7ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.059469 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 53948dc94c334329bdf81f1eba94863c (DEFAULT_TABLE table=default.Table_09f7ae54c9544043afb9f2baef3ab51d [id=487916f3aa3d49a093bbbac97ebb2fe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.059837 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53948dc94c334329bdf81f1eba94863c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.061466 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 9239f74d725649cab5ea1b0077bccef3 (DEFAULT_TABLE table=default.Table_d5b9049d02b8474aafbf1e27d9f103ec [id=553cbebcfbca40f596b31399ef218181]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.061553 9887 tablet_service.cc:1505] Processing CreateTablet for tablet eaed784d81e74f9cb3514d3b55fdc0d1 (DEFAULT_TABLE table=default.Table_cc55a31e64194c35a8e42c5cdf11600d [id=6b6c154211034c95993d2c2f5495a71d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.061802 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9239f74d725649cab5ea1b0077bccef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.061913 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaed784d81e74f9cb3514d3b55fdc0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.062074 9889 tablet_service.cc:1505] Processing CreateTablet for tablet a9a1a3394d53419ebfe327a58388fc25 (DEFAULT_TABLE table=default.Table_fe401505cb114943863339485e5ff2f5 [id=2e3d80b78fdd457fa12ecb6eb34f6828]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.062467 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9a1a3394d53419ebfe327a58388fc25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.064990 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 1329ff0f8ebe4da8a6462a574ede98d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:57.066256 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 85ab6ac53e604ca0bfe051373606fe71 (DEFAULT_TABLE table=default.Table_fdca4795c6d042ec80cf84a378acb979 [id=a92ec852c916461eb59fba87bd1c81eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.071483 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85ab6ac53e604ca0bfe051373606fe71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.065227 9888 tablet_service.cc:1552] Processing DeleteTablet for tablet 69da799c886841a19002d3c7c5084e73 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:57.072639 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 83a040332f994055a9ecf4c66720b7e1 (DEFAULT_TABLE table=default.Table_e6e6313ef68948c79fc2f7d96afae64b [id=1297ccc0eaae40dcb7ac2ef9164bafc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.073063 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83a040332f994055a9ecf4c66720b7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.066540 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 8fc2d7d6601d4cc2849064f4cdddc90d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:57.058277 9608 tablet_service.cc:1505] Processing CreateTablet for tablet 959c48ed8a8e4d2189aa6546fdab4e7d (DEFAULT_TABLE table=default.Table_c58f3dadcca44b7d81025d6b5a7be405 [id=8eacacc3c28941baa04ebff88d8928c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.075314 9895 tablet_service.cc:1505] Processing CreateTablet for tablet e2ff7fc3c1fa4dbd9a5c38d3219b03fa (DEFAULT_TABLE table=default.Table_41ae08dcac5e4355b09bd4d15a35c229 [id=ec2694e19c8c4a698168c21a46b04e5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.075407 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 38f2133010b14aad88b550fb12c966a5 (DEFAULT_TABLE table=default.Table_0812ae8e053c4e6f913221ef573f618f [id=aaad36e806d140bdbbdd90ed29d945e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.075776 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2ff7fc3c1fa4dbd9a5c38d3219b03fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.063760 9594 tablet_service.cc:1505] Processing CreateTablet for tablet e24312671b4b4106a1790c61a5548a76 (DEFAULT_TABLE table=default.Table_aa03fc53fae14398b298f577a92d8b5b [id=f1e7ee19e2af41858faf94994bd6144b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.030125 9749 tablet_service.cc:1505] Processing CreateTablet for tablet fdd0198d5047458aab8bf3ce08ccc0de (DEFAULT_TABLE table=default.Table_c59fd341300a4f2f90f1bbd06750d943 [id=83a5151706a64ece82d758b4ef1e2d9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.075631 9603 tablet_service.cc:1505] Processing CreateTablet for tablet a12bf9c3d0b445dc92d54aa3059b834c (DEFAULT_TABLE table=default.Table_1a2deac4d0444b85b51757688f89b202 [id=71db9ec3df6a4f71aaaefb47e219929c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.034899 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a86c3cc20fc4d149d3847ca281a4ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.037452 9742 tablet_service.cc:1505] Processing CreateTablet for tablet d03918b19787449a88e0f17a925ce599 (DEFAULT_TABLE table=default.Table_b64e1c03b47345e2adc8d87940f6280e [id=c0d010fc3b2e4f87a2c5409ce2473cab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.042798 9743 tablet_service.cc:1505] Processing CreateTablet for tablet 53948dc94c334329bdf81f1eba94863c (DEFAULT_TABLE table=default.Table_09f7ae54c9544043afb9f2baef3ab51d [id=487916f3aa3d49a093bbbac97ebb2fe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.048647 9741 tablet_service.cc:1505] Processing CreateTablet for tablet 9239f74d725649cab5ea1b0077bccef3 (DEFAULT_TABLE table=default.Table_d5b9049d02b8474aafbf1e27d9f103ec [id=553cbebcfbca40f596b31399ef218181]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.042529 9746 tablet_service.cc:1505] Processing CreateTablet for tablet e24312671b4b4106a1790c61a5548a76 (DEFAULT_TABLE table=default.Table_aa03fc53fae14398b298f577a92d8b5b [id=f1e7ee19e2af41858faf94994bd6144b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.080339 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d03918b19787449a88e0f17a925ce599. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.082399 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a (DEFAULT_TABLE table=default.Table_db53bd5823b84458b7cd9564d5e9676e [id=3e816b6785f8495fa468922a28feee04]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.082751 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53948dc94c334329bdf81f1eba94863c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.083325 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 04a50ad9927e441faa3c57579c9cd215 (DEFAULT_TABLE table=default.Table_2ef7853cbd8b4f59a59cec83c91857c2 [id=8da438c490124663982f76f5b075126b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.086378 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9239f74d725649cab5ea1b0077bccef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.092972 10040 ts_tablet_manager.cc:1929] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:57.093186 10040 log.cc:1199] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/970fbf01a4c3406dafe1ea4cd005ea23
I20251027 21:07:57.093600 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e24312671b4b4106a1790c61a5548a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.093703 10040 ts_tablet_manager.cc:1950] T 970fbf01a4c3406dafe1ea4cd005ea23 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:57.094039 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 366) took 3654 ms. Trace:
I20251027 21:07:57.050686 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 959c48ed8a8e4d2189aa6546fdab4e7d (DEFAULT_TABLE table=default.Table_c58f3dadcca44b7d81025d6b5a7be405 [id=8eacacc3c28941baa04ebff88d8928c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.095443 9615 tablet_service.cc:1505] Processing CreateTablet for tablet a37ff8650247492a9c20a3b55300c5f6 (DEFAULT_TABLE table=default.Table_b230ee0b39fd44e0b9a956f480073963 [id=72addfdb5cfb4964a1eccbd639c04039]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.054028 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 38f2133010b14aad88b550fb12c966a5 (DEFAULT_TABLE table=default.Table_0812ae8e053c4e6f913221ef573f618f [id=aaad36e806d140bdbbdd90ed29d945e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.054653 9739 tablet_service.cc:1505] Processing CreateTablet for tablet 04a50ad9927e441faa3c57579c9cd215 (DEFAULT_TABLE table=default.Table_2ef7853cbd8b4f59a59cec83c91857c2 [id=8da438c490124663982f76f5b075126b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.055215 9744 tablet_service.cc:1505] Processing CreateTablet for tablet a12bf9c3d0b445dc92d54aa3059b834c (DEFAULT_TABLE table=default.Table_1a2deac4d0444b85b51757688f89b202 [id=71db9ec3df6a4f71aaaefb47e219929c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.058537 9737 tablet_service.cc:1505] Processing CreateTablet for tablet a37ff8650247492a9c20a3b55300c5f6 (DEFAULT_TABLE table=default.Table_b230ee0b39fd44e0b9a956f480073963 [id=72addfdb5cfb4964a1eccbd639c04039]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.073077 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a (DEFAULT_TABLE table=default.Table_db53bd5823b84458b7cd9564d5e9676e [id=3e816b6785f8495fa468922a28feee04]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.076079 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 33c5bc33130a424fb63e3f92a2cc49fc (DEFAULT_TABLE table=default.Table_fdf2451e0a2a48f9b902827e2a291e90 [id=6eae55ab35bf40bca718266926b1187f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.097395 10032 tablet_bootstrap.cc:492] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:57.097570 10032 ts_tablet_manager.cc:1403] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.060s user 0.002s sys 0.003s
I20251027 21:07:57.076570 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 1b7cf52a2ac249a0ac786ddc24acdae5 (DEFAULT_TABLE table=default.Table_919a2713d2c54825aa74493e188181e6 [id=af408a20521f4e2fbba065b502422579]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.077111 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 6e06ae559ca14b8bb0630e1571ca4852 (DEFAULT_TABLE table=default.Table_ea4ca66f1a144318a185c1324b2f3f61 [id=10a848ac0e3d4c1288be7fc15c1ce4b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.098526 10029 tablet_bootstrap.cc:492] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:57.098672 10029 ts_tablet_manager.cc:1403] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.439s user 0.006s sys 0.006s
I20251027 21:07:57.098800 9896 tablet_service.cc:1505] Processing CreateTablet for tablet cc97e55cfbcb41ec8c6c7410bbf14fb3 (DEFAULT_TABLE table=default.Table_e13c97413ddf417ea96834ba933b537b [id=e08d419154a44c54ad88bddcc5ecdbf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.099398 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc97e55cfbcb41ec8c6c7410bbf14fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.100311 9610 tablet_service.cc:1505] Processing CreateTablet for tablet d03918b19787449a88e0f17a925ce599 (DEFAULT_TABLE table=default.Table_b64e1c03b47345e2adc8d87940f6280e [id=c0d010fc3b2e4f87a2c5409ce2473cab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.103013 9886 tablet_service.cc:1505] Processing CreateTablet for tablet 9e9191649aa84b058abdc1477995ef67 (DEFAULT_TABLE table=default.Table_82f669b5326f44efa26969b7d9c7c3f9 [id=2f84c444434b4ba69526ab21585a27c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.103447 9886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e9191649aa84b058abdc1477995ef67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.053701 9736 tablet_service.cc:1505] Processing CreateTablet for tablet f543e8bcf70c4acab9880ee01af85562 (DEFAULT_TABLE table=default.Table_7d6df7f8ab8945a894c5984d320667b0 [id=4b12cf2789744c5b8d02c1a82b89e118]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.110082 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 4996016747cd484cbda5f11e40e269b8 (DEFAULT_TABLE table=default.Table_1376974f614548189208cb9cf0fd5353 [id=0eb078640c174f8fadb959306a35cada]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.112833 9901 tablet_service.cc:1505] Processing CreateTablet for tablet 280ee23d80d64587b18811ff06340eee (DEFAULT_TABLE table=default.Table_00deabd570914e54b7fa9c4fc4eb02c7 [id=ba8a1f1eb0d445cbaf7f263429085ee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.113102 10032 raft_consensus.cc:359] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.114977 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 1c3c89b7036b438cad79d0472619257e (DEFAULT_TABLE table=default.Table_563c35f040434ab5bd6eee6981ddb400 [id=58beac03856b46a9867d94c01541f20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.115754 10527 tablet.cc:1722] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:07:57.116997 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4996016747cd484cbda5f11e40e269b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.117717 9901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 280ee23d80d64587b18811ff06340eee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.119001 10032 raft_consensus.cc:385] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:57.121281 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c3c89b7036b438cad79d0472619257e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.148845 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 3b2c479d11794a6a99b5e05d3954ad54 (DEFAULT_TABLE table=default.Table_5639561ca4aa45e5bc117ecc76d943d9 [id=f17625d9439b4fa5be25d4aa46a1f479]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.150170 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdd0198d5047458aab8bf3ce08ccc0de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.154733 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38f2133010b14aad88b550fb12c966a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.154958 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b7cf52a2ac249a0ac786ddc24acdae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.154348 9747 tablet_service.cc:1505] Processing CreateTablet for tablet aef964f116ca4f98a5fcfa4d9f44437a (DEFAULT_TABLE table=default.Table_85b289c2c1174393803662453ba6ac8f [id=5f0e3f844ce7452f92e3bdda3be5c58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.156985 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04a50ad9927e441faa3c57579c9cd215. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.156581 10040 rpcz_store.cc:276] 1027 21:07:53.439708 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.439769 (+ 61us) service_pool.cc:225] Handling call
1027 21:07:56.804357 (+3364588us) ts_tablet_manager.cc:1151] Deleting tablet 970fbf01a4c3406dafe1ea4cd005ea23
1027 21:07:56.804366 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.952864 (+148498us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:57.092849 (+139985us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:57.094002 (+ 1153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3364400}
I20251027 21:07:57.101650 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 959c48ed8a8e4d2189aa6546fdab4e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.161334 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:57.161830 9296 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 345d3795a37b4a40b40967cfacf0f893 (table default.Table_ef79e4d09fb741429811f81e1efd90cb [id=93d6307da3a34f87a67018a06825cd15], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 345d3795a37b4a40b40967cfacf0f893 consensus state has no leader
I20251027 21:07:57.101763 9605 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 532) took 1672 ms. Trace:
I20251027 21:07:57.165892 9605 rpcz_store.cc:276] 1027 21:07:55.403683 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.180087 (+776404us) service_pool.cc:225] Handling call
1027 21:07:56.423403 (+243316us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:56.423432 (+ 29us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.920037 (+496605us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:57.075588 (+155551us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:57.076288 (+ 700us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:57.166374 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04a50ad9927e441faa3c57579c9cd215. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.166406 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet fde0980aa64c4496861a662f7ed3ef75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:57.166852 9605 tablet_service.cc:1505] Processing CreateTablet for tablet de2c033a064643b386ee84085cbdad64 (DEFAULT_TABLE table=default.Table_09602bd1d278472bb1ed05abe9d1def4 [id=9a86a04b859446a685bcc946e7331dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.167279 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de2c033a064643b386ee84085cbdad64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.101871 9602 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 519) took 1720 ms. Trace:
I20251027 21:07:57.168979 9602 rpcz_store.cc:276] 1027 21:07:55.371066 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.827488 (+456422us) service_pool.cc:225] Handling call
1027 21:07:55.830918 (+ 3430us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:55.830938 (+ 20us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.966595 (+1135657us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:57.091728 (+125133us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:57.091861 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:57.169399 9602 tablet_service.cc:1505] Processing CreateTablet for tablet eeb667b3b10746dbad351c4c123ffd97 (DEFAULT_TABLE table=default.Table_4aae3c9852244881bf5286450425cf5d [id=f500277871dc4f78ab54c95ad34ef1c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.169785 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeb667b3b10746dbad351c4c123ffd97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.172565 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38f2133010b14aad88b550fb12c966a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.101954 9595 tablet_service.cc:1505] Processing CreateTablet for tablet 33c5bc33130a424fb63e3f92a2cc49fc (DEFAULT_TABLE table=default.Table_fdf2451e0a2a48f9b902827e2a291e90 [id=6eae55ab35bf40bca718266926b1187f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.174985 9595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c5bc33130a424fb63e3f92a2cc49fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.105861 9604 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 538) took 1508 ms. Trace:
I20251027 21:07:57.106566 10358 raft_consensus.cc:493] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:57.177529 10358 raft_consensus.cc:515] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.178346 10358 leader_election.cc:290] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:57.178807 9604 rpcz_store.cc:276] 1027 21:07:55.557204 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.336023 (+778819us) service_pool.cc:225] Handling call
1027 21:07:56.424569 (+ 88546us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:56.424590 (+ 21us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.902090 (+477500us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:57.058492 (+156402us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:57.058623 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:57.179411 9604 tablet_service.cc:1505] Processing CreateTablet for tablet aef964f116ca4f98a5fcfa4d9f44437a (DEFAULT_TABLE table=default.Table_85b289c2c1174393803662453ba6ac8f [id=5f0e3f844ce7452f92e3bdda3be5c58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.179811 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aef964f116ca4f98a5fcfa4d9f44437a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.180480 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e24312671b4b4106a1790c61a5548a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.111485 9612 tablet_service.cc:1505] Processing CreateTablet for tablet 6e06ae559ca14b8bb0630e1571ca4852 (DEFAULT_TABLE table=default.Table_ea4ca66f1a144318a185c1324b2f3f61 [id=10a848ac0e3d4c1288be7fc15c1ce4b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.182754 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e06ae559ca14b8bb0630e1571ca4852. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.113667 9600 tablet_service.cc:1505] Processing CreateTablet for tablet deb8b54e5e0f42bca4330d2160e5524a (DEFAULT_TABLE table=default.Table_851f66cf4b1047879eed5fb768c07156 [id=743b2e05c3674d1fabb93aa073d0df40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.186019 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deb8b54e5e0f42bca4330d2160e5524a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.187086 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 933b33aac5354778a5866d340fd34db4 (DEFAULT_TABLE table=default.Table_7a9e545bc9374c8b986687268e421690 [id=a97617c2cd71489aaf45385d3cb1ca9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.113489 9613 tablet_service.cc:1505] Processing CreateTablet for tablet 1b7cf52a2ac249a0ac786ddc24acdae5 (DEFAULT_TABLE table=default.Table_919a2713d2c54825aa74493e188181e6 [id=af408a20521f4e2fbba065b502422579]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.114177 9614 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 537) took 1564 ms. Trace:
I20251027 21:07:57.187538 9614 rpcz_store.cc:276] 1027 21:07:55.549251 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.330962 (+781711us) service_pool.cc:225] Handling call
1027 21:07:56.424281 (+ 93319us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:56.424302 (+ 21us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.901551 (+477249us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:57.113264 (+211713us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:57.113867 (+ 603us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:57.187690 10032 raft_consensus.cc:740] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:57.187932 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 37cf73d154e24dfeb85a335d8a2d6f10 (DEFAULT_TABLE table=default.Table_5804b71f15924faf9b6c09ac13d09d3c [id=17af96dd9b7f4daabc64b44aaedfef5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.188071 10032 consensus_queue.cc:260] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.189476 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 572f1971c93643db92caa93740faa2e3 (DEFAULT_TABLE table=default.Table_b4fa791919944d64850f1344cb8f980f [id=e10ea5796aa24d95bf7ec0e75394cef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.189918 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b2c479d11794a6a99b5e05d3954ad54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.065550 10515 consensus_queue.cc:1048] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.138s
I20251027 21:07:57.121744 10029 raft_consensus.cc:359] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:57.195690 10029 raft_consensus.cc:385] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:57.195852 10029 raft_consensus.cc:740] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:57.196211 10029 consensus_queue.cc:260] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:57.196555 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 57de49dacafa497dad79ab96540b5888 (DEFAULT_TABLE table=default.Table_2a9fa2fa9615471691f07230a08e1399 [id=876a8959f347456a977578a8dd9e0645]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.196877 10029 ts_tablet_manager.cc:1434] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.098s user 0.001s sys 0.001s
I20251027 21:07:57.194267 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 572f1971c93643db92caa93740faa2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.194566 10032 ts_tablet_manager.cc:1434] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.091s user 0.003s sys 0.000s
I20251027 21:07:57.190701 9891 tablet_service.cc:1505] Processing CreateTablet for tablet d5f4f3990da34f5e8f0385595f457427 (DEFAULT_TABLE table=default.Table_3b49d647ca194ad9bf5524812c6837a2 [id=a12d4ab777c34431beaa8f87ddbc7135]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.197621 10032 tablet_bootstrap.cc:492] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:57.194243 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d13d00353b6740b5bed5a153936d2059" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:57.198469 9924 raft_consensus.cc:2468] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:57.201640 10029 tablet_bootstrap.cc:492] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:57.138378 9598 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 533) took 1732 ms. Trace:
I20251027 21:07:57.201841 9598 rpcz_store.cc:276] 1027 21:07:55.405100 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.299256 (+894156us) service_pool.cc:225] Handling call
1027 21:07:56.423146 (+123890us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:56.423174 (+ 28us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.919042 (+495868us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:57.137077 (+218035us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:57.137216 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:57.202359 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5f4f3990da34f5e8f0385595f457427. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.202556 9598 tablet_service.cc:1505] Processing CreateTablet for tablet 87612b8bd8a04fd9b32f2df3f9e0d997 (DEFAULT_TABLE table=default.Table_45436efc9fd041c0a813930bd895b3dd [id=25ffc58bf0a54adfa648f55c76f3df44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.203186 9568 leader_election.cc:304] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:07:57.203639 10358 raft_consensus.cc:2804] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251027 21:07:57.203728 10358 raft_consensus.cc:493] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:57.203819 10358 raft_consensus.cc:3060] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Advancing to term 2
I20251027 21:07:57.205119 10032 tablet_bootstrap.cc:654] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:57.206789 10358 raft_consensus.cc:515] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.209969 10358 leader_election.cc:290] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:57.211179 9923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d13d00353b6740b5bed5a153936d2059" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:07:57.211401 9923 raft_consensus.cc:3060] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Advancing to term 2
I20251027 21:07:57.214958 9923 raft_consensus.cc:2468] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 2.
I20251027 21:07:57.215444 9568 leader_election.cc:304] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:07:57.218932 10515 raft_consensus.cc:2804] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 2 FOLLOWER]: Leader election won for term 2
I20251027 21:07:57.219153 10515 raft_consensus.cc:697] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 2 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:07:57.219496 10515 consensus_queue.cc:237] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.138491 9611 tablet_service.cc:1505] Processing CreateTablet for tablet f543e8bcf70c4acab9880ee01af85562 (DEFAULT_TABLE table=default.Table_7d6df7f8ab8945a894c5984d320667b0 [id=4b12cf2789744c5b8d02c1a82b89e118]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.138653 10526 tablet.cc:1722] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:07:57.187456 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 933b33aac5354778a5866d340fd34db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.230249 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57de49dacafa497dad79ab96540b5888. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.230512 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37cf73d154e24dfeb85a335d8a2d6f10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.234131 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87612b8bd8a04fd9b32f2df3f9e0d997. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.237187 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b7cf52a2ac249a0ac786ddc24acdae5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:57.241876 9285 kernel_stack_watchdog.cc:198] Thread 9298 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 266ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f95716e3980 (unknown) at ??:0
@ 0x93d797 __sanitizer::internal_sched_yield() at ??:?
@ 0x8a2e4d __interceptor_pthread_create at ??:?
@ 0x7f956e0ccbe1 kudu::Thread::StartThread() at ??:0
@ 0xa75107 kudu::Thread::Create() at ??:?
@ 0x7f956e0eca2b kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f956e0e844d kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f956e0ed77f kudu::ThreadPool::Submit() at ??:0
@ 0x7f956e001c63 kudu::DnsResolver::RefreshAddressesAsync() at ??:0
@ 0x7f9574fb307a kudu::rpc::Proxy::RefreshDnsAndEnqueueRequest() at ??:0
@ 0x7f9574fb6244 kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f9574fb59ee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7f9574fa1a20 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f9574fa28a5 kudu::rpc::OutboundCall::SetFailed() at ??:0
@ 0x7f9574fa2153 kudu::rpc::OutboundCall::SetResponse() at ??:0
I20251027 21:07:57.248172 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a12bf9c3d0b445dc92d54aa3059b834c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.252342 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a37ff8650247492a9c20a3b55300c5f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.255234 10029 tablet_bootstrap.cc:654] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:57.256386 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f543e8bcf70c4acab9880ee01af85562. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.274305 10032 tablet_bootstrap.cc:492] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:57.281034 10032 ts_tablet_manager.cc:1403] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.082s user 0.010s sys 0.007s
I20251027 21:07:57.304577 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f543e8bcf70c4acab9880ee01af85562. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:57.305840 10469 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.054s sys 0.076s
W20251027 21:07:57.305943 10469 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.563s user 0.054s sys 0.076s
I20251027 21:07:57.306059 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 959c48ed8a8e4d2189aa6546fdab4e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.306404 10040 tablet_replica.cc:333] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:57.308524 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 9b9ffa18afa54f2c9ffd7b5a03a3a324
I20251027 21:07:57.309113 10040 raft_consensus.cc:2243] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:57.310065 10040 raft_consensus.cc:2272] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:57.310020 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d13d00353b6740b5bed5a153936d2059" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:57.310279 9771 raft_consensus.cc:2468] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:57.310468 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d13d00353b6740b5bed5a153936d2059" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:57.310652 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e06ae559ca14b8bb0630e1571ca4852. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.310638 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b9ffa18afa54f2c9ffd7b5a03a3a324 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:57.310900 10513 raft_consensus.cc:493] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:57.312291 9775 raft_consensus.cc:3060] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Advancing to term 2
I20251027 21:07:57.312311 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a37ff8650247492a9c20a3b55300c5f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.313447 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet b3abc5f78c4e41e29128e2d790319c75
I20251027 21:07:57.313868 9308 catalog_manager.cc:6719] ReplaceTablet: tablet b3abc5f78c4e41e29128e2d790319c75 deleted and replaced by tablet ccd419ff40a94f9587b024ec87a2517e
I20251027 21:07:57.244861 10043 ts_tablet_manager.cc:1929] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:57.313192 10537 raft_consensus.cc:493] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a4b8b31d47da48aaa386a8e64a5749c2)
I20251027 21:07:57.315222 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet b3abc5f78c4e41e29128e2d790319c75 with delete_type TABLET_DATA_DELETED (replaced by tablet ccd419ff40a94f9587b024ec87a2517e) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:57.315227 10040 ts_tablet_manager.cc:1916] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:57.317145 10041 ts_tablet_manager.cc:1929] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:57.314594 10032 raft_consensus.cc:359] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.317549 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d03918b19787449a88e0f17a925ce599. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.315665 10537 raft_consensus.cc:515] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:07:57.319064 10513 raft_consensus.cc:515] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:57.319684 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a12bf9c3d0b445dc92d54aa3059b834c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.318516 10032 raft_consensus.cc:385] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:57.320890 10032 raft_consensus.cc:740] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:57.323844 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1776590d8387494da5dfbd08db17f4b1" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:57.325882 9710 leader_election.cc:343] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:57.326033 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.326256 10513 leader_election.cc:290] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:57.326257 9743 tablet_service.cc:1505] Processing CreateTablet for tablet 933b33aac5354778a5866d340fd34db4 (DEFAULT_TABLE table=default.Table_7a9e545bc9374c8b986687268e421690 [id=a97617c2cd71489aaf45385d3cb1ca9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.326861 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aef964f116ca4f98a5fcfa4d9f44437a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.327960 9321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18b15927ce5744ad8ba81dda52725c11" candidate_term: 4 candidate_status { last_received { term: 3 index: 819 } } ignore_live_leader: false dest_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" is_pre_election: true
I20251027 21:07:57.328132 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 933b33aac5354778a5866d340fd34db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.335575 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c5bc33130a424fb63e3f92a2cc49fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.339039 10043 log.cc:1199] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/77797eea3e9c4f8099dc9d58f116200b
I20251027 21:07:57.339462 10043 ts_tablet_manager.cc:1950] T 77797eea3e9c4f8099dc9d58f116200b P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:57.340646 10029 tablet_bootstrap.cc:492] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:57.340812 10029 ts_tablet_manager.cc:1403] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.139s user 0.004s sys 0.000s
I20251027 21:07:57.341567 10029 raft_consensus.cc:359] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.341760 10029 raft_consensus.cc:385] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:57.341823 10029 raft_consensus.cc:740] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
W20251027 21:07:57.342414 9298 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.024s sys 0.094s
W20251027 21:07:57.342530 9298 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.542s user 0.024s sys 0.094s
W20251027 21:07:57.342615 9298 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.9.14.65:44531, user_credentials={real_user=slave}} blocked reactor thread for 542287us
I20251027 21:07:57.345842 10526 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:59712 (request call id 547) took 1403 ms. Trace:
I20251027 21:07:57.345953 10526 rpcz_store.cc:276] 1027 21:07:55.942717 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.435545 (+492828us) service_pool.cc:225] Handling call
1027 21:07:57.345808 (+910263us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:56.761288 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1027 21:07:56.900907 (+139619us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1027 21:07:56.900917 (+ 10us) alter_schema_op.cc:66] Acquired schema lock
1027 21:07:56.900925 (+ 8us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1027 21:07:56.900976 (+ 51us) alter_schema_op.cc:128] START. Timestamp: 1761599276900952
1027 21:07:56.900981 (+ 5us) op_driver.cc:348] REPLICATION: starting
1027 21:07:56.906484 (+ 5503us) log.cc:844] Serialized 173 byte log entry
1027 21:07:57.052304 (+145820us) op_driver.cc:464] REPLICATION: finished
1027 21:07:57.137797 (+ 85493us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1027 21:07:57.258529 (+120732us) log.cc:844] Serialized 5 byte log entry
1027 21:07:57.345220 (+ 86691us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:57.345520 (+ 300us) log.cc:844] Serialized 14 byte log entry
1027 21:07:57.345742 (+ 222us) alter_schema_op.cc:73] Released schema lock
1027 21:07:57.345771 (+ 29us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":85395,"prepare.queue_time_us":325385,"prepare.run_cpu_time_us":1001,"prepare.run_wall_time_us":145296,"raft.queue_time_us":860,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":266,"replication_time_us":151287,"thread_start_us":316518,"threads_started":2}]]}
I20251027 21:07:57.349332 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 37cf73d154e24dfeb85a335d8a2d6f10 (DEFAULT_TABLE table=default.Table_5804b71f15924faf9b6c09ac13d09d3c [id=17af96dd9b7f4daabc64b44aaedfef5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.349789 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37cf73d154e24dfeb85a335d8a2d6f10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.355979 9775 raft_consensus.cc:2468] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 2.
I20251027 21:07:57.325452 10032 consensus_queue.cc:260] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.357170 10032 ts_tablet_manager.cc:1434] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.071s user 0.007s sys 0.000s
I20251027 21:07:57.360311 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 57de49dacafa497dad79ab96540b5888 (DEFAULT_TABLE table=default.Table_2a9fa2fa9615471691f07230a08e1399 [id=876a8959f347456a977578a8dd9e0645]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.367841 10528 raft_consensus.cc:493] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:57.368001 10528 raft_consensus.cc:515] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:57.369294 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_349a03f3bc284c719a39a8bb95bbb526" } new_table_name: "default.Table_e4b781970438453eb7f7f4b885331bfb" modify_external_catalogs: true
I20251027 21:07:57.369019 10528 leader_election.cc:290] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:57.370249 10537 leader_election.cc:290] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a4b8b31d47da48aaa386a8e64a5749c2 (127.9.14.126:34057), 6e86ca5e83dd4df5b72f045e7b787e81 (127.9.14.124:45071)
I20251027 21:07:57.370409 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfce0e5c837743ea969374244163be3e" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:57.370761 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e2e73046f34a4cb98d2a4cb5a48c4ca7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:57.371259 9775 raft_consensus.cc:2468] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
W20251027 21:07:57.371316 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2e73046f34a4cb98d2a4cb5a48c4ca7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:57.371775 9566 leader_election.cc:304] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:07:57.371872 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_20d9ef0583ca492a85be3bf2cdf16b5b" } new_table_name: "default.Table_37d4aa59b4364a9ebcfe4096d3250e49" modify_external_catalogs: true
I20251027 21:07:57.372268 10528 raft_consensus.cc:2804] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:57.372406 10528 raft_consensus.cc:493] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:57.372483 10528 raft_consensus.cc:3060] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:57.372654 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_19e1b54d4b7f4d24b1de893d1229bcdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:57.373195 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19e1b54d4b7f4d24b1de893d1229bcdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:57.373232 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_562d4900bd0c4a43b93862a97e5d6561"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:57.373651 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_562d4900bd0c4a43b93862a97e5d6561 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:57.373998 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_004dd356c86a4140b824cdaefda61a23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:57.374405 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_004dd356c86a4140b824cdaefda61a23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:57.376677 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d3a044fd69ff47428850c4d429855c80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:57.377163 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3a044fd69ff47428850c4d429855c80 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:57.378589 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 87612b8bd8a04fd9b32f2df3f9e0d997 (DEFAULT_TABLE table=default.Table_45436efc9fd041c0a813930bd895b3dd [id=25ffc58bf0a54adfa648f55c76f3df44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.379187 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57de49dacafa497dad79ab96540b5888. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:57.381865 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 980eb6755aee4bcfb2ea44d84009021f consensus state has no leader
I20251027 21:07:57.382334 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e8162d3f2ed747b3802569f1e9ab2115"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:57.382369 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 0a58cd07e1bb4ffdbf39e7e16a46022a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a58cd07e1bb4ffdbf39e7e16a46022a already in progress: creating tablet
W20251027 21:07:57.382742 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8162d3f2ed747b3802569f1e9ab2115 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:57.383451 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 27bd3b2a011648b2b8f2f92944513751 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27bd3b2a011648b2b8f2f92944513751 already in progress: creating tablet
I20251027 21:07:57.383713 9739 tablet_service.cc:1505] Processing CreateTablet for tablet 0ac1aed5f56844639b848f90c8eb5b6c (DEFAULT_TABLE table=default.Table_0e8bd764e80e46a5b665da0e3e2b677e [id=288ecea80da64655ae6246da518b77b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.384047 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87612b8bd8a04fd9b32f2df3f9e0d997. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.384718 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ac1aed5f56844639b848f90c8eb5b6c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:57.385262 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 61ba8d7df6fd45abb6f5231ee8c4ec38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61ba8d7df6fd45abb6f5231ee8c4ec38 already in progress: creating tablet
W20251027 21:07:57.385490 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet d872cf3ef39042ff9a615b55e7195afa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d872cf3ef39042ff9a615b55e7195afa already in progress: creating tablet
I20251027 21:07:57.385550 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b126a056ef2c41b28a178327ef69e524" } new_table_name: "default.Table_92fa91c5b7c24ec2bd881c758ba33889" modify_external_catalogs: true
W20251027 21:07:57.385725 9298 catalog_manager.cc:4952] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 4bbb69d62b9a41e3b88a5527f7549881 because the tablet was not found. No further retry: Not found: Tablet not found: 4bbb69d62b9a41e3b88a5527f7549881
W20251027 21:07:57.386044 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 28a2e85021394f9d8a9ea524959b52a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28a2e85021394f9d8a9ea524959b52a1 already in progress: creating tablet
W20251027 21:07:57.386719 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet ccaf5c2f8e544842880ba8ba27300ae6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ccaf5c2f8e544842880ba8ba27300ae6 already in progress: creating tablet
I20251027 21:07:57.390633 10032 tablet_bootstrap.cc:492] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:07:57.391794 9568 leader_election.cc:343] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: cfce0e5c837743ea969374244163be3e
I20251027 21:07:57.392756 10032 tablet_bootstrap.cc:654] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:57.403492 10032 tablet_bootstrap.cc:492] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:57.403632 10032 ts_tablet_manager.cc:1403] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20251027 21:07:57.404438 10032 raft_consensus.cc:359] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.404620 10032 raft_consensus.cc:385] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:57.404726 10032 raft_consensus.cc:740] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:57.404994 10032 consensus_queue.cc:260] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.405597 10032 ts_tablet_manager.cc:1434] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:07:57.406008 10032 tablet_bootstrap.cc:492] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:57.342065 10029 consensus_queue.cc:260] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.407793 10029 ts_tablet_manager.cc:1434] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.067s user 0.000s sys 0.002s
I20251027 21:07:57.408087 10029 tablet_bootstrap.cc:492] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:57.409688 10029 tablet_bootstrap.cc:654] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:57.414557 10032 tablet_bootstrap.cc:654] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:57.414543 9595 tablet_service.cc:1505] Processing CreateTablet for tablet 0ac1aed5f56844639b848f90c8eb5b6c (DEFAULT_TABLE table=default.Table_0e8bd764e80e46a5b665da0e3e2b677e [id=288ecea80da64655ae6246da518b77b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.414939 9595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ac1aed5f56844639b848f90c8eb5b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.419024 9608 tablet_service.cc:1505] Processing CreateTablet for tablet 0d2aa6eb9d6541d5a19b30aa4b93cd1b (DEFAULT_TABLE table=default.Table_35621e756a6b4ccd8f4fae865ec4766e [id=3d392943433944a0952668a2e0d016dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.419370 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d2aa6eb9d6541d5a19b30aa4b93cd1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.422231 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 217fbfb1f58f420f8f258b4178901370 (DEFAULT_TABLE table=default.Table_cd4849448781434f836c682ba134ce00 [id=c24b425ab9eb4bbcabcff356465c3127]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.422588 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 217fbfb1f58f420f8f258b4178901370. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:57.425889 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c0e2772d66404d74891e0893c49a66e0 consensus state has no leader
I20251027 21:07:57.423846 9744 tablet_service.cc:1505] Processing CreateTablet for tablet 0d2aa6eb9d6541d5a19b30aa4b93cd1b (DEFAULT_TABLE table=default.Table_35621e756a6b4ccd8f4fae865ec4766e [id=3d392943433944a0952668a2e0d016dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.428392 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 217fbfb1f58f420f8f258b4178901370 (DEFAULT_TABLE table=default.Table_cd4849448781434f836c682ba134ce00 [id=c24b425ab9eb4bbcabcff356465c3127]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.428731 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 217fbfb1f58f420f8f258b4178901370. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.430775 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 84b9de82d55447bba2a897edb683e5c5 with delete_type TABLET_DATA_DELETED (replaced by tablet befdd3a485a1426b8a3adf340d4eae4f) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:57.434589 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet 84b9de82d55447bba2a897edb683e5c5 with delete_type TABLET_DATA_DELETED (replaced by tablet befdd3a485a1426b8a3adf340d4eae4f) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:57.434792 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet cfce0e5c837743ea969374244163be3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:57.435111 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 431313b4c80940239536aea40721b635 (DEFAULT_TABLE table=default.Table_6b452ea1c8f14d038a841d02b164c04a [id=37af64eacdfc43bfada9fc9097ca7555]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.435443 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 431313b4c80940239536aea40721b635. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.445003 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d2aa6eb9d6541d5a19b30aa4b93cd1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.445297 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet 07a9c3b7ef1548ec9d5ae9f20545f263 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:57.445638 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 9a2432f5c24b4bbca744d5f4806f2b58 (DEFAULT_TABLE table=default.Table_e31d4ff2c48e4eea9fe1f13da4fdf218 [id=ac996f557a39497e8f633b22be565e9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.445969 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a2432f5c24b4bbca744d5f4806f2b58. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:57.450096 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b83daa33ef0f45c6a78c70001e532ea2 consensus state has no leader
I20251027 21:07:57.451742 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 55a545185c0140e1aa1ddf3fb6ceba9a (DEFAULT_TABLE table=default.Table_0f287ac2ad254338bd5151726cc40821 [id=c78c21c0d3fe49cfaf9b8c9c9eff3926]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.452072 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55a545185c0140e1aa1ddf3fb6ceba9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.458293 9612 tablet_service.cc:1505] Processing CreateTablet for tablet eb2d1454d44243559964440cd49a44f5 (DEFAULT_TABLE table=default.Table_c9d359a62fe14203ae1d3b96f4b1bbf1 [id=2ee2f70ca45441fe95780652446e4713]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.458611 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb2d1454d44243559964440cd49a44f5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:57.458938 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93a31187eca14b8797737936a80e7292 consensus state has no leader
I20251027 21:07:57.459223 9304 master_service.cc:954] ReplaceTablet: received request to replace tablet 4debc3f13f3a42939b6b83f6523aab7c from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:57.459160 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_1595883e735448e2b56b64cccacc7219"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:57.459715 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1595883e735448e2b56b64cccacc7219 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:57.461838 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_09602bd1d278472bb1ed05abe9d1def4" } modify_external_catalogs: true
I20251027 21:07:57.462007 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_09602bd1d278472bb1ed05abe9d1def4" } modify_external_catalogs: true
I20251027 21:07:57.462450 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7ae5d7111eac45a7b3061078d4fc83d8" } modify_external_catalogs: true
I20251027 21:07:57.462589 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7ae5d7111eac45a7b3061078d4fc83d8" } modify_external_catalogs: true
I20251027 21:07:57.463971 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_09f7ae54c9544043afb9f2baef3ab51d" } new_table_name: "default.Table_98fe733c6d594a04929bb89536b6c68f" modify_external_catalogs: true
I20251027 21:07:57.464557 10032 tablet_bootstrap.cc:492] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:57.464704 10032 ts_tablet_manager.cc:1403] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.000s
I20251027 21:07:57.465440 9598 tablet_service.cc:1505] Processing CreateTablet for tablet be5a6d7c7c1747ce89d4e4d2359e7ecc (DEFAULT_TABLE table=default.Table_45692264ec9640ecbb170bfb95297d43 [id=cc855d4d4ef641f5a78dd5ba1a47fde4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.465488 10032 raft_consensus.cc:359] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.465699 10032 raft_consensus.cc:385] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:57.465698 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f21fe641a0574b80a1fe5551e551e96c" } modify_external_catalogs: true
I20251027 21:07:57.465764 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be5a6d7c7c1747ce89d4e4d2359e7ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.465780 10032 raft_consensus.cc:740] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:57.465830 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f21fe641a0574b80a1fe5551e551e96c" } modify_external_catalogs: true
I20251027 21:07:57.466028 10032 consensus_queue.cc:260] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.466377 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_61f42d2bf7b941cebdb4d20f78094243"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:57.466621 10032 ts_tablet_manager.cc:1434] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251027 21:07:57.466847 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61f42d2bf7b941cebdb4d20f78094243 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:57.466922 10032 tablet_bootstrap.cc:492] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:57.468779 10032 tablet_bootstrap.cc:654] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:57.469466 9604 tablet_service.cc:1505] Processing CreateTablet for tablet eaed784d81e74f9cb3514d3b55fdc0d1 (DEFAULT_TABLE table=default.Table_cc55a31e64194c35a8e42c5cdf11600d [id=6b6c154211034c95993d2c2f5495a71d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.469806 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaed784d81e74f9cb3514d3b55fdc0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.473958 9606 tablet_service.cc:1505] Processing CreateTablet for tablet a9a1a3394d53419ebfe327a58388fc25 (DEFAULT_TABLE table=default.Table_fe401505cb114943863339485e5ff2f5 [id=2e3d80b78fdd457fa12ecb6eb34f6828]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.477319 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_89e2320f599341e882a2d09c0985b220" } modify_external_catalogs: true
I20251027 21:07:57.477484 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_89e2320f599341e882a2d09c0985b220" } modify_external_catalogs: true
I20251027 21:07:57.480839 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9a1a3394d53419ebfe327a58388fc25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.483124 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 85ab6ac53e604ca0bfe051373606fe71 (DEFAULT_TABLE table=default.Table_fdca4795c6d042ec80cf84a378acb979 [id=a92ec852c916461eb59fba87bd1c81eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.483444 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85ab6ac53e604ca0bfe051373606fe71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.486307 9600 tablet_service.cc:1505] Processing CreateTablet for tablet e2ff7fc3c1fa4dbd9a5c38d3219b03fa (DEFAULT_TABLE table=default.Table_41ae08dcac5e4355b09bd4d15a35c229 [id=ec2694e19c8c4a698168c21a46b04e5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.486616 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2ff7fc3c1fa4dbd9a5c38d3219b03fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.487995 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 83a040332f994055a9ecf4c66720b7e1 (DEFAULT_TABLE table=default.Table_e6e6313ef68948c79fc2f7d96afae64b [id=1297ccc0eaae40dcb7ac2ef9164bafc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.488201 10040 ts_tablet_manager.cc:1929] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:57.488309 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83a040332f994055a9ecf4c66720b7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.488333 10040 log.cc:1199] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/8fcfa6b85e7b46d78a84947e9f11ae46
I20251027 21:07:57.488692 10040 ts_tablet_manager.cc:1950] T 8fcfa6b85e7b46d78a84947e9f11ae46 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:57.489032 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 369) took 4025 ms. Trace:
I20251027 21:07:57.489066 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet ccaf5c2f8e544842880ba8ba27300ae6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:57.489123 10040 rpcz_store.cc:276] 1027 21:07:53.464024 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.464098 (+ 74us) service_pool.cc:225] Handling call
1027 21:07:57.159333 (+3695235us) ts_tablet_manager.cc:1151] Deleting tablet 8fcfa6b85e7b46d78a84947e9f11ae46
1027 21:07:57.160859 (+ 1526us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:57.354077 (+193218us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:57.488092 (+134015us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:57.488983 (+ 891us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1218,"tablet-delete.queue_time_us":3694972}
I20251027 21:07:57.489550 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 368) took 4025 ms. Trace:
I20251027 21:07:57.489630 10040 rpcz_store.cc:276] 1027 21:07:53.463844 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.464104 (+ 260us) service_pool.cc:225] Handling call
1027 21:07:57.489396 (+4025292us) ts_tablet_manager.cc:1151] Deleting tablet da26c83976df40328cf9b11734fcfa44
1027 21:07:57.489405 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:57.489502 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4024928}
I20251027 21:07:57.490015 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 370) took 3954 ms. Trace:
I20251027 21:07:57.491219 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 8fc2d7d6601d4cc2849064f4cdddc90d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:57.491619 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 9e9191649aa84b058abdc1477995ef67 (DEFAULT_TABLE table=default.Table_82f669b5326f44efa26969b7d9c7c3f9 [id=2f84c444434b4ba69526ab21585a27c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.491920 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e9191649aa84b058abdc1477995ef67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.493121 9594 tablet_service.cc:1505] Processing CreateTablet for tablet cc97e55cfbcb41ec8c6c7410bbf14fb3 (DEFAULT_TABLE table=default.Table_e13c97413ddf417ea96834ba933b537b [id=e08d419154a44c54ad88bddcc5ecdbf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.493451 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc97e55cfbcb41ec8c6c7410bbf14fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.495072 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 1c3c89b7036b438cad79d0472619257e (DEFAULT_TABLE table=default.Table_563c35f040434ab5bd6eee6981ddb400 [id=58beac03856b46a9867d94c01541f20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.495388 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c3c89b7036b438cad79d0472619257e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.501708 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 507) took 2268 ms. Trace:
I20251027 21:07:57.501792 10043 rpcz_store.cc:276] 1027 21:07:55.233363 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.778476 (+545113us) service_pool.cc:225] Handling call
1027 21:07:56.931382 (+1152906us) ts_tablet_manager.cc:1151] Deleting tablet 77797eea3e9c4f8099dc9d58f116200b
1027 21:07:56.931392 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:57.134518 (+203126us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:57.244749 (+110231us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:57.501438 (+256689us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1152658}
I20251027 21:07:57.503687 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 512) took 2148 ms. Trace:
I20251027 21:07:57.503775 10043 rpcz_store.cc:276] 1027 21:07:55.353848 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.817748 (+463900us) service_pool.cc:225] Handling call
1027 21:07:57.502135 (+1684387us) ts_tablet_manager.cc:1151] Deleting tablet 874d5974698c448ab79f19812cb0e886
1027 21:07:57.502143 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:57.502250 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1684135}
I20251027 21:07:57.504366 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 515) took 2150 ms. Trace:
I20251027 21:07:57.504441 10043 rpcz_store.cc:276] 1027 21:07:55.354108 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.825071 (+470963us) service_pool.cc:225] Handling call
1027 21:07:57.504077 (+1679006us) ts_tablet_manager.cc:1151] Deleting tablet 479d2cc8b8ce4c26841f50eea7c0e09a
1027 21:07:57.504086 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:57.504160 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1678757}
W20251027 21:07:57.505125 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 874d5974698c448ab79f19812cb0e886 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 874d5974698c448ab79f19812cb0e886 already in progress: creating tablet
W20251027 21:07:57.505399 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 479d2cc8b8ce4c26841f50eea7c0e09a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 479d2cc8b8ce4c26841f50eea7c0e09a already in progress: creating tablet
I20251027 21:07:57.505599 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet 69da799c886841a19002d3c7c5084e73 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:57.507336 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53948dc94c334329bdf81f1eba94863c consensus state has no leader
I20251027 21:07:57.507845 9309 catalog_manager.cc:5649] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 2, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 2 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20251027 21:07:57.508258 9309 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet d13d00353b6740b5bed5a153936d2059 (table default.Table_6b471834498c4afeb57279278d45d9a9 [id=f2edcf33b0b74923b4ddbbc2419e14ca]). Expected version 1 got 0
I20251027 21:07:57.508512 9309 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 7f7ad06a15c04efbb701634e14904351 (table default.Table_318d41ccfb2c4d2c99bafcea875143a1 [id=2bc39b5bc02645ef830eee14e92707db]). Expected version 1 got 0
I20251027 21:07:57.317320 10041 log.cc:1199] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/ed5619d295a246379b625f35692203e4
I20251027 21:07:57.509366 10041 ts_tablet_manager.cc:1950] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:57.509855 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 567) took 1409 ms. Trace:
I20251027 21:07:57.509940 10041 rpcz_store.cc:276] 1027 21:07:56.100104 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.109701 (+ 9597us) service_pool.cc:225] Handling call
1027 21:07:56.853691 (+743990us) ts_tablet_manager.cc:1151] Deleting tablet ed5619d295a246379b625f35692203e4
1027 21:07:56.853701 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:56.994698 (+140997us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:57.317013 (+322315us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:57.509805 (+192792us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":719113}
I20251027 21:07:57.510638 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet ed5619d295a246379b625f35692203e4 (table default.Table_32f38256207d4fa79ed533151449afa7 [id=e3c04aba1dbf46928a0c36444dd69d02]) successfully deleted
I20251027 21:07:57.511387 9891 tablet_service.cc:1505] Processing CreateTablet for tablet b8390bd307784327bacc17d0c0dd43d2 (DEFAULT_TABLE table=default.Table_6acc073c9e6b45958ee3033e7f0695f3 [id=e158527c1bd242d09e946cad68424eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.511762 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8390bd307784327bacc17d0c0dd43d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.514474 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 5f6434c5ccd14bae97ebd000ae7b2fdb (DEFAULT_TABLE table=default.Table_9797869b2b084612acc50e375cd116f8 [id=9f91e0d8b6324200951cd3ed614f4b26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.522358 9738 tablet_service.cc:1505] Processing CreateTablet for tablet be5a6d7c7c1747ce89d4e4d2359e7ecc (DEFAULT_TABLE table=default.Table_45692264ec9640ecbb170bfb95297d43 [id=cc855d4d4ef641f5a78dd5ba1a47fde4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.514732 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 0f90e09e74b6495db1aa2df85de21181 (DEFAULT_TABLE table=default.Table_4cbc59c030d044d8a097385ed57a0b5f [id=83566d870a90461686826b2073b99f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.535089 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_400ccee830744304b52118f36ab774ff" } modify_external_catalogs: true
I20251027 21:07:57.851665 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be5a6d7c7c1747ce89d4e4d2359e7ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.851732 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_400ccee830744304b52118f36ab774ff" } modify_external_catalogs: true
I20251027 21:07:57.534016 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 280ee23d80d64587b18811ff06340eee (DEFAULT_TABLE table=default.Table_00deabd570914e54b7fa9c4fc4eb02c7 [id=ba8a1f1eb0d445cbaf7f263429085ee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.853470 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 280ee23d80d64587b18811ff06340eee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.676067 9594 tablet_service.cc:1505] Processing CreateTablet for tablet b8390bd307784327bacc17d0c0dd43d2 (DEFAULT_TABLE table=default.Table_6acc073c9e6b45958ee3033e7f0695f3 [id=e158527c1bd242d09e946cad68424eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.854421 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8390bd307784327bacc17d0c0dd43d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.626888 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b9ffa18afa54f2c9ffd7b5a03a3a324 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:57.738579 10541 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 21:07:57.358800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.14.124:45071 (local address 127.0.0.1:53524)
1027 21:07:57.395168 (+ 36368us) negotiation.cc:107] Waiting for socket to connect
1027 21:07:57.395189 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1027 21:07:57.398304 (+ 3115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 21:07:57.737616 (+339312us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":36259,"thread_start_us":36153,"threads_started":1}
I20251027 21:07:57.514844 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f6434c5ccd14bae97ebd000ae7b2fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.514441 9901 tablet_service.cc:1505] Processing CreateTablet for tablet 7427bbc47a8643c19fb543626c6aea2a (DEFAULT_TABLE table=default.Table_f21fe641a0574b80a1fe5551e551e96c [id=cf0655bc96d348c1b362025374ae2c7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.514468 9905 tablet_service.cc:1505] Processing CreateTablet for tablet befdd3a485a1426b8a3adf340d4eae4f (DEFAULT_TABLE table=default.Table_4755cbfcd1464ec1b6f0b909b99f0c3f [id=6b70bbbf477d4b788b3ffb10c9c7d7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.520570 9887 tablet_service.cc:1505] Processing CreateTablet for tablet ccd419ff40a94f9587b024ec87a2517e (DEFAULT_TABLE table=default.Table_019b51c3ff544dae94c53821dc97ab9f [id=b9930c5b513749ac905895ebca813733]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.521468 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 44804ae1c85749cb9286777e7b5c5707 (DEFAULT_TABLE table=default.Table_c6736558d45d4ba291581bb4b39a2208 [id=68e2a01e084a4a76aefe1b72c1f3b6bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.522126 9894 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 590) took 1033 ms. Trace:
W20251027 21:07:57.526867 9900 consensus_meta.cc:287] T 87612b8bd8a04fd9b32f2df3f9e0d997 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent flushing consensus metadata: real 0.513s user 0.003s sys 0.000s
I20251027 21:07:57.857556 9901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7427bbc47a8643c19fb543626c6aea2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.527491 9899 tablet_service.cc:1505] Processing CreateTablet for tablet bd6152f7ac8a4123872098955b8837ac (DEFAULT_TABLE table=default.Table_cf9164135a4b4ccc91f15ca37555a7c1 [id=1e0d20c5adfa4c52845b9ea7cf5d8037]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.528741 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 181def804539489e880b47b45023eeb5 (DEFAULT_TABLE table=default.Table_3afa18a27b0b442780b1a3182be6b779 [id=0eccee42dab64af2ab3b930b93efc8a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.532407 10041 tablet_replica.cc:333] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:57.532691 9903 tablet_service.cc:1505] Processing CreateTablet for tablet d3f22b8dbd1b436aa050c89b48833fd9 (DEFAULT_TABLE table=default.Table_4e25bcf24847404786773903009ac9f8 [id=736680f7a24d453db54fa806c1017ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.551680 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40bd202c1b834c949f9a5cee23b3486d" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:57.522864 9745 tablet_service.cc:1505] Processing CreateTablet for tablet eaed784d81e74f9cb3514d3b55fdc0d1 (DEFAULT_TABLE table=default.Table_cc55a31e64194c35a8e42c5cdf11600d [id=6b6c154211034c95993d2c2f5495a71d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.665338 9595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 600) took 1024 ms. Trace:
I20251027 21:07:57.539505 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 7427bbc47a8643c19fb543626c6aea2a
I20251027 21:07:57.570564 10411 raft_consensus.cc:493] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:57.858429 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaed784d81e74f9cb3514d3b55fdc0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.858441 10411 raft_consensus.cc:515] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.858911 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d898f3a5d8ad4819a75089ec91846c21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:57.859143 10411 leader_election.cc:290] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:57.542542 9310 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 7f7ad06a15c04efbb701634e14904351 (table default.Table_318d41ccfb2c4d2c99bafcea875143a1 [id=2bc39b5bc02645ef830eee14e92707db]). Expected version 1 got 0
W20251027 21:07:57.859500 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d898f3a5d8ad4819a75089ec91846c21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:57.859632 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccd419ff40a94f9587b024ec87a2517e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.681021 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 0f90e09e74b6495db1aa2df85de21181 (DEFAULT_TABLE table=default.Table_4cbc59c030d044d8a097385ed57a0b5f [id=83566d870a90461686826b2073b99f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.675915 9598 tablet_service.cc:1505] Processing CreateTablet for tablet 5f6434c5ccd14bae97ebd000ae7b2fdb (DEFAULT_TABLE table=default.Table_9797869b2b084612acc50e375cd116f8 [id=9f91e0d8b6324200951cd3ed614f4b26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.860877 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d985344dc40f440cab5f2b52c2ba4e27" } new_table_name: "default.Table_8f8eb3a5bb404ca59da6c90fea1abfb1" modify_external_catalogs: true
I20251027 21:07:57.637401 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet 1329ff0f8ebe4da8a6462a574ede98d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:57.861328 9367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 4 similar messages]
I20251027 21:07:57.573299 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 4eb30f4e4b094693a2c1088cfaa68ba3 (DEFAULT_TABLE table=default.Table_1595883e735448e2b56b64cccacc7219 [id=55b0c6dee5944db0a95112b0d6890280]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.862178 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eb30f4e4b094693a2c1088cfaa68ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.574302 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 06d81182b73e46019fc18c828045001f (DEFAULT_TABLE table=default.Table_e8162d3f2ed747b3802569f1e9ab2115 [id=b2c80efac3b846669503b0bd18f98815]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.656389 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 44804ae1c85749cb9286777e7b5c5707 (DEFAULT_TABLE table=default.Table_c6736558d45d4ba291581bb4b39a2208 [id=68e2a01e084a4a76aefe1b72c1f3b6bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.550608 9603 tablet_service.cc:1505] Processing CreateTablet for tablet 3b2c479d11794a6a99b5e05d3954ad54 (DEFAULT_TABLE table=default.Table_5639561ca4aa45e5bc117ecc76d943d9 [id=f17625d9439b4fa5be25d4aa46a1f479]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.862886 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06d81182b73e46019fc18c828045001f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.677444 9604 tablet_service.cc:1505] Processing CreateTablet for tablet befdd3a485a1426b8a3adf340d4eae4f (DEFAULT_TABLE table=default.Table_4755cbfcd1464ec1b6f0b909b99f0c3f [id=6b70bbbf477d4b788b3ffb10c9c7d7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.645478 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet b3abc5f78c4e41e29128e2d790319c75 with delete_type TABLET_DATA_DELETED (replaced by tablet ccd419ff40a94f9587b024ec87a2517e) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:57.863574 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44804ae1c85749cb9286777e7b5c5707. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.544679 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet d1f24d21bb954ac9b62ce458c6a22462
I20251027 21:07:57.864529 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 3b2c479d11794a6a99b5e05d3954ad54 (DEFAULT_TABLE table=default.Table_5639561ca4aa45e5bc117ecc76d943d9 [id=f17625d9439b4fa5be25d4aa46a1f479]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.864615 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5a7e103ca3a64375a49f7c4efdfb045a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:57.864863 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b2c479d11794a6a99b5e05d3954ad54. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:57.865061 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a7e103ca3a64375a49f7c4efdfb045a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:57.544884 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet de2c033a064643b386ee84085cbdad64
I20251027 21:07:57.523351 9736 tablet_service.cc:1505] Processing CreateTablet for tablet a9a1a3394d53419ebfe327a58388fc25 (DEFAULT_TABLE table=default.Table_fe401505cb114943863339485e5ff2f5 [id=2e3d80b78fdd457fa12ecb6eb34f6828]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.575935 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 292192b6d0d646de9144bbd6adbd4fbe (DEFAULT_TABLE table=default.Table_004dd356c86a4140b824cdaefda61a23 [id=3a572a109ef24e3ca40144e70129ddec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.857508 9894 rpcz_store.cc:276] 1027 21:07:56.488918 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.489878 (+ 960us) service_pool.cc:225] Handling call
1027 21:07:56.606083 (+116205us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:56.606109 (+ 26us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:56.989842 (+383733us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:57.376145 (+386303us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:57.522069 (+145924us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:57.866407 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9a1a3394d53419ebfe327a58388fc25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.866451 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292192b6d0d646de9144bbd6adbd4fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.866607 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_86b3f1d7a96f4c788e7ab443853fd5cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:57.866770 9894 tablet_service.cc:1505] Processing CreateTablet for tablet 356a04cc6b7444df9ae17c12d5bff727 (DEFAULT_TABLE table=default.Table_61f42d2bf7b941cebdb4d20f78094243 [id=6a7b389bf1d54986a4ad348f44196cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.867117 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 356a04cc6b7444df9ae17c12d5bff727. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:57.867095 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86b3f1d7a96f4c788e7ab443853fd5cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:57.523721 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 1329ff0f8ebe4da8a6462a574ede98d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:57.523990 9743 tablet_service.cc:1552] Processing DeleteTablet for tablet 1776590d8387494da5dfbd08db17f4b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 7aa8c538b6c849a28a28ffe706306891) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:57.545619 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 2532bf7190064443a898e47e843fd9ac
I20251027 21:07:57.524142 9737 tablet_service.cc:1552] Processing DeleteTablet for tablet 69da799c886841a19002d3c7c5084e73 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:57.524415 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 8fc2d7d6601d4cc2849064f4cdddc90d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:57.524839 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 85ab6ac53e604ca0bfe051373606fe71 (DEFAULT_TABLE table=default.Table_fdca4795c6d042ec80cf84a378acb979 [id=a92ec852c916461eb59fba87bd1c81eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.525498 9749 tablet_service.cc:1505] Processing CreateTablet for tablet e2ff7fc3c1fa4dbd9a5c38d3219b03fa (DEFAULT_TABLE table=default.Table_41ae08dcac5e4355b09bd4d15a35c229 [id=ec2694e19c8c4a698168c21a46b04e5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.525961 9741 tablet_service.cc:1505] Processing CreateTablet for tablet 9e9191649aa84b058abdc1477995ef67 (DEFAULT_TABLE table=default.Table_82f669b5326f44efa26969b7d9c7c3f9 [id=2f84c444434b4ba69526ab21585a27c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.526482 9751 tablet_service.cc:1505] Processing CreateTablet for tablet cc97e55cfbcb41ec8c6c7410bbf14fb3 (DEFAULT_TABLE table=default.Table_e13c97413ddf417ea96834ba933b537b [id=e08d419154a44c54ad88bddcc5ecdbf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.528803 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 9a2432f5c24b4bbca744d5f4806f2b58 (DEFAULT_TABLE table=default.Table_e31d4ff2c48e4eea9fe1f13da4fdf218 [id=ac996f557a39497e8f633b22be565e9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.529073 9747 tablet_service.cc:1505] Processing CreateTablet for tablet eb2d1454d44243559964440cd49a44f5 (DEFAULT_TABLE table=default.Table_c9d359a62fe14203ae1d3b96f4b1bbf1 [id=2ee2f70ca45441fe95780652446e4713]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.529378 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 431313b4c80940239536aea40721b635 (DEFAULT_TABLE table=default.Table_6b452ea1c8f14d038a841d02b164c04a [id=37af64eacdfc43bfada9fc9097ca7555]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.868534 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_352d903f119f41d3b405c925eaf2ebaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:57.529743 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 55a545185c0140e1aa1ddf3fb6ceba9a (DEFAULT_TABLE table=default.Table_0f287ac2ad254338bd5151726cc40821 [id=c78c21c0d3fe49cfaf9b8c9c9eff3926]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.533996 9744 tablet_service.cc:1505] Processing CreateTablet for tablet 83a040332f994055a9ecf4c66720b7e1 (DEFAULT_TABLE table=default.Table_e6e6313ef68948c79fc2f7d96afae64b [id=1297ccc0eaae40dcb7ac2ef9164bafc4]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:57.868944 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_352d903f119f41d3b405c925eaf2ebaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:57.577720 9886 tablet_service.cc:1505] Processing CreateTablet for tablet eec11e8fcb60425fb7fd742b6051fc2d (DEFAULT_TABLE table=default.Table_d3a044fd69ff47428850c4d429855c80 [id=e1789ed0ee49428e9829cbd26cc82169]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.652742 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 181def804539489e880b47b45023eeb5 (DEFAULT_TABLE table=default.Table_3afa18a27b0b442780b1a3182be6b779 [id=0eccee42dab64af2ab3b930b93efc8a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.869974 9886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eec11e8fcb60425fb7fd742b6051fc2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.869999 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85ab6ac53e604ca0bfe051373606fe71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.681366 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 7427bbc47a8643c19fb543626c6aea2a (DEFAULT_TABLE table=default.Table_f21fe641a0574b80a1fe5551e551e96c [id=cf0655bc96d348c1b362025374ae2c7c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:57.534694 10031 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.000s sys 0.009s
I20251027 21:07:57.681499 9609 tablet_service.cc:1505] Processing CreateTablet for tablet ccd419ff40a94f9587b024ec87a2517e (DEFAULT_TABLE table=default.Table_019b51c3ff544dae94c53821dc97ab9f [id=b9930c5b513749ac905895ebca813733]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:57.870174 10031 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.843s user 0.000s sys 0.009s
I20251027 21:07:57.589080 9607 tablet_service.cc:1505] Processing CreateTablet for tablet d5f4f3990da34f5e8f0385595f457427 (DEFAULT_TABLE table=default.Table_3b49d647ca194ad9bf5524812c6837a2 [id=a12d4ab777c34431beaa8f87ddbc7135]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.535912 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 4996016747cd484cbda5f11e40e269b8 (DEFAULT_TABLE table=default.Table_1376974f614548189208cb9cf0fd5353 [id=0eb078640c174f8fadb959306a35cada]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.536549 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 280ee23d80d64587b18811ff06340eee (DEFAULT_TABLE table=default.Table_00deabd570914e54b7fa9c4fc4eb02c7 [id=ba8a1f1eb0d445cbaf7f263429085ee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.603381 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 572f1971c93643db92caa93740faa2e3 (DEFAULT_TABLE table=default.Table_b4fa791919944d64850f1344cb8f980f [id=e10ea5796aa24d95bf7ec0e75394cef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.543097 9739 tablet_service.cc:1505] Processing CreateTablet for tablet 1c3c89b7036b438cad79d0472619257e (DEFAULT_TABLE table=default.Table_563c35f040434ab5bd6eee6981ddb400 [id=58beac03856b46a9867d94c01541f20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.490130 10040 rpcz_store.cc:276] 1027 21:07:53.535121 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.535176 (+ 55us) service_pool.cc:225] Handling call
1027 21:07:57.489840 (+3954664us) ts_tablet_manager.cc:1151] Deleting tablet 04c579bb008448e9baa841135b6dbc61
1027 21:07:57.489847 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:57.489970 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3952195}
I20251027 21:07:57.669476 9614 tablet_service.cc:1505] Processing CreateTablet for tablet bd6152f7ac8a4123872098955b8837ac (DEFAULT_TABLE table=default.Table_cf9164135a4b4ccc91f15ca37555a7c1 [id=1e0d20c5adfa4c52845b9ea7cf5d8037]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.553799 9613 tablet_service.cc:1505] Processing CreateTablet for tablet 4996016747cd484cbda5f11e40e269b8 (DEFAULT_TABLE table=default.Table_1376974f614548189208cb9cf0fd5353 [id=0eb078640c174f8fadb959306a35cada]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.543401 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 07a9c3b7ef1548ec9d5ae9f20545f263 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:57.648563 9608 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 601) took 1010 ms. Trace:
I20251027 21:07:57.668833 9612 tablet_service.cc:1505] Processing CreateTablet for tablet d3f22b8dbd1b436aa050c89b48833fd9 (DEFAULT_TABLE table=default.Table_4e25bcf24847404786773903009ac9f8 [id=736680f7a24d453db54fa806c1017ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.668267 10029 tablet_bootstrap.cc:492] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:57.540714 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 518) took 2145 ms. Trace:
I20251027 21:07:57.549005 10528 raft_consensus.cc:515] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:57.637245 10526 tablet.cc:1722] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:07:57.871888 10041 raft_consensus.cc:2243] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:57.872114 10041 raft_consensus.cc:2272] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:57.872671 10041 ts_tablet_manager.cc:1916] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:57.586617 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_cf9f3cddff8a486fbe76334c2fd833c5" } new_table_name: "default.Table_69ef4f8f04d042a18d248c08958ee9f7" modify_external_catalogs: true
I20251027 21:07:57.873108 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f90e09e74b6495db1aa2df85de21181. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.579130 9897 tablet_service.cc:1505] Processing CreateTablet for tablet b0540fe4fe8744f79ee5e4d8d88f7436 (DEFAULT_TABLE table=default.Table_e2e73046f34a4cb98d2a4cb5a48c4ca7 [id=6c60245e7b144353be21e56169b93f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.874311 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0540fe4fe8744f79ee5e4d8d88f7436. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.556118 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8fc2d7d6601d4cc2849064f4cdddc90d" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:57.874523 9900 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 613) took 1332 ms. Trace:
I20251027 21:07:57.874647 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6152f7ac8a4123872098955b8837ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.874953 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3f22b8dbd1b436aa050c89b48833fd9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:57.737612 9565 connection.cc:537] client connection to 127.9.14.124:45071 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251027 21:07:57.786819 10515 raft_consensus.cc:493] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:57.586644 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_93bb0a51ff97439e8ebcb08268bd2462"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:57.875509 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93bb0a51ff97439e8ebcb08268bd2462 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:57.580478 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 4ca7e79ef67d4d20bf1def6d533302a6 (DEFAULT_TABLE table=default.Table_2a89c7b51adb4448b74b750f3ec00c53 [id=fe61bfcc87d0495fa474890e62e2a8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.876624 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ca7e79ef67d4d20bf1def6d533302a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.556540 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78884c2be75f45f881f5424d3b3f4e08" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:57.556792 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1aa5f7f76cb4a1ba15f016942f17067" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:57.556871 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed5619d295a246379b625f35692203e4" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:57.556924 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69da799c886841a19002d3c7c5084e73" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:57.557116 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7aa8c538b6c849a28a28ffe706306891" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:57.557416 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0968c2f25b64f6685f38a999ee9d60e" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:57.556985 9711 leader_election.cc:343] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: c49775486ff24686b0f45c47e2757aba
I20251027 21:07:57.878729 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet befdd3a485a1426b8a3adf340d4eae4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.628768 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_581b993e984d49cd9178284ab5c6a500" } new_table_name: "default.Table_fa60bcd5277c4bf1bbf49d0e971340ba" modify_external_catalogs: true
I20251027 21:07:57.581707 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 0bd99431c0694098b41b79f7f89e69d7 (DEFAULT_TABLE table=default.Table_c9c1ea88906a4ee6bbd763eced368b50 [id=5cead33841a749918c161e2180c90a7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.582943 9895 tablet_service.cc:1505] Processing CreateTablet for tablet fc2cf41a7b474f7d86a46af76cef4100 (DEFAULT_TABLE table=default.Table_562d4900bd0c4a43b93862a97e5d6561 [id=09be1f34af4f4c27b33e97c55206b9fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.881456 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bd99431c0694098b41b79f7f89e69d7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:57.737785 9708 connection.cc:537] client connection to 127.9.14.124:45071 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251027 21:07:57.759443 9697 kernel_stack_watchdog.cc:198] Thread 9711 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 144ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f8d87682980 (unknown) at ??:0
@ 0x7f8d8767c9c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f8d812f8904 google::LogMessage::Flush() at ??:0
@ 0x7f8d812f8a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f8d8fe62443 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f8d8fe64fba kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f8d8fe64d7e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7f8d8af54e0f kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f8d8af549ee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7f8d8af40a20 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f8d8af40f6c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f8d8aedc46a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f8d8aedae26 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7f8d8aeed2f5 ev::base<>::method_thunk<>() at ??:0
I20251027 21:07:57.882416 9743 tablet_service.cc:1505] Processing CreateTablet for tablet 572f1971c93643db92caa93740faa2e3 (DEFAULT_TABLE table=default.Table_b4fa791919944d64850f1344cb8f980f [id=e10ea5796aa24d95bf7ec0e75394cef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.629177 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_8d267a7e37b447158fe1ab962b3283d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:57.882827 9737 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b9ffa18afa54f2c9ffd7b5a03a3a324 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:57.882952 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d267a7e37b447158fe1ab962b3283d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:57.882974 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet b3abc5f78c4e41e29128e2d790319c75 with delete_type TABLET_DATA_DELETED (replaced by tablet ccd419ff40a94f9587b024ec87a2517e) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:57.584362 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 719321644ea3479ea4529a8a5d048754 (DEFAULT_TABLE table=default.Table_19e1b54d4b7f4d24b1de893d1229bcdb [id=cf22877b5b594a3c871cb103df58e055]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:57.737372 9296 connection.cc:537] client connection to 127.9.14.124:45071 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251027 21:07:57.549065 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 601c74459d3a4a41b812fdcdf8b0a62c (table default.Table_0ac2c74fb86c48459bad366bd8efccd5 [id=7c97543d81eb4a1fa0c81678857179ba]) successfully deleted
W20251027 21:07:57.884078 9297 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 334994us
I20251027 21:07:57.634480 10556 raft_consensus.cc:493] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:57.637475 10527 tablet.cc:1722] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:07:57.677337 10032 tablet_bootstrap.cc:492] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
W20251027 21:07:57.737396 9858 connection.cc:537] client connection to 127.9.14.124:45071 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251027 21:07:57.884482 10032 ts_tablet_manager.cc:1403] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.418s user 0.025s sys 0.057s
I20251027 21:07:57.788548 10560 raft_consensus.cc:493] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251027 21:07:57.886027 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 2a4aa1fe05114c1287ca580c1f411d3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a4aa1fe05114c1287ca580c1f411d3f already in progress: creating tablet
I20251027 21:07:57.887435 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 181def804539489e880b47b45023eeb5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:57.889285 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 94f2dbf47a5c4458bee54b26f272156e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94f2dbf47a5c4458bee54b26f272156e already in progress: creating tablet
W20251027 21:07:57.889560 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 30ce703650c94c06b93bce976cb5d1fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30ce703650c94c06b93bce976cb5d1fd already in progress: creating tablet
W20251027 21:07:57.889781 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 30ce703650c94c06b93bce976cb5d1fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30ce703650c94c06b93bce976cb5d1fd already in progress: creating tablet
W20251027 21:07:57.890094 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet f34512059f0c47fc9af7db2645f86b40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f34512059f0c47fc9af7db2645f86b40 already in progress: creating tablet
W20251027 21:07:57.890353 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet f6789dc2f7cb4a3c826ac507d5c9a239 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6789dc2f7cb4a3c826ac507d5c9a239 already in progress: creating tablet
I20251027 21:07:57.891707 9900 rpcz_store.cc:276] 1027 21:07:56.541746 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.839366 (+297620us) service_pool.cc:225] Handling call
1027 21:07:56.840914 (+ 1548us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:56.840996 (+ 82us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:57.013768 (+172772us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:57.857706 (+843938us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:57.874480 (+ 16774us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:57.892046 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet 49d9e5c5960a4534a6603875006e6266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:57.891739 10560 raft_consensus.cc:515] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.891824 10556 raft_consensus.cc:515] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.893906 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 181def804539489e880b47b45023eeb5 (DEFAULT_TABLE table=default.Table_3afa18a27b0b442780b1a3182be6b779 [id=0eccee42dab64af2ab3b930b93efc8a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.894556 9711 leader_election.cc:304] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:57.894862 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 44804ae1c85749cb9286777e7b5c5707 (DEFAULT_TABLE table=default.Table_c6736558d45d4ba291581bb4b39a2208 [id=68e2a01e084a4a76aefe1b72c1f3b6bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.895412 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55a545185c0140e1aa1ddf3fb6ceba9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.896325 9900 tablet_service.cc:1552] Processing DeleteTablet for tablet 7427bbc47a8643c19fb543626c6aea2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:57.896301 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc2cf41a7b474f7d86a46af76cef4100. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.896360 10032 raft_consensus.cc:359] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.896832 9740 tablet_service.cc:1505] Processing CreateTablet for tablet bd6152f7ac8a4123872098955b8837ac (DEFAULT_TABLE table=default.Table_cf9164135a4b4ccc91f15ca37555a7c1 [id=1e0d20c5adfa4c52845b9ea7cf5d8037]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.897014 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfce0e5c837743ea969374244163be3e" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:57.897125 9748 tablet_service.cc:1505] Processing CreateTablet for tablet d5f4f3990da34f5e8f0385595f457427 (DEFAULT_TABLE table=default.Table_3b49d647ca194ad9bf5524812c6837a2 [id=a12d4ab777c34431beaa8f87ddbc7135]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.897639 9900 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bf9c2a78aac42528d67846dc58adc40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:57.897716 9771 raft_consensus.cc:3060] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:57.897852 9900 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b04349b15f3494bb0933db56b7b24bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:57.899629 10411 raft_consensus.cc:493] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:57.900889 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet f6249fc5c6174e8e81b9d02d598d5ebd (table default.Table_943417713b3b456ab4ca691d2fb168fd [id=1cb32eea25114c6dbcb3b28044036c08]) successfully deleted
W20251027 21:07:57.901274 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 588e2f58f07b4a32a78dffd5b92ceb12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 588e2f58f07b4a32a78dffd5b92ceb12 already in progress: creating tablet
W20251027 21:07:57.901513 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet f4187b2f66f24055b1df170f1062cc88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4187b2f66f24055b1df170f1062cc88 already in progress: creating tablet
W20251027 21:07:57.901729 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet bbb4116d1edd492893adfbf5fb3ddeef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbb4116d1edd492893adfbf5fb3ddeef already in progress: creating tablet
W20251027 21:07:57.901928 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet b719dc40d7e348608cdfc78401c19824 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b719dc40d7e348608cdfc78401c19824 already in progress: creating tablet
W20251027 21:07:57.832077 9285 kernel_stack_watchdog.cc:198] Thread 9297 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 214ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f95716e3980 (unknown) at ??:0
@ 0x7f95716dd9c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f956b359904 google::LogMessage::Flush() at ??:0
@ 0x7f956b359a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f9581ca7568 kudu::master::AsyncDeleteReplica::HandleResponse() at ??:0
@ 0x7f9581bfedfd kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f9581caab60 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f9581caa97e _ZNSt17_Function_handlerIFvvEZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7f9574fb5e0f kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f9574fb59ee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7f9574fa1a20 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f9574fa1f6c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f9574f3d46a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f9574f3be26 kudu::rpc::Connection::ReadHandler() at ??:0
I20251027 21:07:57.906102 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 371) took 4365 ms. Trace:
I20251027 21:07:57.906236 10040 rpcz_store.cc:276] 1027 21:07:53.540131 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.540873 (+ 742us) service_pool.cc:225] Handling call
1027 21:07:57.870595 (+4329722us) ts_tablet_manager.cc:1151] Deleting tablet 76e4fb2bea7f4a4696fe9f6bb892bccc
1027 21:07:57.905011 (+ 34416us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:57.905674 (+ 663us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4329476}
W20251027 21:07:57.905973 9367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.124:45071, user_credentials={real_user=slave}} blocked reactor thread for 44630.8us
I20251027 21:07:57.907079 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 367) took 4451 ms. Trace:
I20251027 21:07:57.907168 10040 rpcz_store.cc:276] 1027 21:07:53.455507 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.547724 (+ 92217us) service_pool.cc:225] Handling call
1027 21:07:57.906487 (+4358763us) ts_tablet_manager.cc:1151] Deleting tablet 4debc3f13f3a42939b6b83f6523aab7c
1027 21:07:57.906495 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:57.906573 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4358554}
I20251027 21:07:57.907765 10043 rpcz_store.cc:276] 1027 21:07:55.359273 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.827251 (+467978us) service_pool.cc:225] Handling call
1027 21:07:57.504712 (+1677461us) ts_tablet_manager.cc:1151] Deleting tablet 631ff72613734913818a3cad24f20dbc
1027 21:07:57.504720 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:57.504793 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1677244}
I20251027 21:07:57.909832 10515 raft_consensus.cc:515] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:57.909823 9595 rpcz_store.cc:276] 1027 21:07:56.638386 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.414274 (+775888us) service_pool.cc:225] Handling call
1027 21:07:57.414745 (+ 471us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:57.414775 (+ 30us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:57.534911 (+120136us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:57.663042 (+128131us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:57.663204 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:57.909803 10029 ts_tablet_manager.cc:1403] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.463s user 0.002s sys 0.002s
I20251027 21:07:57.910755 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f90e09e74b6495db1aa2df85de21181. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.909693 9608 rpcz_store.cc:276] 1027 21:07:56.638451 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.418819 (+780368us) service_pool.cc:225] Handling call
1027 21:07:57.419180 (+ 361us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:57.419211 (+ 31us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:57.529297 (+110086us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:57.648407 (+119110us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:57.648530 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:57.911718 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet ae124514964440c5b88b16ac9f8a2567 with delete_type TABLET_DATA_DELETED (replaced by tablet 01e9db9eaf8e41c6a2ee13b9ed127421) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:57.911405 10528 leader_election.cc:290] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:57.913566 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 4eb30f4e4b094693a2c1088cfaa68ba3 (DEFAULT_TABLE table=default.Table_1595883e735448e2b56b64cccacc7219 [id=55b0c6dee5944db0a95112b0d6890280]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:57.914429 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 1d64a13667194c12864fa388eb88a380 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d64a13667194c12864fa388eb88a380 already in progress: creating tablet
I20251027 21:07:57.919503 9602 tablet_service.cc:1505] Processing CreateTablet for tablet eec11e8fcb60425fb7fd742b6051fc2d (DEFAULT_TABLE table=default.Table_d3a044fd69ff47428850c4d429855c80 [id=e1789ed0ee49428e9829cbd26cc82169]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.919831 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44804ae1c85749cb9286777e7b5c5707. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.920235 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6bf9c2a78aac42528d67846dc58adc40" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:57.919772 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f22f013b25074d4db9c7f13caf5b9b42" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:57.921646 9608 tablet_service.cc:1505] Processing CreateTablet for tablet b0540fe4fe8744f79ee5e4d8d88f7436 (DEFAULT_TABLE table=default.Table_e2e73046f34a4cb98d2a4cb5a48c4ca7 [id=6c60245e7b144353be21e56169b93f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.922031 9595 tablet_service.cc:1505] Processing CreateTablet for tablet 292192b6d0d646de9144bbd6adbd4fbe (DEFAULT_TABLE table=default.Table_004dd356c86a4140b824cdaefda61a23 [id=3a572a109ef24e3ca40144e70129ddec]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:57.922645 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 631ff72613734913818a3cad24f20dbc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 631ff72613734913818a3cad24f20dbc already in progress: creating tablet
I20251027 21:07:57.923173 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e8c83b501d8426fa0cad6a89f9a0b9b" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:57.924208 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6152f7ac8a4123872098955b8837ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.924594 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f6434c5ccd14bae97ebd000ae7b2fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.925998 10515 leader_election.cc:290] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:57.924590 10029 raft_consensus.cc:359] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
W20251027 21:07:57.927750 9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.124:45071, user_credentials={real_user=slave}} blocked reactor thread for 43690.2us
W20251027 21:07:57.928483 9568 leader_election.cc:343] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: cfce0e5c837743ea969374244163be3e
W20251027 21:07:57.929683 9860 leader_election.cc:343] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:57.935204 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 970fbf01a4c3406dafe1ea4cd005ea23 (table default.Table_dddd93d5fb3c4727b64ded3ff6cc3382 [id=741c9ce04c9d47d9a016bce1f4403ec7]) successfully deleted
W20251027 21:07:57.936851 9296 consensus_peers.cc:597] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 -> Peer 6e86ca5e83dd4df5b72f045e7b787e81 (127.9.14.124:45071): Couldn't send request to peer 6e86ca5e83dd4df5b72f045e7b787e81. Status: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 21:07:57.937170 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 8fcfa6b85e7b46d78a84947e9f11ae46 (table default.Table_4723d51b934b4f6a90c790ee4bc7ea86 [id=e68b63cf420d49289a5bd1db81e1cbc1]) successfully deleted
W20251027 21:07:57.937377 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet da26c83976df40328cf9b11734fcfa44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da26c83976df40328cf9b11734fcfa44 already in progress: creating tablet
I20251027 21:07:57.939972 10032 raft_consensus.cc:385] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:57.940878 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 719321644ea3479ea4529a8a5d048754. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.941540 10556 leader_election.cc:290] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:57.941016 10411 raft_consensus.cc:515] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:57.945169 9900 tablet_service.cc:1552] Processing DeleteTablet for tablet d8e008c868d04281beba0e5b204eb69a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:57.949046 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 06d81182b73e46019fc18c828045001f (DEFAULT_TABLE table=default.Table_e8162d3f2ed747b3802569f1e9ab2115 [id=b2c80efac3b846669503b0bd18f98815]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:57.950400 10029 raft_consensus.cc:385] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:57.950723 10032 raft_consensus.cc:740] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:57.951197 9900 tablet_service.cc:1552] Processing DeleteTablet for tablet d1f24d21bb954ac9b62ce458c6a22462 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:57.950289 10560 leader_election.cc:290] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:57.955621 10411 leader_election.cc:290] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:57.955696 9900 tablet_service.cc:1552] Processing DeleteTablet for tablet de2c033a064643b386ee84085cbdad64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:57.957158 10032 consensus_queue.cc:260] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:57.957268 9860 leader_election.cc:343] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:07:57.961269 9860 leader_election.cc:343] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:57.962373 9900 tablet_service.cc:1552] Processing DeleteTablet for tablet 2532bf7190064443a898e47e843fd9ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:57.970561 10032 ts_tablet_manager.cc:1434] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.079s user 0.001s sys 0.001s
I20251027 21:07:57.972508 10032 tablet_bootstrap.cc:492] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:57.976637 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 520) took 2555 ms. Trace:
I20251027 21:07:57.976758 10043 rpcz_store.cc:276] 1027 21:07:55.374127 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.828802 (+454675us) service_pool.cc:225] Handling call
1027 21:07:57.908116 (+2079314us) ts_tablet_manager.cc:1151] Deleting tablet 37ae3b6bf590473e8e09092fc5f7df1b
1027 21:07:57.927016 (+ 18900us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:57.929206 (+ 2190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2079099}
W20251027 21:07:57.977198 9568 leader_election.cc:343] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: ae124514964440c5b88b16ac9f8a2567
W20251027 21:07:57.977622 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 37ae3b6bf590473e8e09092fc5f7df1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37ae3b6bf590473e8e09092fc5f7df1b already in progress: creating tablet
I20251027 21:07:57.977828 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 181def804539489e880b47b45023eeb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.982595 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccaf5c2f8e544842880ba8ba27300ae6" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:57.986721 10029 raft_consensus.cc:740] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:57.986989 10029 consensus_queue.cc:260] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:57.988024 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccd419ff40a94f9587b024ec87a2517e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.988752 10029 ts_tablet_manager.cc:1434] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.077s user 0.000s sys 0.002s
I20251027 21:07:57.989058 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 522) took 2599 ms. Trace:
I20251027 21:07:57.989162 10043 rpcz_store.cc:276] 1027 21:07:55.377541 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.857691 (+480150us) service_pool.cc:225] Handling call
1027 21:07:57.977026 (+2119335us) ts_tablet_manager.cc:1151] Deleting tablet 92a2393cff8044028c86c8cd2bccd930
1027 21:07:57.977038 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:57.977152 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2119110}
W20251027 21:07:57.989722 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 92a2393cff8044028c86c8cd2bccd930 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92a2393cff8044028c86c8cd2bccd930 already in progress: creating tablet
I20251027 21:07:57.995620 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 534) took 2487 ms. Trace:
I20251027 21:07:57.995720 10043 rpcz_store.cc:276] 1027 21:07:55.504814 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.322985 (+818171us) service_pool.cc:225] Handling call
1027 21:07:57.989436 (+1666451us) ts_tablet_manager.cc:1151] Deleting tablet 95f9595784df434dbb0dd29d9a05511e
1027 21:07:57.990894 (+ 1458us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:57.992519 (+ 1625us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1565667}
W20251027 21:07:57.996381 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 95f9595784df434dbb0dd29d9a05511e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95f9595784df434dbb0dd29d9a05511e already in progress: creating tablet
I20251027 21:07:57.996649 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06d81182b73e46019fc18c828045001f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:57.999032 10029 tablet_bootstrap.cc:492] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:57.999640 10032 tablet_bootstrap.cc:654] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:58.004878 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4996016747cd484cbda5f11e40e269b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.010088 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 373) took 4234 ms. Trace:
I20251027 21:07:58.010232 10040 rpcz_store.cc:276] 1027 21:07:53.672550 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.672730 (+ 180us) service_pool.cc:225] Handling call
1027 21:07:57.907379 (+4234649us) ts_tablet_manager.cc:1151] Deleting tablet 59b6818d29624da5a49dd1a7d20994c6
1027 21:07:57.907387 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:57.907470 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4234437}
I20251027 21:07:58.010640 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 431313b4c80940239536aea40721b635. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.010663 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 535) took 2457 ms. Trace:
I20251027 21:07:58.010743 10043 rpcz_store.cc:276] 1027 21:07:55.540995 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.325955 (+784960us) service_pool.cc:225] Handling call
1027 21:07:57.995989 (+1670034us) ts_tablet_manager.cc:1151] Deleting tablet 11627a84ff9b478c87c323617a09570b
1027 21:07:57.996000 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:57.998707 (+ 2707us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1571968}
W20251027 21:07:58.011351 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 11627a84ff9b478c87c323617a09570b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11627a84ff9b478c87c323617a09570b already in progress: creating tablet
I20251027 21:07:58.011942 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 374) took 4337 ms. Trace:
I20251027 21:07:58.012038 10040 rpcz_store.cc:276] 1027 21:07:53.672679 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.672985 (+ 306us) service_pool.cc:225] Handling call
1027 21:07:58.010468 (+4337483us) ts_tablet_manager.cc:1151] Deleting tablet cb95618d9fd54a4f8121b43e717d5616
1027 21:07:58.010479 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.010595 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4337311}
I20251027 21:07:58.012825 10031 tablet_bootstrap.cc:492] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:58.012897 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae124514964440c5b88b16ac9f8a2567" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:58.012969 10031 ts_tablet_manager.cc:1403] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 1.271s user 0.002s sys 0.013s
I20251027 21:07:58.014114 10031 raft_consensus.cc:359] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.014295 10031 raft_consensus.cc:385] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.014619 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 280ee23d80d64587b18811ff06340eee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.015388 10031 raft_consensus.cc:740] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:58.015604 10031 consensus_queue.cc:260] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.015784 9771 raft_consensus.cc:2468] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:58.019330 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 375) took 4337 ms. Trace:
I20251027 21:07:58.019428 10040 rpcz_store.cc:276] 1027 21:07:53.675561 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.675606 (+ 45us) service_pool.cc:225] Handling call
1027 21:07:58.012657 (+4337051us) ts_tablet_manager.cc:1151] Deleting tablet 9861a0efcba64141aae450663c5d83de
1027 21:07:58.012665 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.012743 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4336876}
I20251027 21:07:58.020025 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3f22b8dbd1b436aa050c89b48833fd9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:58.027477 9566 leader_election.cc:343] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.027614 9566 leader_election.cc:304] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:58.028725 10515 raft_consensus.cc:2749] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:58.028647 9566 leader_election.cc:304] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:07:58.029255 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 539) took 2453 ms. Trace:
I20251027 21:07:58.029748 10515 raft_consensus.cc:2804] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:58.029901 10515 raft_consensus.cc:697] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:07:58.030860 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 376) took 4319 ms. Trace:
I20251027 21:07:58.030874 10515 consensus_queue.cc:237] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.030951 10040 rpcz_store.cc:276] 1027 21:07:53.700493 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.700540 (+ 47us) service_pool.cc:225] Handling call
1027 21:07:58.019632 (+4319092us) ts_tablet_manager.cc:1151] Deleting tablet d52d71f4976143e4a514350621ebe309
1027 21:07:58.019640 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.019717 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4318882}
I20251027 21:07:58.031237 10040 tablet_replica.cc:333] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:58.031426 10040 raft_consensus.cc:2243] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:58.031594 10040 raft_consensus.cc:2272] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:58.031632 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5f4f3990da34f5e8f0385595f457427. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.032126 10040 ts_tablet_manager.cc:1916] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:58.032109 10043 rpcz_store.cc:276] 1027 21:07:55.557334 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.345981 (+788647us) service_pool.cc:225] Handling call
1027 21:07:58.010967 (+1664986us) ts_tablet_manager.cc:1151] Deleting tablet 5054bbc3e27243298b4df6b0a87af8be
1027 21:07:58.010975 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.011057 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1586171}
W20251027 21:07:58.032697 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 5054bbc3e27243298b4df6b0a87af8be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5054bbc3e27243298b4df6b0a87af8be already in progress: creating tablet
I20251027 21:07:58.033794 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 540) took 2474 ms. Trace:
I20251027 21:07:58.033885 10043 rpcz_store.cc:276] 1027 21:07:55.557588 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.361403 (+803815us) service_pool.cc:225] Handling call
1027 21:07:58.032355 (+1670952us) ts_tablet_manager.cc:1151] Deleting tablet bc9322398d024e629a3c2aae7b6a77be
1027 21:07:58.032365 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.032452 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1605489}
W20251027 21:07:58.034471 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet bc9322398d024e629a3c2aae7b6a77be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc9322398d024e629a3c2aae7b6a77be already in progress: creating tablet
I20251027 21:07:58.034827 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 545) took 2305 ms. Trace:
I20251027 21:07:58.034916 10043 rpcz_store.cc:276] 1027 21:07:55.728742 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.435151 (+706409us) service_pool.cc:225] Handling call
1027 21:07:58.034081 (+1598930us) ts_tablet_manager.cc:1151] Deleting tablet aabaa34787cf4d179822269b7f323314
1027 21:07:58.034089 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.034187 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1598726}
I20251027 21:07:58.035252 10041 ts_tablet_manager.cc:1929] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:58.035372 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 572f1971c93643db92caa93740faa2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.035408 10041 log.cc:1199] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/f1aa5f7f76cb4a1ba15f016942f17067
W20251027 21:07:58.035436 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet aabaa34787cf4d179822269b7f323314 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aabaa34787cf4d179822269b7f323314 already in progress: creating tablet
I20251027 21:07:58.035858 10041 ts_tablet_manager.cc:1950] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:58.036128 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eb30f4e4b094693a2c1088cfaa68ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.036401 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7427bbc47a8643c19fb543626c6aea2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.036473 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 577) took 1801 ms. Trace:
I20251027 21:07:58.036612 10041 rpcz_store.cc:276] 1027 21:07:56.237694 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.237842 (+ 148us) service_pool.cc:225] Handling call
1027 21:07:57.532334 (+1294492us) ts_tablet_manager.cc:1151] Deleting tablet f1aa5f7f76cb4a1ba15f016942f17067
1027 21:07:57.532342 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:57.943299 (+410957us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.035087 (+ 91788us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.036426 (+ 1339us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13718,"tablet-delete.queue_time_us":1284473}
I20251027 21:07:58.036978 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b2c479d11794a6a99b5e05d3954ad54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.037133 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet f1aa5f7f76cb4a1ba15f016942f17067 (table default.Table_3e0f707f8e2844bfaf937150b747a935 [id=d7dcec1e03494fd5a219a97ea8b1ab1d]) successfully deleted
I20251027 21:07:58.037606 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 578) took 1793 ms. Trace:
I20251027 21:07:58.037747 10041 rpcz_store.cc:276] 1027 21:07:56.243641 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.243879 (+ 238us) service_pool.cc:225] Handling call
1027 21:07:58.037433 (+1793554us) ts_tablet_manager.cc:1151] Deleting tablet bbf92e29993b46389a69efd9e9a2341f
1027 21:07:58.037446 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.037560 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1787021}
I20251027 21:07:58.038100 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet befdd3a485a1426b8a3adf340d4eae4f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:58.038297 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet bbf92e29993b46389a69efd9e9a2341f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbf92e29993b46389a69efd9e9a2341f already in progress: creating tablet
I20251027 21:07:58.038713 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 572f1971c93643db92caa93740faa2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.038733 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 576) took 1805 ms. Trace:
I20251027 21:07:58.038861 10041 rpcz_store.cc:276] 1027 21:07:56.233718 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.237900 (+ 4182us) service_pool.cc:225] Handling call
1027 21:07:58.038563 (+1800663us) ts_tablet_manager.cc:1151] Deleting tablet e477ca4b69c84327b04618d90406e19a
1027 21:07:58.038575 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.038691 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1787973}
I20251027 21:07:58.036048 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 546) took 2303 ms. Trace:
W20251027 21:07:58.039322 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet e477ca4b69c84327b04618d90406e19a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e477ca4b69c84327b04618d90406e19a already in progress: creating tablet
I20251027 21:07:58.039322 10043 rpcz_store.cc:276] 1027 21:07:55.732604 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.435377 (+702773us) service_pool.cc:225] Handling call
1027 21:07:58.035248 (+1599871us) ts_tablet_manager.cc:1151] Deleting tablet e10ae34eaaf3476594813de8e96c7e77
1027 21:07:58.035258 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.035800 (+ 542us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1599718}
I20251027 21:07:58.039726 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 575) took 1812 ms. Trace:
I20251027 21:07:58.039860 10041 rpcz_store.cc:276] 1027 21:07:56.232917 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.237977 (+ 5060us) service_pool.cc:225] Handling call
1027 21:07:58.039576 (+1801599us) ts_tablet_manager.cc:1151] Deleting tablet 9351a369dc1e4ac49f6ad419eab3a4c5
1027 21:07:58.039587 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.039682 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1763592}
I20251027 21:07:58.040472 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 541) took 2482 ms. Trace:
W20251027 21:07:58.151122 9367 leader_election.cc:336] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6e86ca5e83dd4df5b72f045e7b787e81 (127.9.14.124:45071): Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: connect: Connection refused (error 111)
I20251027 21:07:58.046898 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb2d1454d44243559964440cd49a44f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.203258 9367 leader_election.cc:304] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18b15927ce5744ad8ba81dda52725c11; no voters: 6e86ca5e83dd4df5b72f045e7b787e81, a4b8b31d47da48aaa386a8e64a5749c2
I20251027 21:07:58.080013 9752 tablet_service.cc:1505] Processing CreateTablet for tablet b8390bd307784327bacc17d0c0dd43d2 (DEFAULT_TABLE table=default.Table_6acc073c9e6b45958ee3033e7f0695f3 [id=e158527c1bd242d09e946cad68424eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.080812 10579 raft_consensus.cc:2749] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:58.085937 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 5f6434c5ccd14bae97ebd000ae7b2fdb (DEFAULT_TABLE table=default.Table_9797869b2b084612acc50e375cd116f8 [id=9f91e0d8b6324200951cd3ed614f4b26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.081494 9738 tablet_service.cc:1505] Processing CreateTablet for tablet d3f22b8dbd1b436aa050c89b48833fd9 (DEFAULT_TABLE table=default.Table_4e25bcf24847404786773903009ac9f8 [id=736680f7a24d453db54fa806c1017ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.092695 9736 tablet_service.cc:1505] Processing CreateTablet for tablet befdd3a485a1426b8a3adf340d4eae4f (DEFAULT_TABLE table=default.Table_4755cbfcd1464ec1b6f0b909b99f0c3f [id=6b70bbbf477d4b788b3ffb10c9c7d7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.093760 9743 tablet_service.cc:1505] Processing CreateTablet for tablet 0f90e09e74b6495db1aa2df85de21181 (DEFAULT_TABLE table=default.Table_4cbc59c030d044d8a097385ed57a0b5f [id=83566d870a90461686826b2073b99f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.095153 10031 ts_tablet_manager.cc:1434] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.082s user 0.000s sys 0.013s
I20251027 21:07:58.108623 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 7427bbc47a8643c19fb543626c6aea2a (DEFAULT_TABLE table=default.Table_f21fe641a0574b80a1fe5551e551e96c [id=cf0655bc96d348c1b362025374ae2c7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.115165 9745 tablet_service.cc:1505] Processing CreateTablet for tablet ccd419ff40a94f9587b024ec87a2517e (DEFAULT_TABLE table=default.Table_019b51c3ff544dae94c53821dc97ab9f [id=b9930c5b513749ac905895ebca813733]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:58.173771 9711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 614674us
W20251027 21:07:58.206048 9711 leader_election.cc:343] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 1e5f5b57f5d948f3b2352451ccaacb09
I20251027 21:07:58.209019 9711 leader_election.cc:304] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
W20251027 21:07:58.223649 9367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.124:45071, user_credentials={real_user=slave}} blocked reactor thread for 72202.7us
I20251027 21:07:58.040570 10043 rpcz_store.cc:276] 1027 21:07:55.557672 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.423794 (+866122us) service_pool.cc:225] Handling call
1027 21:07:58.039653 (+1615859us) ts_tablet_manager.cc:1151] Deleting tablet c8f0cd910b9d4daea287b3b192a81bd2
1027 21:07:58.039660 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.039741 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1602400}
I20251027 21:07:58.040917 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0540fe4fe8744f79ee5e4d8d88f7436. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.224046 10587 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 18b15927ce5744ad8ba81dda52725c11 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251027 21:07:58.300696 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 548) took 2242 ms. Trace:
I20251027 21:07:58.300844 9595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292192b6d0d646de9144bbd6adbd4fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.300812 10043 rpcz_store.cc:276] 1027 21:07:56.058594 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.437272 (+378678us) service_pool.cc:225] Handling call
1027 21:07:58.300456 (+1863184us) ts_tablet_manager.cc:1151] Deleting tablet 371aa8a57aee466ab01652d6061db4c2
1027 21:07:58.300472 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.300620 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1862996}
I20251027 21:07:58.070981 10029 tablet_bootstrap.cc:654] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:58.301371 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 549) took 2242 ms. Trace:
I20251027 21:07:58.301494 10043 rpcz_store.cc:276] 1027 21:07:56.058698 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.437472 (+378774us) service_pool.cc:225] Handling call
1027 21:07:58.301142 (+1863670us) ts_tablet_manager.cc:1151] Deleting tablet 4d2c40f983e743b0b53f987b6a7ba437
1027 21:07:58.301154 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.301298 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1863522}
I20251027 21:07:58.301944 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 550) took 2243 ms. Trace:
I20251027 21:07:58.110704 9601 tablet_service.cc:1505] Processing CreateTablet for tablet fc2cf41a7b474f7d86a46af76cef4100 (DEFAULT_TABLE table=default.Table_562d4900bd0c4a43b93862a97e5d6561 [id=09be1f34af4f4c27b33e97c55206b9fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.111346 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 719321644ea3479ea4529a8a5d048754 (DEFAULT_TABLE table=default.Table_19e1b54d4b7f4d24b1de893d1229bcdb [id=cf22877b5b594a3c871cb103df58e055]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.113224 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 356a04cc6b7444df9ae17c12d5bff727 (DEFAULT_TABLE table=default.Table_61f42d2bf7b941cebdb4d20f78094243 [id=6a7b389bf1d54986a4ad348f44196cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.302421 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc2cf41a7b474f7d86a46af76cef4100. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.302740 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 356a04cc6b7444df9ae17c12d5bff727. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:58.039975 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet e10ae34eaaf3476594813de8e96c7e77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e10ae34eaaf3476594813de8e96c7e77 already in progress: creating tablet
W20251027 21:07:58.209486 9711 leader_election.cc:343] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 1776590d8387494da5dfbd08db17f4b1
I20251027 21:07:58.401363 9711 leader_election.cc:304] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
W20251027 21:07:58.401434 9298 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.9.14.65:44531, user_credentials={real_user=slave}} blocked reactor thread for 361435us
I20251027 21:07:58.118012 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet 7427bbc47a8643c19fb543626c6aea2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:58.185274 9297 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 53948dc94c334329bdf81f1eba94863c (table default.Table_98fe733c6d594a04929bb89536b6c68f [id=487916f3aa3d49a093bbbac97ebb2fe5], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 53948dc94c334329bdf81f1eba94863c consensus state has no leader
I20251027 21:07:58.190346 9771 raft_consensus.cc:1275] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218 [term 2 FOLLOWER]: Refusing update from remote peer 51902b6b61ff4d5584ff21896e847e14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251027 21:07:58.118865 9611 tablet_service.cc:1552] Processing DeleteTablet for tablet d1f24d21bb954ac9b62ce458c6a22462 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:58.040360 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 9351a369dc1e4ac49f6ad419eab3a4c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9351a369dc1e4ac49f6ad419eab3a4c5 already in progress: creating tablet
W20251027 21:07:58.402531 9299 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 362182us
I20251027 21:07:58.123010 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet de2c033a064643b386ee84085cbdad64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:58.352850 9285 kernel_stack_watchdog.cc:198] Thread 9299 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 145ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f95716e3980 (unknown) at ??:0
@ 0x7f95716dd9c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f956b359904 google::LogMessage::Flush() at ??:0
@ 0x7f956b359a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f9581ca87e5 kudu::master::AsyncDeleteReplica::HandleResponse() at ??:0
@ 0x7f9581bfedfd kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f9581caab60 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f9581caa97e _ZNSt17_Function_handlerIFvvEZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7f9574fb5e0f kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f9574fb59ee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7f9574fa1a20 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f9574fa1f6c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f9574f3d46a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f9574f3be26 kudu::rpc::Connection::ReadHandler() at ??:0
I20251027 21:07:58.402938 10529 consensus_queue.cc:1048] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:07:58.040684 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 574) took 1845 ms. Trace:
I20251027 21:07:58.126613 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 4ca7e79ef67d4d20bf1def6d533302a6 (DEFAULT_TABLE table=default.Table_2a89c7b51adb4448b74b750f3ec00c53 [id=fe61bfcc87d0495fa474890e62e2a8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.403395 10041 rpcz_store.cc:276] 1027 21:07:56.195455 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.270700 (+ 75245us) service_pool.cc:225] Handling call
1027 21:07:58.040537 (+1769837us) ts_tablet_manager.cc:1151] Deleting tablet 0e4d35f229bc46e1859ded7c8196ab9d
1027 21:07:58.040548 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.040640 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1754165}
I20251027 21:07:58.187583 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 0bd99431c0694098b41b79f7f89e69d7 (DEFAULT_TABLE table=default.Table_c9c1ea88906a4ee6bbd763eced368b50 [id=5cead33841a749918c161e2180c90a7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.188073 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet 2532bf7190064443a898e47e843fd9ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:58.403781 10041 tablet_replica.cc:333] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:58.287055 10515 raft_consensus.cc:493] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:58.404009 10041 raft_consensus.cc:2243] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:58.403957 10515 raft_consensus.cc:515] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.404246 10041 raft_consensus.cc:2272] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:58.404565 10515 leader_election.cc:290] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:58.306347 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 719321644ea3479ea4529a8a5d048754. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.404852 10041 ts_tablet_manager.cc:1916] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:58.205718 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8390bd307784327bacc17d0c0dd43d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.174448 10572 raft_consensus.cc:493] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:58.406476 10572 raft_consensus.cc:515] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.407346 10572 leader_election.cc:290] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:58.407531 10572 raft_consensus.cc:2749] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:58.206779 10031 tablet_bootstrap.cc:492] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:58.409739 10031 tablet_bootstrap.cc:654] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:58.209676 10579 raft_consensus.cc:2749] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:58.074226 10411 raft_consensus.cc:493] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:58.302045 10043 rpcz_store.cc:276] 1027 21:07:56.058799 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.437632 (+378833us) service_pool.cc:225] Handling call
1027 21:07:58.301742 (+1864110us) ts_tablet_manager.cc:1151] Deleting tablet 618f1657a10947fd98976820564c20b3
1027 21:07:58.301753 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.301881 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1863943}
I20251027 21:07:58.410283 10411 raft_consensus.cc:515] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.410425 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83a040332f994055a9ecf4c66720b7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.410476 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 542) took 2705 ms. Trace:
I20251027 21:07:58.410560 10043 rpcz_store.cc:276] 1027 21:07:55.704846 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.424040 (+719194us) service_pool.cc:225] Handling call
1027 21:07:58.410314 (+1986274us) ts_tablet_manager.cc:1151] Deleting tablet e6ed9d7a700446edafcd0daa02aa7ef6
1027 21:07:58.410324 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.410423 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1971965}
I20251027 21:07:58.410840 10411 leader_election.cc:290] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:58.410913 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 543) took 2700 ms. Trace:
I20251027 21:07:58.178565 10032 tablet_bootstrap.cc:492] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:58.410991 10043 rpcz_store.cc:276] 1027 21:07:55.710438 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.424820 (+714382us) service_pool.cc:225] Handling call
1027 21:07:58.410764 (+1985944us) ts_tablet_manager.cc:1151] Deleting tablet c3b9a461567a48fcb4af514ecdf618b4
1027 21:07:58.410772 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.410851 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1967170}
I20251027 21:07:58.411235 10032 ts_tablet_manager.cc:1403] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.440s user 0.011s sys 0.043s
I20251027 21:07:58.411329 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 544) took 2682 ms. Trace:
I20251027 21:07:58.411408 10043 rpcz_store.cc:276] 1027 21:07:55.728638 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.427304 (+698666us) service_pool.cc:225] Handling call
1027 21:07:58.411178 (+1983874us) ts_tablet_manager.cc:1151] Deleting tablet c9efffa9766e46aca8eb286fc2ad48d2
1027 21:07:58.411185 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.411280 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1961776}
I20251027 21:07:58.411722 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 554) took 2252 ms. Trace:
I20251027 21:07:58.411815 10043 rpcz_store.cc:276] 1027 21:07:56.159694 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.449426 (+289732us) service_pool.cc:225] Handling call
1027 21:07:58.411576 (+1962150us) ts_tablet_manager.cc:1151] Deleting tablet ed5619d295a246379b625f35692203e4
1027 21:07:58.411583 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.411669 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1852104}
I20251027 21:07:58.411984 10032 raft_consensus.cc:359] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.412143 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 553) took 2303 ms. Trace:
I20251027 21:07:58.412159 10032 raft_consensus.cc:385] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.412233 10032 raft_consensus.cc:740] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:58.412227 10043 rpcz_store.cc:276] 1027 21:07:56.108482 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.443619 (+335137us) service_pool.cc:225] Handling call
1027 21:07:58.412010 (+1968391us) ts_tablet_manager.cc:1151] Deleting tablet 64e39cc210664c3d83eaf1fc576f83ce
1027 21:07:58.412020 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.412089 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1844896}
I20251027 21:07:58.412509 10043 tablet_replica.cc:333] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:58.412457 10032 consensus_queue.cc:260] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.412678 10043 raft_consensus.cc:2243] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:58.412837 10043 raft_consensus.cc:2272] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:58.412916 10032 ts_tablet_manager.cc:1434] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:58.413244 10032 tablet_bootstrap.cc:492] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:58.413345 10043 ts_tablet_manager.cc:1916] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:58.175959 10584 raft_consensus.cc:493] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:58.414316 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a2432f5c24b4bbca744d5f4806f2b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.325276 10029 tablet_bootstrap.cc:492] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:58.414567 10029 ts_tablet_manager.cc:1403] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.424s user 0.005s sys 0.007s
I20251027 21:07:58.415405 10029 raft_consensus.cc:359] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.415618 10029 raft_consensus.cc:385] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.415700 10029 raft_consensus.cc:740] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:58.415915 10029 consensus_queue.cc:260] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.416417 10029 ts_tablet_manager.cc:1434] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:58.416709 10029 tablet_bootstrap.cc:492] T 995284883d834bbb9ef583b736ca8402 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:58.185149 9924 raft_consensus.cc:1275] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 2 FOLLOWER]: Refusing update from remote peer 51902b6b61ff4d5584ff21896e847e14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251027 21:07:58.307665 10581 raft_consensus.cc:493] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:58.289973 10560 raft_consensus.cc:493] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251027 21:07:58.189568 9860 leader_election.cc:343] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:07:58.420056 9860 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.65:44531, user_credentials={real_user=slave}} blocked reactor thread for 230505us
I20251027 21:07:58.414265 10584 raft_consensus.cc:515] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.425705 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f6434c5ccd14bae97ebd000ae7b2fdb. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:58.428046 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 0e4d35f229bc46e1859ded7c8196ab9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e4d35f229bc46e1859ded7c8196ab9d already in progress: creating tablet
I20251027 21:07:58.425559 10581 raft_consensus.cc:515] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.431145 10032 tablet_bootstrap.cc:654] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:58.432128 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet c8f0cd910b9d4daea287b3b192a81bd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8f0cd910b9d4daea287b3b192a81bd2 already in progress: creating tablet
W20251027 21:07:58.432367 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 371aa8a57aee466ab01652d6061db4c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 371aa8a57aee466ab01652d6061db4c2 already in progress: creating tablet
W20251027 21:07:58.432570 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 4d2c40f983e743b0b53f987b6a7ba437 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d2c40f983e743b0b53f987b6a7ba437 already in progress: creating tablet
W20251027 21:07:58.433104 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 618f1657a10947fd98976820564c20b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 618f1657a10947fd98976820564c20b3 already in progress: creating tablet
W20251027 21:07:58.433303 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet e6ed9d7a700446edafcd0daa02aa7ef6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6ed9d7a700446edafcd0daa02aa7ef6 already in progress: creating tablet
W20251027 21:07:58.433506 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet c3b9a461567a48fcb4af514ecdf618b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3b9a461567a48fcb4af514ecdf618b4 already in progress: creating tablet
W20251027 21:07:58.433698 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet c9efffa9766e46aca8eb286fc2ad48d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9efffa9766e46aca8eb286fc2ad48d2 already in progress: creating tablet
W20251027 21:07:58.433902 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet ed5619d295a246379b625f35692203e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed5619d295a246379b625f35692203e4 already in progress: creating tablet
I20251027 21:07:58.434257 10560 raft_consensus.cc:515] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:58.434110 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 64e39cc210664c3d83eaf1fc576f83ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64e39cc210664c3d83eaf1fc576f83ce already in progress: creating tablet
I20251027 21:07:58.434931 10560 leader_election.cc:290] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:58.436046 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b04349b15f3494bb0933db56b7b24bc" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:58.436154 9710 leader_election.cc:343] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 49b3d23798b740008c98cd0754ae33f4
I20251027 21:07:58.436770 10584 leader_election.cc:290] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:58.436988 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49d9e5c5960a4534a6603875006e6266" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:58.437354 10581 leader_election.cc:290] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
W20251027 21:07:58.438664 9568 leader_election.cc:343] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 49d9e5c5960a4534a6603875006e6266
I20251027 21:07:58.438984 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bd99431c0694098b41b79f7f89e69d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.439284 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eec11e8fcb60425fb7fd742b6051fc2d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:58.439396 9566 leader_election.cc:343] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.440881 10041 ts_tablet_manager.cc:1929] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 21:07:58.441068 10041 log.cc:1199] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/cb1686fc7c3a4996aa4462884d30c41f
I20251027 21:07:58.441628 10041 ts_tablet_manager.cc:1950] T cb1686fc7c3a4996aa4462884d30c41f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:58.441884 9566 leader_election.cc:304] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:58.442282 10529 raft_consensus.cc:2749] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:58.443408 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 579) took 2137 ms. Trace:
I20251027 21:07:58.443565 10041 rpcz_store.cc:276] 1027 21:07:56.306115 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.306224 (+ 109us) service_pool.cc:225] Handling call
1027 21:07:58.403705 (+2097481us) ts_tablet_manager.cc:1151] Deleting tablet cb1686fc7c3a4996aa4462884d30c41f
1027 21:07:58.403714 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.431228 (+ 27514us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.440741 (+ 9513us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.443321 (+ 2580us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2097206}
I20251027 21:07:58.444640 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 580) took 1969 ms. Trace:
W20251027 21:07:58.444897 9860 leader_election.cc:343] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.444852 10041 rpcz_store.cc:276] 1027 21:07:56.474809 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.474889 (+ 80us) service_pool.cc:225] Handling call
1027 21:07:58.444310 (+1969421us) ts_tablet_manager.cc:1151] Deleting tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a
1027 21:07:58.444320 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.444518 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1969096}
I20251027 21:07:58.446300 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3f22b8dbd1b436aa050c89b48833fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.446287 10515 consensus_queue.cc:1048] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20251027 21:07:58.446527 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4996016747cd484cbda5f11e40e269b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.446719 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8e008c868d04281beba0e5b204eb69a" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:58.446964 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ca7e79ef67d4d20bf1def6d533302a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.447235 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc97e55cfbcb41ec8c6c7410bbf14fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.447652 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c3c89b7036b438cad79d0472619257e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.453769 10029 tablet_bootstrap.cc:654] T 995284883d834bbb9ef583b736ca8402 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:58.454624 9710 leader_election.cc:343] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 10a44aecfade42969454126b990eca24
W20251027 21:07:58.454968 9860 leader_election.cc:343] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:07:58.456310 9710 leader_election.cc:343] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 9248f458baca409aa3505cc0db321b34
I20251027 21:07:58.456586 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccd419ff40a94f9587b024ec87a2517e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.458668 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f90e09e74b6495db1aa2df85de21181. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.460017 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7427bbc47a8643c19fb543626c6aea2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.460238 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 181def804539489e880b47b45023eeb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.463215 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet befdd3a485a1426b8a3adf340d4eae4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.464396 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44804ae1c85749cb9286777e7b5c5707. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.462277 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet cb1686fc7c3a4996aa4462884d30c41f (table default.Table_4a21d4b2faee414e88ba4533b814cce2 [id=3f2856bb858142989104b3f1d9e7f510]) successfully deleted
W20251027 21:07:58.468487 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a already in progress: creating tablet
I20251027 21:07:58.470248 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 582) took 1971 ms. Trace:
I20251027 21:07:58.470364 10041 rpcz_store.cc:276] 1027 21:07:56.475111 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.475597 (+ 486us) service_pool.cc:225] Handling call
1027 21:07:58.445719 (+1970122us) ts_tablet_manager.cc:1151] Deleting tablet 82366f8bf4154ffd87ef45e4a4d5f54b
1027 21:07:58.445781 (+ 62us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.447043 (+ 1262us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1962606}
W20251027 21:07:58.470908 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 82366f8bf4154ffd87ef45e4a4d5f54b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82366f8bf4154ffd87ef45e4a4d5f54b already in progress: creating tablet
I20251027 21:07:58.473554 10032 tablet_bootstrap.cc:492] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:58.473714 10032 ts_tablet_manager.cc:1403] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.061s user 0.001s sys 0.010s
I20251027 21:07:58.474541 10041 tablet_replica.cc:333] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:58.474764 10041 raft_consensus.cc:2243] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:58.475510 10041 raft_consensus.cc:2272] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:58.476047 10041 ts_tablet_manager.cc:1916] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:58.474738 10032 raft_consensus.cc:359] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.477988 10032 raft_consensus.cc:385] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.478111 10032 raft_consensus.cc:740] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:58.478446 10032 consensus_queue.cc:260] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.478991 10032 ts_tablet_manager.cc:1434] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251027 21:07:58.479353 10032 tablet_bootstrap.cc:492] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:58.481333 10032 tablet_bootstrap.cc:654] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:58.483229 9859 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.9.14.66:38579 timed out after 3.010s (SENT)
I20251027 21:07:58.489698 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e9191649aa84b058abdc1477995ef67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.489969 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2ff7fc3c1fa4dbd9a5c38d3219b03fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.494212 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6152f7ac8a4123872098955b8837ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.500309 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5f4f3990da34f5e8f0385595f457427. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:58.501031 9711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 287331us
I20251027 21:07:58.512691 10040 ts_tablet_manager.cc:1929] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:58.512841 10040 log.cc:1199] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/d89c3c588a1747d7875ebf45924fb890
I20251027 21:07:58.513302 10040 ts_tablet_manager.cc:1950] T d89c3c588a1747d7875ebf45924fb890 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:58.513674 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 378) took 4733 ms. Trace:
I20251027 21:07:58.513803 10040 rpcz_store.cc:276] 1027 21:07:53.780687 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.780873 (+ 186us) service_pool.cc:225] Handling call
1027 21:07:58.031159 (+4250286us) ts_tablet_manager.cc:1151] Deleting tablet d89c3c588a1747d7875ebf45924fb890
1027 21:07:58.031168 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.050397 (+ 19229us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.511654 (+461257us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.513624 (+ 1970us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4250024}
I20251027 21:07:58.514325 10040 tablet_replica.cc:333] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:58.514539 10040 raft_consensus.cc:2243] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:58.514703 10040 raft_consensus.cc:2272] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:58.515362 10040 ts_tablet_manager.cc:1916] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:58.525242 10526 tablet.cc:1722] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:07:58.531551 10031 tablet_bootstrap.cc:492] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:58.501325 9711 leader_election.cc:304] T 40bd202c1b834c949f9a5cee23b3486d P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
W20251027 21:07:58.533627 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 04c579bb008448e9baa841135b6dbc61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04c579bb008448e9baa841135b6dbc61 already in progress: creating tablet
W20251027 21:07:58.534556 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 76e4fb2bea7f4a4696fe9f6bb892bccc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76e4fb2bea7f4a4696fe9f6bb892bccc already in progress: creating tablet
W20251027 21:07:58.535223 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 4debc3f13f3a42939b6b83f6523aab7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4debc3f13f3a42939b6b83f6523aab7c already in progress: creating tablet
I20251027 21:07:58.535655 9745 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 639) took 1007 ms. Trace:
W20251027 21:07:58.535671 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 59b6818d29624da5a49dd1a7d20994c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59b6818d29624da5a49dd1a7d20994c6 already in progress: creating tablet
W20251027 21:07:58.536037 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet cb95618d9fd54a4f8121b43e717d5616 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb95618d9fd54a4f8121b43e717d5616 already in progress: creating tablet
W20251027 21:07:58.536415 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 9861a0efcba64141aae450663c5d83de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9861a0efcba64141aae450663c5d83de already in progress: creating tablet
W20251027 21:07:58.536765 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet d52d71f4976143e4a514350621ebe309 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d52d71f4976143e4a514350621ebe309 already in progress: creating tablet
I20251027 21:07:58.538702 10043 ts_tablet_manager.cc:1929] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:58.538872 10043 log.cc:1199] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/94e94cd50c91429d8da9804149501c86
I20251027 21:07:58.539358 10043 ts_tablet_manager.cc:1950] T 94e94cd50c91429d8da9804149501c86 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
W20251027 21:07:58.539520 9859 leader_election.cc:343] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.539973 9859 leader_election.cc:304] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:58.540196 10041 ts_tablet_manager.cc:1929] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:58.540320 10041 log.cc:1199] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/78884c2be75f45f881f5424d3b3f4e08
I20251027 21:07:58.541056 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 557) took 2239 ms. Trace:
W20251027 21:07:58.541181 9859 leader_election.cc:343] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.541302 9859 leader_election.cc:304] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:58.541210 10043 rpcz_store.cc:276] 1027 21:07:56.301122 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.570527 (+269405us) service_pool.cc:225] Handling call
1027 21:07:58.412432 (+1841905us) ts_tablet_manager.cc:1151] Deleting tablet 94e94cd50c91429d8da9804149501c86
1027 21:07:58.412443 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.518194 (+105751us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.538589 (+ 20395us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.541023 (+ 2434us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1808850}
W20251027 21:07:58.541505 9859 raft_consensus.cc:2686] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251027 21:07:58.541769 9859 leader_election.cc:343] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.541896 9859 leader_election.cc:304] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:07:58.542052 9859 raft_consensus.cc:2686] T f1aa5f7f76cb4a1ba15f016942f17067 P 0fbd567f929a4f6d8e4d78feef7ed678: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251027 21:07:58.542192 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 562) took 2240 ms. Trace:
I20251027 21:07:58.542344 10043 rpcz_store.cc:276] 1027 21:07:56.301699 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.730432 (+428733us) service_pool.cc:225] Handling call
1027 21:07:58.541953 (+1811521us) ts_tablet_manager.cc:1151] Deleting tablet 9351a369dc1e4ac49f6ad419eab3a4c5
1027 21:07:58.541961 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.542162 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1775086}
W20251027 21:07:58.542698 9859 leader_election.cc:343] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.542817 9859 leader_election.cc:304] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:58.543138 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 563) took 2241 ms. Trace:
W20251027 21:07:58.543182 9859 leader_election.cc:343] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.543285 9859 leader_election.cc:304] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:58.543337 10043 rpcz_store.cc:276] 1027 21:07:56.301815 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.766883 (+465068us) service_pool.cc:225] Handling call
1027 21:07:58.542967 (+1776084us) ts_tablet_manager.cc:1151] Deleting tablet e477ca4b69c84327b04618d90406e19a
1027 21:07:58.542978 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.543103 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1775946}
I20251027 21:07:58.543660 10411 raft_consensus.cc:2749] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:58.543821 10411 raft_consensus.cc:2749] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:58.543969 10560 raft_consensus.cc:2749] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:58.543972 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 564) took 2242 ms. Trace:
I20251027 21:07:58.544099 10043 rpcz_store.cc:276] 1027 21:07:56.301908 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.767021 (+465113us) service_pool.cc:225] Handling call
1027 21:07:58.543822 (+1776801us) ts_tablet_manager.cc:1151] Deleting tablet bbf92e29993b46389a69efd9e9a2341f
1027 21:07:58.543833 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.543939 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1776658}
I20251027 21:07:58.544555 10043 tablet_replica.cc:333] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:58.544832 10043 raft_consensus.cc:2243] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:07:58.545212 10041 ts_tablet_manager.cc:1950] T 78884c2be75f45f881f5424d3b3f4e08 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:58.545382 10043 raft_consensus.cc:2272] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:58.546030 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 581) took 2071 ms. Trace:
I20251027 21:07:58.546219 10043 ts_tablet_manager.cc:1916] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:58.550315 10041 rpcz_store.cc:276] 1027 21:07:56.474982 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.475565 (+ 583us) service_pool.cc:225] Handling call
1027 21:07:58.474442 (+1998877us) ts_tablet_manager.cc:1151] Deleting tablet 78884c2be75f45f881f5424d3b3f4e08
1027 21:07:58.474452 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.498761 (+ 24309us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.538873 (+ 40112us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.545990 (+ 7117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1991206}
I20251027 21:07:58.555552 10029 tablet_bootstrap.cc:492] T 995284883d834bbb9ef583b736ca8402 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:58.555749 10029 ts_tablet_manager.cc:1403] T 995284883d834bbb9ef583b736ca8402 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.139s user 0.014s sys 0.018s
I20251027 21:07:58.556473 10031 ts_tablet_manager.cc:1403] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.325s user 0.000s sys 0.010s
I20251027 21:07:58.557277 10029 raft_consensus.cc:359] T 995284883d834bbb9ef583b736ca8402 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.557523 10029 raft_consensus.cc:385] T 995284883d834bbb9ef583b736ca8402 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.557718 10029 raft_consensus.cc:740] T 995284883d834bbb9ef583b736ca8402 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:58.558063 10029 consensus_queue.cc:260] T 995284883d834bbb9ef583b736ca8402 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.558727 10031 raft_consensus.cc:359] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.559021 10610 raft_consensus.cc:2749] T 40bd202c1b834c949f9a5cee23b3486d P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251027 21:07:58.561517 9744 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 621) took 1004 ms. Trace:
I20251027 21:07:58.565068 10029 ts_tablet_manager.cc:1434] T 995284883d834bbb9ef583b736ca8402 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20251027 21:07:58.563170 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 06d81182b73e46019fc18c828045001f (DEFAULT_TABLE table=default.Table_e8162d3f2ed747b3802569f1e9ab2115 [id=b2c80efac3b846669503b0bd18f98815]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.565305 9744 rpcz_store.cc:276] 1027 21:07:57.526704 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.533794 (+ 7090us) service_pool.cc:225] Handling call
1027 21:07:57.904600 (+370806us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:57.904653 (+ 53us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:58.413788 (+509135us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.527386 (+113598us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:58.531161 (+ 3775us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":384}
I20251027 21:07:58.567595 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet d89c3c588a1747d7875ebf45924fb890 (table default.Table_f05923c6594b43378fa156d0ab2bd1da [id=ad541dd12124493583f4fdba032b7ed4]) successfully deleted
I20251027 21:07:58.568394 9745 rpcz_store.cc:276] 1027 21:07:57.528294 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.113715 (+585421us) service_pool.cc:225] Handling call
1027 21:07:58.206224 (+ 92509us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:58.206290 (+ 66us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:58.532903 (+326613us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.535481 (+ 2578us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:58.535618 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:58.568590 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 292192b6d0d646de9144bbd6adbd4fbe (DEFAULT_TABLE table=default.Table_004dd356c86a4140b824cdaefda61a23 [id=3a572a109ef24e3ca40144e70129ddec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.569851 9740 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 632) took 1014 ms. Trace:
I20251027 21:07:58.570595 10029 tablet_bootstrap.cc:492] T d94c695320f24a7f9776c493e462ed24 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:58.570916 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 78884c2be75f45f881f5424d3b3f4e08 (table default.Table_1376974f614548189208cb9cf0fd5353 [id=0eb078640c174f8fadb959306a35cada]) successfully deleted
I20251027 21:07:58.571094 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 94e94cd50c91429d8da9804149501c86 (table default.Table_b23313ec4d7b4e32a4283dd404578616 [id=6a545a959eb74d0eb589f1be92a0bc0a]) successfully deleted
I20251027 21:07:58.571164 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 4eb30f4e4b094693a2c1088cfaa68ba3 (DEFAULT_TABLE table=default.Table_1595883e735448e2b56b64cccacc7219 [id=55b0c6dee5944db0a95112b0d6890280]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:58.571359 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 9351a369dc1e4ac49f6ad419eab3a4c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9351a369dc1e4ac49f6ad419eab3a4c5 already in progress: creating tablet
I20251027 21:07:58.571460 9747 tablet_service.cc:1505] Processing CreateTablet for tablet eec11e8fcb60425fb7fd742b6051fc2d (DEFAULT_TABLE table=default.Table_d3a044fd69ff47428850c4d429855c80 [id=e1789ed0ee49428e9829cbd26cc82169]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.571745 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 4ca7e79ef67d4d20bf1def6d533302a6 (DEFAULT_TABLE table=default.Table_2a89c7b51adb4448b74b750f3ec00c53 [id=fe61bfcc87d0495fa474890e62e2a8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.572222 10029 tablet_bootstrap.cc:654] T d94c695320f24a7f9776c493e462ed24 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:58.573257 10031 raft_consensus.cc:385] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.573376 10031 raft_consensus.cc:740] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:58.573688 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292192b6d0d646de9144bbd6adbd4fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.573740 9744 tablet_service.cc:1505] Processing CreateTablet for tablet b0540fe4fe8744f79ee5e4d8d88f7436 (DEFAULT_TABLE table=default.Table_e2e73046f34a4cb98d2a4cb5a48c4ca7 [id=6c60245e7b144353be21e56169b93f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.574052 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eec11e8fcb60425fb7fd742b6051fc2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.574318 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ca7e79ef67d4d20bf1def6d533302a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.574796 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eb30f4e4b094693a2c1088cfaa68ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.575141 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06d81182b73e46019fc18c828045001f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.577688 9740 rpcz_store.cc:276] 1027 21:07:57.527691 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.883058 (+355367us) service_pool.cc:225] Handling call
1027 21:07:58.049254 (+166196us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:58.049275 (+ 21us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:58.539509 (+490234us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.541990 (+ 2481us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:58.542126 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7552}
W20251027 21:07:58.579572 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet e477ca4b69c84327b04618d90406e19a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e477ca4b69c84327b04618d90406e19a already in progress: creating tablet
W20251027 21:07:58.579809 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet bbf92e29993b46389a69efd9e9a2341f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbf92e29993b46389a69efd9e9a2341f already in progress: creating tablet
I20251027 21:07:58.588397 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 583) took 2103 ms. Trace:
I20251027 21:07:58.588939 9749 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 618) took 1055 ms. Trace:
I20251027 21:07:58.589030 9749 rpcz_store.cc:276] 1027 21:07:57.525095 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.525275 (+ 180us) service_pool.cc:225] Handling call
1027 21:07:57.868369 (+343094us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:57.868388 (+ 19us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:58.560490 (+692102us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.580428 (+ 19938us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:58.580521 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:58.589927 10043 ts_tablet_manager.cc:1929] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 21:07:58.590471 9740 tablet_service.cc:1505] Processing CreateTablet for tablet fc2cf41a7b474f7d86a46af76cef4100 (DEFAULT_TABLE table=default.Table_562d4900bd0c4a43b93862a97e5d6561 [id=09be1f34af4f4c27b33e97c55206b9fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.590934 10029 tablet_bootstrap.cc:492] T d94c695320f24a7f9776c493e462ed24 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:58.591158 10029 ts_tablet_manager.cc:1403] T d94c695320f24a7f9776c493e462ed24 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.003s
I20251027 21:07:58.591400 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 0bd99431c0694098b41b79f7f89e69d7 (DEFAULT_TABLE table=default.Table_c9c1ea88906a4ee6bbd763eced368b50 [id=5cead33841a749918c161e2180c90a7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.591717 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0540fe4fe8744f79ee5e4d8d88f7436. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.592242 10029 raft_consensus.cc:359] T d94c695320f24a7f9776c493e462ed24 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.592535 10029 raft_consensus.cc:385] T d94c695320f24a7f9776c493e462ed24 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.592654 10029 raft_consensus.cc:740] T d94c695320f24a7f9776c493e462ed24 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:58.593011 10029 consensus_queue.cc:260] T d94c695320f24a7f9776c493e462ed24 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.598798 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 719321644ea3479ea4529a8a5d048754 (DEFAULT_TABLE table=default.Table_19e1b54d4b7f4d24b1de893d1229bcdb [id=cf22877b5b594a3c871cb103df58e055]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.603075 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bd99431c0694098b41b79f7f89e69d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.603991 9739 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 624) took 1058 ms. Trace:
I20251027 21:07:58.604094 9739 rpcz_store.cc:276] 1027 21:07:57.527018 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.542890 (+ 15872us) service_pool.cc:225] Handling call
1027 21:07:57.905093 (+362203us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:57.905119 (+ 26us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:58.459840 (+554721us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.585007 (+125167us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:58.585175 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:58.604411 10031 consensus_queue.cc:260] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.604771 10029 ts_tablet_manager.cc:1434] T d94c695320f24a7f9776c493e462ed24 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251027 21:07:58.610319 9750 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 630) took 1059 ms. Trace:
I20251027 21:07:58.610423 9750 rpcz_store.cc:276] 1027 21:07:57.527524 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.870672 (+343148us) service_pool.cc:225] Handling call
1027 21:07:58.049043 (+178371us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:58.049084 (+ 41us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:58.517618 (+468534us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.587063 (+ 69445us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:58.587201 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:58.610888 10029 tablet_bootstrap.cc:492] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:58.611430 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 719321644ea3479ea4529a8a5d048754. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.603523 10041 rpcz_store.cc:276] 1027 21:07:56.475246 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.475288 (+ 42us) service_pool.cc:225] Handling call
1027 21:07:58.555250 (+2079962us) ts_tablet_manager.cc:1151] Deleting tablet 465be1b24d16457e834023a2b2194049
1027 21:07:58.555263 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.578374 (+ 23111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2070362}
I20251027 21:07:58.613798 10029 tablet_bootstrap.cc:654] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:58.614181 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 465be1b24d16457e834023a2b2194049 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 465be1b24d16457e834023a2b2194049 already in progress: creating tablet
I20251027 21:07:58.613960 10043 log.cc:1199] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/cb1686fc7c3a4996aa4462884d30c41f
I20251027 21:07:58.615900 10043 ts_tablet_manager.cc:1950] T cb1686fc7c3a4996aa4462884d30c41f P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:58.616906 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 565) took 2314 ms. Trace:
I20251027 21:07:58.617096 10043 rpcz_store.cc:276] 1027 21:07:56.302731 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.767170 (+464439us) service_pool.cc:225] Handling call
1027 21:07:58.544502 (+1777332us) ts_tablet_manager.cc:1151] Deleting tablet cb1686fc7c3a4996aa4462884d30c41f
1027 21:07:58.544511 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.581778 (+ 37267us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.589804 (+ 8026us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.616781 (+ 26977us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1777191}
I20251027 21:07:58.618618 10031 ts_tablet_manager.cc:1434] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.054s user 0.000s sys 0.002s
I20251027 21:07:58.619096 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet cb1686fc7c3a4996aa4462884d30c41f (table default.Table_4a21d4b2faee414e88ba4533b814cce2 [id=3f2856bb858142989104b3f1d9e7f510]) successfully deleted
I20251027 21:07:58.619580 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 566) took 2199 ms. Trace:
I20251027 21:07:58.619691 9739 tablet_service.cc:1505] Processing CreateTablet for tablet 356a04cc6b7444df9ae17c12d5bff727 (DEFAULT_TABLE table=default.Table_61f42d2bf7b941cebdb4d20f78094243 [id=6a7b389bf1d54986a4ad348f44196cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.619830 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc2cf41a7b474f7d86a46af76cef4100. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.619779 10043 rpcz_store.cc:276] 1027 21:07:56.420127 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.767319 (+347192us) service_pool.cc:225] Handling call
1027 21:07:58.619410 (+1852091us) ts_tablet_manager.cc:1151] Deleting tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a
1027 21:07:58.619422 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.619526 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1851955}
I20251027 21:07:58.619998 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 356a04cc6b7444df9ae17c12d5bff727. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:58.620302 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a already in progress: creating tablet
I20251027 21:07:58.620801 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 567) took 2186 ms. Trace:
I20251027 21:07:58.620995 10043 rpcz_store.cc:276] 1027 21:07:56.434490 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.767456 (+332966us) service_pool.cc:225] Handling call
1027 21:07:58.620575 (+1853119us) ts_tablet_manager.cc:1151] Deleting tablet 78884c2be75f45f881f5424d3b3f4e08
1027 21:07:58.620587 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.620681 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1852973}
W20251027 21:07:58.624111 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 78884c2be75f45f881f5424d3b3f4e08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78884c2be75f45f881f5424d3b3f4e08 already in progress: creating tablet
I20251027 21:07:58.626024 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 568) took 2183 ms. Trace:
I20251027 21:07:58.626205 10043 rpcz_store.cc:276] 1027 21:07:56.442167 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.767610 (+325443us) service_pool.cc:225] Handling call
1027 21:07:58.625840 (+1858230us) ts_tablet_manager.cc:1151] Deleting tablet 82366f8bf4154ffd87ef45e4a4d5f54b
1027 21:07:58.625852 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.625964 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1858049}
W20251027 21:07:58.626798 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 82366f8bf4154ffd87ef45e4a4d5f54b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82366f8bf4154ffd87ef45e4a4d5f54b already in progress: creating tablet
I20251027 21:07:58.627285 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 569) took 2170 ms. Trace:
I20251027 21:07:58.627429 10043 rpcz_store.cc:276] 1027 21:07:56.456370 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.767801 (+311431us) service_pool.cc:225] Handling call
1027 21:07:58.627112 (+1859311us) ts_tablet_manager.cc:1151] Deleting tablet 465be1b24d16457e834023a2b2194049
1027 21:07:58.627123 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.627232 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1859155}
W20251027 21:07:58.627977 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 465be1b24d16457e834023a2b2194049 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 465be1b24d16457e834023a2b2194049 already in progress: creating tablet
I20251027 21:07:58.628679 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 570) took 2108 ms. Trace:
I20251027 21:07:58.628825 10043 rpcz_store.cc:276] 1027 21:07:56.520656 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.767970 (+247314us) service_pool.cc:225] Handling call
1027 21:07:58.628512 (+1860542us) ts_tablet_manager.cc:1151] Deleting tablet a7d8d060916941a5b7e944e8b08a2882
1027 21:07:58.628524 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.628622 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1860344}
W20251027 21:07:58.629349 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet a7d8d060916941a5b7e944e8b08a2882 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7d8d060916941a5b7e944e8b08a2882 already in progress: creating tablet
I20251027 21:07:58.629841 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 571) took 2062 ms. Trace:
I20251027 21:07:58.630218 9736 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 636) took 1084 ms. Trace:
I20251027 21:07:58.630340 9736 rpcz_store.cc:276] 1027 21:07:57.527992 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.092324 (+564332us) service_pool.cc:225] Handling call
1027 21:07:58.208515 (+116191us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:58.208551 (+ 36us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:58.510658 (+302107us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.611025 (+100367us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:58.612312 (+ 1287us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:58.630412 10043 rpcz_store.cc:276] 1027 21:07:56.570540 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.768173 (+197633us) service_pool.cc:225] Handling call
1027 21:07:58.629658 (+1861485us) ts_tablet_manager.cc:1151] Deleting tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a
1027 21:07:58.629669 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.629794 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1861320}
I20251027 21:07:58.630640 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet d1f24d21bb954ac9b62ce458c6a22462 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:58.630810 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet de2c033a064643b386ee84085cbdad64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:58.630851 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a already in progress: creating tablet
I20251027 21:07:58.630995 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 2532bf7190064443a898e47e843fd9ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:58.631326 9736 tablet_service.cc:1505] Processing CreateTablet for tablet deb8b54e5e0f42bca4330d2160e5524a (DEFAULT_TABLE table=default.Table_851f66cf4b1047879eed5fb768c07156 [id=743b2e05c3674d1fabb93aa073d0df40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.631616 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 7427bbc47a8643c19fb543626c6aea2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:58.632169 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 561) took 2330 ms. Trace:
I20251027 21:07:58.632983 10029 tablet_bootstrap.cc:492] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:58.633091 10043 rpcz_store.cc:276] 1027 21:07:56.301610 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.666926 (+365316us) service_pool.cc:225] Handling call
1027 21:07:58.631984 (+1965058us) ts_tablet_manager.cc:1151] Deleting tablet 0e4d35f229bc46e1859ded7c8196ab9d
1027 21:07:58.631994 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.632112 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1846127}
I20251027 21:07:58.634661 10040 ts_tablet_manager.cc:1929] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251027 21:07:58.634986 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 0e4d35f229bc46e1859ded7c8196ab9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e4d35f229bc46e1859ded7c8196ab9d already in progress: creating tablet
I20251027 21:07:58.633643 10029 ts_tablet_manager.cc:1403] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.004s
I20251027 21:07:58.636698 10029 raft_consensus.cc:359] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.637161 10029 raft_consensus.cc:385] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.637300 10029 raft_consensus.cc:740] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:58.637646 10029 consensus_queue.cc:260] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.638706 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet fde0980aa64c4496861a662f7ed3ef75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:58.642657 10031 tablet_bootstrap.cc:492] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:58.642994 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f22f013b25074d4db9c7f13caf5b9b42" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:58.643529 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6bf9c2a78aac42528d67846dc58adc40" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:58.644150 10029 ts_tablet_manager.cc:1434] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
W20251027 21:07:58.646735 9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.124:45071, user_credentials={real_user=slave}} blocked reactor thread for 134577us
I20251027 21:07:58.647101 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 584) took 2156 ms. Trace:
I20251027 21:07:58.647223 10032 tablet_bootstrap.cc:492] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
W20251027 21:07:58.647986 9859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 103082us
I20251027 21:07:58.648041 10041 rpcz_store.cc:276] 1027 21:07:56.483476 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.484971 (+ 1495us) service_pool.cc:225] Handling call
1027 21:07:58.612026 (+2127055us) ts_tablet_manager.cc:1151] Deleting tablet a7d8d060916941a5b7e944e8b08a2882
1027 21:07:58.612036 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.639713 (+ 27677us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2126857}
W20251027 21:07:58.648243 9859 leader_election.cc:343] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:07:58.648890 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet a7d8d060916941a5b7e944e8b08a2882 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7d8d060916941a5b7e944e8b08a2882 already in progress: creating tablet
I20251027 21:07:58.649680 9755 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 607) took 1096 ms. Trace:
I20251027 21:07:58.649437 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 585) took 2161 ms. Trace:
I20251027 21:07:58.649438 10628 raft_consensus.cc:493] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:58.650362 10032 ts_tablet_manager.cc:1403] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.168s user 0.002s sys 0.004s
I20251027 21:07:58.652738 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8e008c868d04281beba0e5b204eb69a" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:58.653041 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed5619d295a246379b625f35692203e4" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:58.653259 10029 tablet_bootstrap.cc:492] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:58.653736 10614 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:07:58.654304 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e8c83b501d8426fa0cad6a89f9a0b9b" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:58.654721 10614 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: This master's current role is: LEADER
I20251027 21:07:58.654846 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 592) took 2017 ms. Trace:
I20251027 21:07:58.655318 10041 rpcz_store.cc:276] 1027 21:07:56.487939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.488052 (+ 113us) service_pool.cc:225] Handling call
1027 21:07:58.648410 (+2160358us) ts_tablet_manager.cc:1151] Deleting tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a
1027 21:07:58.648424 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.649312 (+ 888us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2157005}
I20251027 21:07:58.656010 9755 rpcz_store.cc:276] 1027 21:07:57.521622 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.528625 (+ 7003us) service_pool.cc:225] Handling call
1027 21:07:57.868569 (+339944us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:57.869707 (+ 1138us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:58.465496 (+595789us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.618325 (+152829us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:58.618467 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:58.655953 10032 raft_consensus.cc:359] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.658777 10032 raft_consensus.cc:385] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.658893 10032 raft_consensus.cc:740] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
W20251027 21:07:58.659298 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a already in progress: creating tablet
I20251027 21:07:58.659199 10032 consensus_queue.cc:260] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.659485 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deb8b54e5e0f42bca4330d2160e5524a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.659818 10032 ts_tablet_manager.cc:1434] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251027 21:07:58.660156 10032 tablet_bootstrap.cc:492] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:58.661067 10040 log.cc:1199] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/10a44aecfade42969454126b990eca24
I20251027 21:07:58.662329 10032 tablet_bootstrap.cc:654] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:58.659264 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 606) took 2148 ms. Trace:
I20251027 21:07:58.665632 10041 rpcz_store.cc:276] 1027 21:07:56.510696 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.710946 (+200250us) service_pool.cc:225] Handling call
1027 21:07:58.659091 (+1948145us) ts_tablet_manager.cc:1151] Deleting tablet fde0980aa64c4496861a662f7ed3ef75
1027 21:07:58.659110 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.659227 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1939665}
I20251027 21:07:58.666328 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 617) took 1988 ms. Trace:
W20251027 21:07:58.666476 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet fde0980aa64c4496861a662f7ed3ef75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fde0980aa64c4496861a662f7ed3ef75 already in progress: creating tablet
I20251027 21:07:58.666504 10041 rpcz_store.cc:276] 1027 21:07:56.677584 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.860650 (+183066us) service_pool.cc:225] Handling call
1027 21:07:58.666176 (+1805526us) ts_tablet_manager.cc:1151] Deleting tablet 84b9de82d55447bba2a897edb683e5c5
1027 21:07:58.666186 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.666290 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1804775}
I20251027 21:07:58.667066 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 618) took 1936 ms. Trace:
W20251027 21:07:58.667088 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 84b9de82d55447bba2a897edb683e5c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84b9de82d55447bba2a897edb683e5c5 already in progress: creating tablet
I20251027 21:07:58.667249 10041 rpcz_store.cc:276] 1027 21:07:56.730891 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.861621 (+130730us) service_pool.cc:225] Handling call
1027 21:07:58.666946 (+1805325us) ts_tablet_manager.cc:1151] Deleting tablet 07a9c3b7ef1548ec9d5ae9f20545f263
1027 21:07:58.666953 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.667030 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1779601}
W20251027 21:07:58.667656 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 07a9c3b7ef1548ec9d5ae9f20545f263 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07a9c3b7ef1548ec9d5ae9f20545f263 already in progress: creating tablet
I20251027 21:07:58.667786 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 626) took 1909 ms. Trace:
I20251027 21:07:58.667930 10041 rpcz_store.cc:276] 1027 21:07:56.757930 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.064912 (+306982us) service_pool.cc:225] Handling call
1027 21:07:58.667603 (+1602691us) ts_tablet_manager.cc:1151] Deleting tablet 1329ff0f8ebe4da8a6462a574ede98d4
1027 21:07:58.667613 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.667701 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1601597}
I20251027 21:07:58.668354 10041 tablet_replica.cc:333] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:58.668331 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 2752b86151a6428c88bfaac1ff72cfb2
I20251027 21:07:58.668385 10043 rpcz_store.cc:276] 1027 21:07:56.637606 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.166261 (+528655us) service_pool.cc:225] Handling call
1027 21:07:58.654690 (+1488429us) ts_tablet_manager.cc:1151] Deleting tablet fde0980aa64c4496861a662f7ed3ef75
1027 21:07:58.654700 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.654795 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1488084}
I20251027 21:07:58.668622 10041 raft_consensus.cc:2243] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:58.668869 9307 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:41710 (request call id 835) took 1156 ms. Trace:
I20251027 21:07:58.669452 9741 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 619) took 1143 ms. Trace:
I20251027 21:07:58.668977 9307 rpcz_store.cc:276] 1027 21:07:57.512350 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.512398 (+ 48us) service_pool.cc:225] Handling call
1027 21:07:57.535034 (+ 22636us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x616000051154. stack: 00007f956e0a89f6 00007f956e0a87b0 00007f956d0843ad 0000000000983f04 0000000000983ecd 00007f957d3af6bd 00007f957d3af7b3 00007f9579f76441 00007f9581c309dd 00007f9581dbe256 00007f95788cf486 00007f95788cf1b5 00007f957504727e 00007f957504615e 00007f957504c395 00007f957504ccbf
1027 21:07:57.851932 (+316898us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1027 21:07:57.852113 (+ 181us) catalog_manager.cc:2815] Modifying in-memory table state
1027 21:07:57.852223 (+ 110us) catalog_manager.cc:2824] Locking tablets
1027 21:07:57.852405 (+ 182us) catalog_manager.cc:2838] Removing table and tablets from system table
1027 21:07:58.665457 (+813052us) catalog_manager.cc:2856] Removing table from by-name map
1027 21:07:58.665554 (+ 97us) catalog_manager.cc:2872] Committing in-memory state
1027 21:07:58.665598 (+ 44us) catalog_manager.cc:2876] Aborting table tasks
1027 21:07:58.668806 (+ 3208us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:57.855292 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:07:57.855418 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:07:57.855426 (+ 8us) write_op.cc:435] Acquired schema lock
1027 21:07:57.855428 (+ 2us) tablet.cc:660] Decoding operations
1027 21:07:57.855589 (+ 161us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:07:57.855605 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1027 21:07:57.855610 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:07:57.855726 (+ 116us) tablet.cc:699] Row locks acquired
1027 21:07:57.855728 (+ 2us) write_op.cc:260] PREPARE: finished
1027 21:07:57.855770 (+ 42us) write_op.cc:270] Start()
1027 21:07:57.855807 (+ 37us) write_op.cc:276] Timestamp: P: 1761599277855754 usec, L: 0
1027 21:07:57.855811 (+ 4us) op_driver.cc:348] REPLICATION: starting
1027 21:07:57.856079 (+ 268us) log.cc:844] Serialized 756 byte log entry
1027 21:07:58.505553 (+649474us) op_driver.cc:464] REPLICATION: finished
1027 21:07:58.662279 (+156726us) write_op.cc:301] APPLY: starting
1027 21:07:58.662301 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
1027 21:07:58.662347 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
1027 21:07:58.662372 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:07:58.662482 (+ 110us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:07:58.662517 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1027 21:07:58.662524 (+ 7us) write_op.cc:312] APPLY: finished
1027 21:07:58.662624 (+ 100us) log.cc:844] Serialized 29 byte log entry
1027 21:07:58.662703 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.662724 (+ 21us) write_op.cc:454] Released schema lock
1027 21:07:58.662732 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":2668,"spinlock_wait_cycles":51539712,"child_traces":[["op",{"apply.queue_time_us":152112,"num_ops":2,"prepare.queue_time_us":2132,"prepare.run_cpu_time_us":909,"prepare.run_wall_time_us":910,"raft.queue_time_us":105,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":276,"replication_time_us":649700,"thread_start_us":143538,"threads_started":1}]]}
I20251027 21:07:58.671356 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 603) took 2032 ms. Trace:
I20251027 21:07:58.671443 10043 rpcz_store.cc:276] 1027 21:07:56.638622 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.434531 (+795909us) service_pool.cc:225] Handling call
1027 21:07:58.671202 (+1236671us) ts_tablet_manager.cc:1151] Deleting tablet 84b9de82d55447bba2a897edb683e5c5
1027 21:07:58.671211 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.671307 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1236500}
W20251027 21:07:58.671583 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 1329ff0f8ebe4da8a6462a574ede98d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1329ff0f8ebe4da8a6462a574ede98d4 already in progress: creating tablet
I20251027 21:07:58.671777 10041 raft_consensus.cc:2272] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:58.671836 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 2752b86151a6428c88bfaac1ff72cfb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:58.672562 10041 ts_tablet_manager.cc:1916] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:58.673659 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet fde0980aa64c4496861a662f7ed3ef75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fde0980aa64c4496861a662f7ed3ef75 already in progress: creating tablet
I20251027 21:07:58.673703 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b04349b15f3494bb0933db56b7b24bc" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:58.673745 9711 leader_election.cc:343] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 49b3d23798b740008c98cd0754ae33f4
I20251027 21:07:58.673789 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e8c83b501d8426fa0cad6a89f9a0b9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:58.673882 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 84b9de82d55447bba2a897edb683e5c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84b9de82d55447bba2a897edb683e5c5 already in progress: creating tablet
I20251027 21:07:58.672036 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccaf5c2f8e544842880ba8ba27300ae6" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:58.673882 9711 leader_election.cc:304] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:58.674595 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 2752b86151a6428c88bfaac1ff72cfb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:58.675298 9750 tablet_service.cc:1505] Processing CreateTablet for tablet de2c033a064643b386ee84085cbdad64 (DEFAULT_TABLE table=default.Table_09602bd1d278472bb1ed05abe9d1def4 [id=9a86a04b859446a685bcc946e7331dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.676492 10043 tablet_replica.cc:333] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:58.676684 10043 raft_consensus.cc:2243] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:07:58.677018 10043 pending_rounds.cc:70] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14: Trying to abort 1 pending ops.
I20251027 21:07:58.677103 10043 pending_rounds.cc:77] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510642905460736 op_type: NO_OP noop_request { }
I20251027 21:07:58.677219 10043 raft_consensus.cc:2889] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:07:58.677291 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de2c033a064643b386ee84085cbdad64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.677356 10043 raft_consensus.cc:2272] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:58.677986 10029 tablet_bootstrap.cc:654] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:58.678068 9743 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 637) took 1127 ms. Trace:
I20251027 21:07:58.678180 9743 rpcz_store.cc:276] 1027 21:07:57.528090 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.093538 (+565448us) service_pool.cc:225] Handling call
1027 21:07:58.206367 (+112829us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:58.206434 (+ 67us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:58.542524 (+336090us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.639215 (+ 96691us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:58.655696 (+ 16481us) spinlock_profiling.cc:243] Waited 5.26 ms on lock 0x61800000d518. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d840302fd 00007f8d8403136e 00007f8d97c6cc9d 00007f8d97beb203 00007f8d93e1ddb9 00007f8d93e12028 00007f8d93e13967 00007f8d93d62482 00007f8d8dbff6a6 00007f8d8dbff3d5
1027 21:07:58.655779 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12105600}
I20251027 21:07:58.678423 9737 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 631) took 1139 ms. Trace:
I20251027 21:07:58.678491 9737 rpcz_store.cc:276] 1027 21:07:57.527606 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.882956 (+355350us) service_pool.cc:225] Handling call
1027 21:07:58.049163 (+166207us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:58.049190 (+ 27us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:58.565819 (+516629us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.666841 (+101022us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:58.666968 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4893}
I20251027 21:07:58.678681 9741 rpcz_store.cc:276] 1027 21:07:57.525718 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.525765 (+ 47us) service_pool.cc:225] Handling call
1027 21:07:57.904938 (+379173us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:57.904964 (+ 26us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:58.570722 (+665758us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.669297 (+ 98575us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:58.669416 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251027 21:07:58.678910 9711 leader_election.cc:343] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 10a44aecfade42969454126b990eca24
I20251027 21:07:58.675822 9751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 620) took 1146 ms. Trace:
I20251027 21:07:58.679708 9755 tablet_service.cc:1505] Processing CreateTablet for tablet eeb667b3b10746dbad351c4c123ffd97 (DEFAULT_TABLE table=default.Table_4aae3c9852244881bf5286450425cf5d [id=f500277871dc4f78ab54c95ad34ef1c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.680239 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet f22f013b25074d4db9c7f13caf5b9b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:58.681160 10610 raft_consensus.cc:2749] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:58.651185 10627 tablet.cc:1722] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:07:58.651208 10628 raft_consensus.cc:515] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.684815 10628 leader_election.cc:290] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:58.692099 10032 tablet_bootstrap.cc:492] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:58.692238 10032 ts_tablet_manager.cc:1403] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.006s
I20251027 21:07:58.692907 9746 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 622) took 1154 ms. Trace:
I20251027 21:07:58.693033 9746 rpcz_store.cc:276] 1027 21:07:57.526795 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.535673 (+ 8878us) service_pool.cc:225] Handling call
1027 21:07:57.905299 (+369626us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:57.905323 (+ 24us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:58.572737 (+667414us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.681482 (+108745us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:58.681628 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:58.683492 9711 leader_election.cc:304] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:58.650525 9859 leader_election.cc:304] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:07:58.694203 9859 raft_consensus.cc:2686] T e0968c2f25b64f6685f38a999ee9d60e P 0fbd567f929a4f6d8e4d78feef7ed678: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251027 21:07:58.694214 10040 ts_tablet_manager.cc:1950] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
W20251027 21:07:58.695333 9711 raft_consensus.cc:2686] T 10a44aecfade42969454126b990eca24 P 77b0513f17c0472088901a1ce8b08218: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251027 21:07:58.695645 9711 leader_election.cc:343] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 9248f458baca409aa3505cc0db321b34
W20251027 21:07:58.696151 9859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 47895.2us
I20251027 21:07:58.694376 10032 raft_consensus.cc:359] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:58.697230 10032 raft_consensus.cc:385] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.697368 10032 raft_consensus.cc:740] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:58.698082 9307 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet a9339c6b1154400eace53d765528fdf9 (table default.Table_5a6211c8465040dbad1cc86377a370ea [id=91fb9f95fe9343fca1d8cb0d0f781afb]). Expected version 2 got 0
I20251027 21:07:58.697642 10032 consensus_queue.cc:260] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:58.698823 10032 ts_tablet_manager.cc:1434] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251027 21:07:58.699077 9307 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet d13d00353b6740b5bed5a153936d2059 (table default.Table_6b471834498c4afeb57279278d45d9a9 [id=f2edcf33b0b74923b4ddbbc2419e14ca]). Expected version 1 got 0
I20251027 21:07:58.699146 10032 tablet_bootstrap.cc:492] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:07:58.700450 9859 leader_election.cc:343] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.700613 9859 leader_election.cc:304] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:58.700887 10628 raft_consensus.cc:2749] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:58.700943 10032 tablet_bootstrap.cc:654] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:58.701339 9859 leader_election.cc:343] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.702768 9859 leader_election.cc:304] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:58.702826 9273 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu
/tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.14.124:45071
--webserver_interface=127.9.14.124
--webserver_port=41907
--builtin_ntp_servers=127.9.14.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.14.126:34057,127.9.14.125:46313,127.9.14.124:45071
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251027 21:07:58.695799 9751 rpcz_store.cc:276] 1027 21:07:57.526238 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.526288 (+ 50us) service_pool.cc:225] Handling call
1027 21:07:57.904835 (+378547us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:57.904861 (+ 26us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:58.558536 (+653675us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.666770 (+108234us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:58.672570 (+ 5800us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5683}
I20251027 21:07:58.704082 10031 tablet_bootstrap.cc:654] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:58.704152 10628 raft_consensus.cc:2749] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:58.704422 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeb667b3b10746dbad351c4c123ffd97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.705963 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 380) took 4807 ms. Trace:
I20251027 21:07:58.706080 10040 rpcz_store.cc:276] 1027 21:07:53.888927 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.888985 (+ 58us) service_pool.cc:225] Handling call
1027 21:07:58.514212 (+4625227us) ts_tablet_manager.cc:1151] Deleting tablet 10a44aecfade42969454126b990eca24
1027 21:07:58.514222 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.560590 (+ 46368us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.632166 (+ 71576us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.696198 (+ 64032us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4622779}
I20251027 21:07:58.707576 10041 ts_tablet_manager.cc:1929] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:58.710798 10041 log.cc:1199] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/69da799c886841a19002d3c7c5084e73
I20251027 21:07:58.711225 10041 ts_tablet_manager.cc:1950] T 69da799c886841a19002d3c7c5084e73 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:58.711320 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d94c695320f24a7f9776c493e462ed24" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:58.711539 9632 raft_consensus.cc:2468] T d94c695320f24a7f9776c493e462ed24 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 0.
I20251027 21:07:58.712275 9711 leader_election.cc:304] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:58.712618 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 1776590d8387494da5dfbd08db17f4b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 7aa8c538b6c849a28a28ffe706306891) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:58.712658 9860 leader_election.cc:304] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:07:58.713423 10628 raft_consensus.cc:2804] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:58.713510 10628 raft_consensus.cc:493] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:58.713570 10628 raft_consensus.cc:3060] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:58.715576 10610 raft_consensus.cc:2749] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:58.715664 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d94c695320f24a7f9776c493e462ed24" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:58.715816 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet 2752b86151a6428c88bfaac1ff72cfb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:58.716424 10628 raft_consensus.cc:515] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.716966 10628 leader_election.cc:290] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:58.718528 10515 raft_consensus.cc:493] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:58.719565 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 10a44aecfade42969454126b990eca24 (table default.Table_b0f0589285ac4a56b7a2b23ace6248f6 [id=43ebd8ba9e694d17a9def870c95d7aa6]) successfully deleted
I20251027 21:07:58.719704 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d94c695320f24a7f9776c493e462ed24" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:07:58.719898 9632 raft_consensus.cc:3060] T d94c695320f24a7f9776c493e462ed24 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:58.722369 10515 raft_consensus.cc:515] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.722903 10515 leader_election.cc:290] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:07:58.723961 9568 leader_election.cc:343] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: b87e809ff897417c88e5edc19e2d591e
I20251027 21:07:58.724020 9632 raft_consensus.cc:2468] T d94c695320f24a7f9776c493e462ed24 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 1.
I20251027 21:07:58.724351 9860 leader_election.cc:304] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:07:58.724575 10043 ts_tablet_manager.cc:1916] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:58.724709 10628 raft_consensus.cc:2804] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:58.726217 10628 raft_consensus.cc:697] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Becoming Leader. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Running, Role: LEADER
I20251027 21:07:58.726588 10628 consensus_queue.cc:237] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.727058 10560 raft_consensus.cc:493] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:58.727206 10560 raft_consensus.cc:515] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.727806 10560 leader_election.cc:290] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:58.728624 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c400101d2a39490e83722500839198cd" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:58.729046 9860 leader_election.cc:343] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.729112 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b87e809ff897417c88e5edc19e2d591e" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:58.729300 9859 leader_election.cc:343] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:07:58.729521 9566 leader_election.cc:343] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.729588 9859 leader_election.cc:304] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:58.729655 9566 leader_election.cc:304] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:58.729914 10560 raft_consensus.cc:2749] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:58.730065 10515 raft_consensus.cc:2749] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:58.730336 10040 tablet_replica.cc:333] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
W20251027 21:07:58.730448 9859 leader_election.cc:343] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.730607 9859 leader_election.cc:304] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:07:58.730844 9859 raft_consensus.cc:2686] T ed5619d295a246379b625f35692203e4 P 0fbd567f929a4f6d8e4d78feef7ed678: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251027 21:07:58.731510 9859 leader_election.cc:343] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.731658 9859 leader_election.cc:304] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:58.731901 10560 raft_consensus.cc:2749] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:07:58.732322 9859 leader_election.cc:343] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.732795 9859 leader_election.cc:304] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:58.733284 10560 raft_consensus.cc:2749] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:58.732193 10630 tablet.cc:1722] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251027 21:07:58.733685 9859 leader_election.cc:343] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.734601 10029 tablet_bootstrap.cc:492] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:58.734608 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 627) took 1927 ms. Trace:
I20251027 21:07:58.734819 10029 ts_tablet_manager.cc:1403] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.084s user 0.005s sys 0.000s
I20251027 21:07:58.734812 9859 leader_election.cc:304] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:58.735235 10560 raft_consensus.cc:2749] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:58.735795 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d94c695320f24a7f9776c493e462ed24" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:58.736153 10040 raft_consensus.cc:2243] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:58.736475 10029 raft_consensus.cc:359] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.736550 9306 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41710 (request call id 840) took 1106 ms. Trace:
I20251027 21:07:58.736671 10029 raft_consensus.cc:385] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.736742 10029 raft_consensus.cc:740] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:58.736665 9306 rpcz_store.cc:276] 1027 21:07:57.629716 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.858706 (+228990us) service_pool.cc:225] Handling call
1027 21:07:57.859592 (+ 886us) catalog_manager.cc:2101] Acquired catalog manager lock
1027 21:07:57.860166 (+ 574us) catalog_manager.cc:2147] Created new table and tablet info
1027 21:07:58.736264 (+876098us) catalog_manager.cc:2207] Wrote table and tablets to system table
1027 21:07:58.736391 (+ 127us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1027 21:07:58.736508 (+ 117us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:57.882002 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:07:57.882095 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:07:57.882101 (+ 6us) write_op.cc:435] Acquired schema lock
1027 21:07:57.882104 (+ 3us) tablet.cc:660] Decoding operations
1027 21:07:57.882260 (+ 156us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:07:57.882274 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1027 21:07:57.882279 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:07:57.882353 (+ 74us) tablet.cc:699] Row locks acquired
1027 21:07:57.882356 (+ 3us) write_op.cc:260] PREPARE: finished
1027 21:07:57.882403 (+ 47us) write_op.cc:270] Start()
1027 21:07:57.882436 (+ 33us) write_op.cc:276] Timestamp: P: 1761599277882388 usec, L: 0
1027 21:07:57.882440 (+ 4us) op_driver.cc:348] REPLICATION: starting
1027 21:07:57.882639 (+ 199us) log.cc:844] Serialized 433 byte log entry
1027 21:07:58.715003 (+832364us) op_driver.cc:464] REPLICATION: finished
1027 21:07:58.733140 (+ 18137us) write_op.cc:301] APPLY: starting
1027 21:07:58.733158 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1027 21:07:58.733208 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
1027 21:07:58.733243 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:07:58.733387 (+ 144us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:07:58.733420 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1027 21:07:58.733428 (+ 8us) write_op.cc:312] APPLY: finished
1027 21:07:58.733509 (+ 81us) log.cc:844] Serialized 29 byte log entry
1027 21:07:58.735637 (+ 2128us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.736138 (+ 501us) write_op.cc:454] Released schema lock
1027 21:07:58.736148 (+ 10us) write_op.cc:341] FINISH: Updating metrics
1027 21:07:58.736244 (+ 96us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.736261 (+ 17us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":18053,"apply.run_cpu_time_us":605,"apply.run_wall_time_us":3103,"num_ops":2,"prepare.queue_time_us":21397,"prepare.run_cpu_time_us":719,"prepare.run_wall_time_us":718,"replication_time_us":832535}]]}
I20251027 21:07:58.737007 10029 consensus_queue.cc:260] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.737493 9748 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 626) took 1203 ms. Trace:
W20251027 21:07:58.738334 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 83f5fbf5757a49df93ab949b7657c61f consensus state has no leader
I20251027 21:07:58.738516 9310 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:41710 (request call id 841) took 1085 ms. Trace:
I20251027 21:07:58.738605 9310 rpcz_store.cc:276] 1027 21:07:57.653131 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.860710 (+207579us) service_pool.cc:225] Handling call
1027 21:07:57.861033 (+ 323us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1027 21:07:57.861247 (+ 214us) catalog_manager.cc:3708] Apply alter schema
1027 21:07:57.861393 (+ 146us) catalog_manager.cc:3740] Acquired catalog manager lock
1027 21:07:57.861446 (+ 53us) catalog_manager.cc:3898] Updating metadata on disk
1027 21:07:58.738094 (+876648us) catalog_manager.cc:3929] Committing alterations to in-memory state
1027 21:07:58.738456 (+ 362us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:57.882770 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:07:57.882838 (+ 68us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:07:57.882844 (+ 6us) write_op.cc:435] Acquired schema lock
1027 21:07:57.882846 (+ 2us) tablet.cc:660] Decoding operations
1027 21:07:57.882932 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:07:57.882947 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1027 21:07:57.882953 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1027 21:07:57.883003 (+ 50us) tablet.cc:699] Row locks acquired
1027 21:07:57.883006 (+ 3us) write_op.cc:260] PREPARE: finished
1027 21:07:57.883036 (+ 30us) write_op.cc:270] Start()
1027 21:07:57.883064 (+ 28us) write_op.cc:276] Timestamp: P: 1761599277883022 usec, L: 0
1027 21:07:57.883068 (+ 4us) op_driver.cc:348] REPLICATION: starting
1027 21:07:57.883223 (+ 155us) log.cc:844] Serialized 405 byte log entry
1027 21:07:58.715090 (+831867us) op_driver.cc:464] REPLICATION: finished
1027 21:07:58.736349 (+ 21259us) write_op.cc:301] APPLY: starting
1027 21:07:58.736361 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
1027 21:07:58.736396 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
1027 21:07:58.737187 (+ 791us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:07:58.737814 (+ 627us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:07:58.737871 (+ 57us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:07:58.737878 (+ 7us) write_op.cc:312] APPLY: finished
1027 21:07:58.737949 (+ 71us) log.cc:844] Serialized 23 byte log entry
1027 21:07:58.738005 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.738020 (+ 15us) write_op.cc:454] Released schema lock
1027 21:07:58.738025 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1027 21:07:58.738098 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.738102 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":21225,"apply.run_cpu_time_us":427,"apply.run_wall_time_us":1740,"num_ops":1,"prepare.queue_time_us":20925,"prepare.run_cpu_time_us":499,"prepare.run_wall_time_us":498,"replication_time_us":831991,"thread_start_us":18101,"threads_started":1}]]}
W20251027 21:07:58.735971 9859 leader_election.cc:343] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.739426 10041 rpcz_store.cc:276] 1027 21:07:56.807400 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.065108 (+257708us) service_pool.cc:225] Handling call
1027 21:07:58.668303 (+1603195us) ts_tablet_manager.cc:1151] Deleting tablet 69da799c886841a19002d3c7c5084e73
1027 21:07:58.668311 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.694751 (+ 26440us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.707471 (+ 12720us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.734561 (+ 27090us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1595955}
I20251027 21:07:58.740526 10029 ts_tablet_manager.cc:1434] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251027 21:07:58.741034 10040 raft_consensus.cc:2272] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:58.741568 10040 ts_tablet_manager.cc:1916] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:58.741617 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 69da799c886841a19002d3c7c5084e73 (table default.Table_c2e56e5918cb4007b681c71977325233 [id=0cf4bae5f10d45c1a41880b45fa0b0f0]) successfully deleted
I20251027 21:07:58.741695 10041 tablet_replica.cc:333] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:58.741868 10041 raft_consensus.cc:2243] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:58.742369 9305 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41710 (request call id 843) took 1088 ms. Trace:
I20251027 21:07:58.742719 10041 raft_consensus.cc:2272] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:58.742465 9305 rpcz_store.cc:276] 1027 21:07:57.654282 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.866462 (+212180us) service_pool.cc:225] Handling call
1027 21:07:57.867206 (+ 744us) catalog_manager.cc:2101] Acquired catalog manager lock
1027 21:07:57.867625 (+ 419us) catalog_manager.cc:2147] Created new table and tablet info
1027 21:07:58.742040 (+874415us) catalog_manager.cc:2207] Wrote table and tablets to system table
1027 21:07:58.742196 (+ 156us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1027 21:07:58.742326 (+ 130us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:57.883841 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:07:57.883951 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:07:57.883958 (+ 7us) write_op.cc:435] Acquired schema lock
1027 21:07:57.883960 (+ 2us) tablet.cc:660] Decoding operations
1027 21:07:57.894270 (+ 10310us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:07:57.894280 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1027 21:07:57.894286 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:07:57.894351 (+ 65us) tablet.cc:699] Row locks acquired
1027 21:07:57.894354 (+ 3us) write_op.cc:260] PREPARE: finished
1027 21:07:57.894385 (+ 31us) write_op.cc:270] Start()
1027 21:07:57.894414 (+ 29us) write_op.cc:276] Timestamp: P: 1761599277894371 usec, L: 0
1027 21:07:57.894417 (+ 3us) op_driver.cc:348] REPLICATION: starting
1027 21:07:57.894571 (+ 154us) log.cc:844] Serialized 433 byte log entry
1027 21:07:58.738978 (+844407us) op_driver.cc:464] REPLICATION: finished
1027 21:07:58.740707 (+ 1729us) spinlock_profiling.cc:243] Waited 1.09 ms on lock 0x619000100158. stack: 00007f956e0a89f6 00007f956e0a87b0 00007f956d0843ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f957b4660f4 00007f957b32d0e1 00007f957b32e24e 00007f957b331ccc 00007f957b3319f1 00007f9581d22fbd 00007f9579f80b9c 00007f9579f0b8b9 00007f9579f4cd67 00007f9579e8dffc
1027 21:07:58.740885 (+ 178us) write_op.cc:301] APPLY: starting
1027 21:07:58.740898 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
1027 21:07:58.740928 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
1027 21:07:58.740931 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:07:58.741034 (+ 103us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:07:58.741060 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1027 21:07:58.741066 (+ 6us) write_op.cc:312] APPLY: finished
1027 21:07:58.741136 (+ 70us) log.cc:844] Serialized 29 byte log entry
1027 21:07:58.741187 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.741202 (+ 15us) write_op.cc:454] Released schema lock
1027 21:07:58.741207 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1027 21:07:58.741266 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.741269 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":115,"apply.run_cpu_time_us":384,"apply.run_wall_time_us":384,"num_ops":2,"prepare.queue_time_us":15844,"prepare.run_cpu_time_us":621,"prepare.run_wall_time_us":10813,"replication_time_us":844536,"spinlock_wait_cycles":2497792}]]}
I20251027 21:07:58.743434 9312 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41710 (request call id 844) took 1088 ms. Trace:
I20251027 21:07:58.743732 10029 tablet_bootstrap.cc:492] T 69da799c886841a19002d3c7c5084e73 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:58.743546 9312 rpcz_store.cc:276] 1027 21:07:57.655392 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.868390 (+212998us) service_pool.cc:225] Handling call
1027 21:07:57.869023 (+ 633us) catalog_manager.cc:2101] Acquired catalog manager lock
1027 21:07:57.869403 (+ 380us) catalog_manager.cc:2147] Created new table and tablet info
1027 21:07:58.743251 (+873848us) catalog_manager.cc:2207] Wrote table and tablets to system table
1027 21:07:58.743329 (+ 78us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1027 21:07:58.743405 (+ 76us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:57.894684 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:07:57.894781 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:07:57.894786 (+ 5us) write_op.cc:435] Acquired schema lock
1027 21:07:57.894788 (+ 2us) tablet.cc:660] Decoding operations
1027 21:07:57.894864 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:07:57.894873 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1027 21:07:57.894877 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:07:57.894970 (+ 93us) tablet.cc:699] Row locks acquired
1027 21:07:57.894972 (+ 2us) write_op.cc:260] PREPARE: finished
1027 21:07:57.895010 (+ 38us) write_op.cc:270] Start()
1027 21:07:57.895054 (+ 44us) write_op.cc:276] Timestamp: P: 1761599277894993 usec, L: 0
1027 21:07:57.895058 (+ 4us) op_driver.cc:348] REPLICATION: starting
1027 21:07:57.895229 (+ 171us) log.cc:844] Serialized 433 byte log entry
1027 21:07:58.740794 (+845565us) op_driver.cc:464] REPLICATION: finished
1027 21:07:58.741324 (+ 530us) write_op.cc:301] APPLY: starting
1027 21:07:58.741334 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1027 21:07:58.741362 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
1027 21:07:58.741392 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:07:58.741479 (+ 87us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:07:58.741504 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1027 21:07:58.741511 (+ 7us) write_op.cc:312] APPLY: finished
1027 21:07:58.741567 (+ 56us) log.cc:844] Serialized 29 byte log entry
1027 21:07:58.741621 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.741635 (+ 14us) write_op.cc:454] Released schema lock
1027 21:07:58.741642 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1027 21:07:58.741699 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.741703 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":496,"apply.run_cpu_time_us":368,"apply.run_wall_time_us":367,"num_ops":2,"prepare.queue_time_us":24929,"prepare.run_cpu_time_us":619,"prepare.run_wall_time_us":617,"replication_time_us":845714}]]}
I20251027 21:07:58.744112 9312 master_service.cc:954] ReplaceTablet: received request to replace tablet 23673513189f47638dbbc737da0991d1 from {username='slave'} at 127.0.0.1:41710
W20251027 21:07:58.744501 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b99232a6018a4b1a850da954c551d38d consensus state has no leader
I20251027 21:07:58.744655 9303 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:41710 (request call id 837) took 1158 ms. Trace:
I20251027 21:07:58.743520 10041 ts_tablet_manager.cc:1916] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:58.744737 9303 rpcz_store.cc:276] 1027 21:07:57.586366 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.586411 (+ 45us) service_pool.cc:225] Handling call
1027 21:07:57.873190 (+286779us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1027 21:07:57.873389 (+ 199us) catalog_manager.cc:3708] Apply alter schema
1027 21:07:57.873553 (+ 164us) catalog_manager.cc:3740] Acquired catalog manager lock
1027 21:07:57.873642 (+ 89us) catalog_manager.cc:3898] Updating metadata on disk
1027 21:07:58.744321 (+870679us) catalog_manager.cc:3929] Committing alterations to in-memory state
1027 21:07:58.744608 (+ 287us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:57.895333 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:07:57.895441 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:07:57.895448 (+ 7us) write_op.cc:435] Acquired schema lock
1027 21:07:57.895452 (+ 4us) tablet.cc:660] Decoding operations
1027 21:07:57.895537 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:07:57.895548 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1027 21:07:57.895556 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
1027 21:07:57.895605 (+ 49us) tablet.cc:699] Row locks acquired
1027 21:07:57.895609 (+ 4us) write_op.cc:260] PREPARE: finished
1027 21:07:57.895640 (+ 31us) write_op.cc:270] Start()
1027 21:07:57.895667 (+ 27us) write_op.cc:276] Timestamp: P: 1761599277895624 usec, L: 0
1027 21:07:57.895682 (+ 15us) op_driver.cc:348] REPLICATION: starting
1027 21:07:57.895827 (+ 145us) log.cc:844] Serialized 405 byte log entry
1027 21:07:58.740864 (+845037us) op_driver.cc:464] REPLICATION: finished
1027 21:07:58.741756 (+ 892us) write_op.cc:301] APPLY: starting
1027 21:07:58.741767 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
1027 21:07:58.741790 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
1027 21:07:58.741807 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:07:58.741839 (+ 32us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:07:58.741859 (+ 20us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:07:58.741865 (+ 6us) write_op.cc:312] APPLY: finished
1027 21:07:58.741920 (+ 55us) log.cc:844] Serialized 23 byte log entry
1027 21:07:58.741959 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.741969 (+ 10us) write_op.cc:454] Released schema lock
1027 21:07:58.741974 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":859,"apply.run_cpu_time_us":274,"apply.run_wall_time_us":273,"num_ops":1,"prepare.queue_time_us":21322,"prepare.run_cpu_time_us":566,"prepare.run_wall_time_us":565,"replication_time_us":845157,"threads_started":1}]]}
I20251027 21:07:58.745972 9754 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 638) took 1204 ms. Trace:
I20251027 21:07:58.747004 10032 tablet_bootstrap.cc:492] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:58.747005 9308 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41710 (request call id 842) took 1093 ms. Trace:
I20251027 21:07:58.747164 10032 ts_tablet_manager.cc:1403] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.048s user 0.000s sys 0.003s
I20251027 21:07:58.747107 9308 rpcz_store.cc:276] 1027 21:07:57.653262 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.864450 (+211188us) service_pool.cc:225] Handling call
1027 21:07:57.865190 (+ 740us) catalog_manager.cc:2101] Acquired catalog manager lock
1027 21:07:57.865597 (+ 407us) catalog_manager.cc:2147] Created new table and tablet info
1027 21:07:58.746776 (+881179us) catalog_manager.cc:2207] Wrote table and tablets to system table
1027 21:07:58.746896 (+ 120us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1027 21:07:58.746974 (+ 78us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:57.883299 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:07:57.883411 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:07:57.883415 (+ 4us) write_op.cc:435] Acquired schema lock
1027 21:07:57.883417 (+ 2us) tablet.cc:660] Decoding operations
1027 21:07:57.883480 (+ 63us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:07:57.883488 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1027 21:07:57.883492 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:07:57.883567 (+ 75us) tablet.cc:699] Row locks acquired
1027 21:07:57.883569 (+ 2us) write_op.cc:260] PREPARE: finished
1027 21:07:57.883595 (+ 26us) write_op.cc:270] Start()
1027 21:07:57.883616 (+ 21us) write_op.cc:276] Timestamp: P: 1761599277883581 usec, L: 0
1027 21:07:57.883619 (+ 3us) op_driver.cc:348] REPLICATION: starting
1027 21:07:57.883744 (+ 125us) log.cc:844] Serialized 433 byte log entry
1027 21:07:58.733738 (+849994us) op_driver.cc:464] REPLICATION: finished
1027 21:07:58.737679 (+ 3941us) write_op.cc:301] APPLY: starting
1027 21:07:58.737691 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
1027 21:07:58.737748 (+ 57us) tablet.cc:1368] finished BulkCheckPresence
1027 21:07:58.737778 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:07:58.737904 (+ 126us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:07:58.737939 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1027 21:07:58.737947 (+ 8us) write_op.cc:312] APPLY: finished
1027 21:07:58.738014 (+ 67us) log.cc:844] Serialized 29 byte log entry
1027 21:07:58.740004 (+ 1990us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.740026 (+ 22us) write_op.cc:454] Released schema lock
1027 21:07:58.740036 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3883,"num_ops":2,"prepare.queue_time_us":17342,"prepare.run_cpu_time_us":488,"prepare.run_wall_time_us":487,"replication_time_us":850091,"thread_start_us":5143,"threads_started":1}]]}
I20251027 21:07:58.748008 10032 raft_consensus.cc:359] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.751942 10560 raft_consensus.cc:493] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:58.752089 10560 raft_consensus.cc:515] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.752761 10560 leader_election.cc:290] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:58.753557 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f1129eecfb54448965222269d9c32b7" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:58.754273 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c400101d2a39490e83722500839198cd" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:58.754351 9859 leader_election.cc:343] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.754773 10032 raft_consensus.cc:385] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.754925 10032 raft_consensus.cc:740] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
W20251027 21:07:58.755357 9859 leader_election.cc:343] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.755528 9859 leader_election.cc:304] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:58.755777 10560 raft_consensus.cc:2749] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:07:58.756986 9859 leader_election.cc:343] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.757634 10031 tablet_bootstrap.cc:492] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:58.757793 10031 ts_tablet_manager.cc:1403] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.139s user 0.003s sys 0.002s
I20251027 21:07:58.755208 10032 consensus_queue.cc:260] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:58.758431 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93a3b22cdc1543aebcc7fe72f2f2c0ae consensus state has no leader
I20251027 21:07:58.758522 10032 ts_tablet_manager.cc:1434] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251027 21:07:58.758553 9304 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41710 (request call id 839) took 1130 ms. Trace:
I20251027 21:07:58.758589 9311 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:41710 (request call id 838) took 1130 ms. Trace:
I20251027 21:07:58.758891 10032 tablet_bootstrap.cc:492] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:58.758661 9304 rpcz_store.cc:276] 1027 21:07:57.628482 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.628984 (+ 502us) service_pool.cc:225] Handling call
1027 21:07:57.883036 (+254052us) catalog_manager.cc:2101] Acquired catalog manager lock
1027 21:07:57.883423 (+ 387us) catalog_manager.cc:2147] Created new table and tablet info
1027 21:07:58.758322 (+874899us) catalog_manager.cc:2207] Wrote table and tablets to system table
1027 21:07:58.758427 (+ 105us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1027 21:07:58.758510 (+ 83us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:57.897403 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:07:57.897541 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:07:57.897546 (+ 5us) write_op.cc:435] Acquired schema lock
1027 21:07:57.897550 (+ 4us) tablet.cc:660] Decoding operations
1027 21:07:57.897638 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:07:57.897649 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1027 21:07:57.897656 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:07:57.897732 (+ 76us) tablet.cc:699] Row locks acquired
1027 21:07:57.897736 (+ 4us) write_op.cc:260] PREPARE: finished
1027 21:07:57.897764 (+ 28us) write_op.cc:270] Start()
1027 21:07:57.897797 (+ 33us) write_op.cc:276] Timestamp: P: 1761599277897749 usec, L: 0
1027 21:07:57.897802 (+ 5us) op_driver.cc:348] REPLICATION: starting
1027 21:07:57.898021 (+ 219us) log.cc:844] Serialized 433 byte log entry
1027 21:07:58.757171 (+859150us) op_driver.cc:464] REPLICATION: finished
1027 21:07:58.757906 (+ 735us) write_op.cc:301] APPLY: starting
1027 21:07:58.757918 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
1027 21:07:58.757946 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
1027 21:07:58.757973 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:07:58.758058 (+ 85us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:07:58.758088 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1027 21:07:58.758094 (+ 6us) write_op.cc:312] APPLY: finished
1027 21:07:58.758180 (+ 86us) log.cc:844] Serialized 29 byte log entry
1027 21:07:58.758233 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.758247 (+ 14us) write_op.cc:454] Released schema lock
1027 21:07:58.758252 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":700,"num_ops":2,"prepare.queue_time_us":13617,"prepare.run_cpu_time_us":683,"prepare.run_wall_time_us":681,"replication_time_us":859354}]]}
I20251027 21:07:58.759284 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_559e10c49ef84eeb960dcdb1ebddf817"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:58.759400 9304 master_service.cc:954] ReplaceTablet: received request to replace tablet 133e474f9133487f89c44dbfa6266ca7 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:58.758630 10031 raft_consensus.cc:359] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.759629 10031 raft_consensus.cc:385] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.759712 10031 raft_consensus.cc:740] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
W20251027 21:07:58.759873 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_559e10c49ef84eeb960dcdb1ebddf817 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:58.759908 10031 consensus_queue.cc:260] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.760583 10032 tablet_bootstrap.cc:654] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:58.760713 10031 ts_tablet_manager.cc:1434] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:07:58.761020 10031 tablet_bootstrap.cc:492] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:58.758704 9311 rpcz_store.cc:276] 1027 21:07:57.628312 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.628554 (+ 242us) service_pool.cc:225] Handling call
1027 21:07:57.880433 (+251879us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1027 21:07:57.880622 (+ 189us) catalog_manager.cc:3708] Apply alter schema
1027 21:07:57.880767 (+ 145us) catalog_manager.cc:3740] Acquired catalog manager lock
1027 21:07:57.880816 (+ 49us) catalog_manager.cc:3898] Updating metadata on disk
1027 21:07:58.758201 (+877385us) catalog_manager.cc:3929] Committing alterations to in-memory state
1027 21:07:58.758548 (+ 347us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:57.896569 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:07:57.896652 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:07:57.896657 (+ 5us) write_op.cc:435] Acquired schema lock
1027 21:07:57.896661 (+ 4us) tablet.cc:660] Decoding operations
1027 21:07:57.896739 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:07:57.896751 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1027 21:07:57.896757 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1027 21:07:57.896810 (+ 53us) tablet.cc:699] Row locks acquired
1027 21:07:57.896815 (+ 5us) write_op.cc:260] PREPARE: finished
1027 21:07:57.896859 (+ 44us) write_op.cc:270] Start()
1027 21:07:57.896882 (+ 23us) write_op.cc:276] Timestamp: P: 1761599277896831 usec, L: 0
1027 21:07:57.896887 (+ 5us) op_driver.cc:348] REPLICATION: starting
1027 21:07:57.897303 (+ 416us) log.cc:844] Serialized 405 byte log entry
1027 21:07:58.757123 (+859820us) op_driver.cc:464] REPLICATION: finished
1027 21:07:58.757440 (+ 317us) write_op.cc:301] APPLY: starting
1027 21:07:58.757454 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1027 21:07:58.757486 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
1027 21:07:58.757517 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:07:58.757566 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:07:58.757599 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:07:58.757605 (+ 6us) write_op.cc:312] APPLY: finished
1027 21:07:58.757671 (+ 66us) log.cc:844] Serialized 23 byte log entry
1027 21:07:58.757727 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.757742 (+ 15us) write_op.cc:454] Released schema lock
1027 21:07:58.757748 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1027 21:07:58.757815 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.757818 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":271,"apply.run_cpu_time_us":371,"apply.run_wall_time_us":371,"num_ops":1,"prepare.queue_time_us":15353,"prepare.run_cpu_time_us":566,"prepare.run_wall_time_us":796,"replication_time_us":860216}]]}
I20251027 21:07:58.761739 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3f5ab2fc1a934004816ae27b651671ed" } new_table_name: "default.Table_b02344913e414bada0cc6b5503d42c02" modify_external_catalogs: true
I20251027 21:07:58.762795 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_989e3ba3204f41f6b0d7f408eab2f4a7" } new_table_name: "default.Table_dd85b9fbcede45888d66f6226d2c5f2f" modify_external_catalogs: true
I20251027 21:07:58.763116 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f4c9a8d0ca5f4277b9ff989e1b832985"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:58.763288 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f1129eecfb54448965222269d9c32b7" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:58.763598 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4c9a8d0ca5f4277b9ff989e1b832985 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:58.763777 9860 leader_election.cc:343] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.763904 10029 tablet_bootstrap.cc:654] T 69da799c886841a19002d3c7c5084e73 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:58.763914 9860 leader_election.cc:304] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:58.764361 10411 raft_consensus.cc:2749] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:58.766371 10043 ts_tablet_manager.cc:1929] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:07:58.766530 10043 log.cc:1199] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/cfce0e5c837743ea969374244163be3e
I20251027 21:07:58.766945 10043 ts_tablet_manager.cc:1950] T cfce0e5c837743ea969374244163be3e P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:58.767692 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 604) took 2107 ms. Trace:
I20251027 21:07:58.767791 10043 rpcz_store.cc:276] 1027 21:07:56.660253 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.434723 (+774470us) service_pool.cc:225] Handling call
1027 21:07:58.676397 (+1241674us) ts_tablet_manager.cc:1151] Deleting tablet cfce0e5c837743ea969374244163be3e
1027 21:07:58.676408 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.730563 (+ 54155us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.763122 (+ 32559us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.767645 (+ 4523us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1241508}
I20251027 21:07:58.768191 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 605) took 2004 ms. Trace:
I20251027 21:07:58.768332 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet cfce0e5c837743ea969374244163be3e (table default.Table_4568794575f345c08a45afeb2d4f9648 [id=13a25b18846c4580a337b91cee116a3d]) successfully deleted
I20251027 21:07:58.768826 9310 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 995284883d834bbb9ef583b736ca8402 (table default.Table_140424c161ea4581bed312cf5deb5ab5 [id=186c9cce53d74040b9799ea386fdffd7]). Expected version 4 got 0
I20251027 21:07:58.770535 9306 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet d13d00353b6740b5bed5a153936d2059 (table default.Table_6b471834498c4afeb57279278d45d9a9 [id=f2edcf33b0b74923b4ddbbc2419e14ca]). Expected version 1 got 0
I20251027 21:07:58.772435 9307 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_318d41ccfb2c4d2c99bafcea875143a1 [id=2bc39b5bc02645ef830eee14e92707db] alter complete (version 1)
I20251027 21:07:58.748252 9754 rpcz_store.cc:276] 1027 21:07:57.528200 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.108365 (+580165us) service_pool.cc:225] Handling call
1027 21:07:58.208638 (+100273us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:58.208682 (+ 44us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:58.530736 (+322054us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.731613 (+200877us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:58.732434 (+ 821us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:58.775255 9748 rpcz_store.cc:276] 1027 21:07:57.527198 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.867902 (+340704us) service_pool.cc:225] Handling call
1027 21:07:58.049337 (+181435us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:07:58.049353 (+ 16us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:07:58.582783 (+533430us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.727117 (+144334us) tablet_replica.cc:206] Creating consensus instance
1027 21:07:58.731003 (+ 3886us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:07:58.768273 10043 rpcz_store.cc:276] 1027 21:07:56.763299 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.445228 (+681929us) service_pool.cc:225] Handling call
1027 21:07:58.768043 (+1322815us) ts_tablet_manager.cc:1151] Deleting tablet 07a9c3b7ef1548ec9d5ae9f20545f263
1027 21:07:58.768052 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.768141 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1322628}
I20251027 21:07:58.782326 9309 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41710 (request call id 836) took 1196 ms. Trace:
W20251027 21:07:58.782634 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 07a9c3b7ef1548ec9d5ae9f20545f263 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07a9c3b7ef1548ec9d5ae9f20545f263 already in progress: creating tablet
I20251027 21:07:58.782423 9309 rpcz_store.cc:276] 1027 21:07:57.586255 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.586411 (+ 156us) service_pool.cc:225] Handling call
1027 21:07:57.875594 (+289183us) catalog_manager.cc:2101] Acquired catalog manager lock
1027 21:07:57.875968 (+ 374us) catalog_manager.cc:2147] Created new table and tablet info
1027 21:07:58.782059 (+906091us) catalog_manager.cc:2207] Wrote table and tablets to system table
1027 21:07:58.782159 (+ 100us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1027 21:07:58.782273 (+ 114us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:57.895961 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:07:57.896082 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:07:57.896089 (+ 7us) write_op.cc:435] Acquired schema lock
1027 21:07:57.896093 (+ 4us) tablet.cc:660] Decoding operations
1027 21:07:57.896179 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:07:57.896188 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1027 21:07:57.896195 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:07:57.896261 (+ 66us) tablet.cc:699] Row locks acquired
1027 21:07:57.896265 (+ 4us) write_op.cc:260] PREPARE: finished
1027 21:07:57.896304 (+ 39us) write_op.cc:270] Start()
1027 21:07:57.896328 (+ 24us) write_op.cc:276] Timestamp: P: 1761599277896290 usec, L: 0
1027 21:07:57.896331 (+ 3us) op_driver.cc:348] REPLICATION: starting
1027 21:07:57.896470 (+ 139us) log.cc:844] Serialized 433 byte log entry
1027 21:07:58.757045 (+860575us) op_driver.cc:464] REPLICATION: finished
1027 21:07:58.757331 (+ 286us) write_op.cc:301] APPLY: starting
1027 21:07:58.757345 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1027 21:07:58.757393 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1027 21:07:58.757465 (+ 72us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:07:58.773675 (+ 16210us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:07:58.773701 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1027 21:07:58.773708 (+ 7us) write_op.cc:312] APPLY: finished
1027 21:07:58.776314 (+ 2606us) log.cc:844] Serialized 29 byte log entry
1027 21:07:58.781900 (+ 5586us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.781918 (+ 18us) write_op.cc:454] Released schema lock
1027 21:07:58.781924 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1027 21:07:58.782023 (+ 99us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.782027 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":107776,"child_traces":[["op",{"apply.queue_time_us":201,"apply.run_cpu_time_us":701,"apply.run_wall_time_us":24672,"num_ops":2,"prepare.queue_time_us":19608,"prepare.run_cpu_time_us":578,"prepare.run_wall_time_us":578,"replication_time_us":860682}]]}
I20251027 21:07:58.783226 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 615) took 1983 ms. Trace:
I20251027 21:07:58.784226 10043 rpcz_store.cc:276] 1027 21:07:56.799587 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.491022 (+691435us) service_pool.cc:225] Handling call
1027 21:07:58.783013 (+1291991us) ts_tablet_manager.cc:1151] Deleting tablet 8fc2d7d6601d4cc2849064f4cdddc90d
1027 21:07:58.783028 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.783160 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1291664}
W20251027 21:07:58.786031 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 8fc2d7d6601d4cc2849064f4cdddc90d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fc2d7d6601d4cc2849064f4cdddc90d already in progress: creating tablet
I20251027 21:07:58.785284 10029 tablet_bootstrap.cc:492] T 69da799c886841a19002d3c7c5084e73 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:58.786463 10029 ts_tablet_manager.cc:1403] T 69da799c886841a19002d3c7c5084e73 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.002s
I20251027 21:07:58.787326 10029 raft_consensus.cc:359] T 69da799c886841a19002d3c7c5084e73 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.787568 10029 raft_consensus.cc:385] T 69da799c886841a19002d3c7c5084e73 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.787699 10029 raft_consensus.cc:740] T 69da799c886841a19002d3c7c5084e73 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:58.787734 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5cd332429fd147e1b99e2fed13e0a0e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:58.788100 10029 consensus_queue.cc:260] T 69da799c886841a19002d3c7c5084e73 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:58.788255 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5cd332429fd147e1b99e2fed13e0a0e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:58.788812 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 133e474f9133487f89c44dbfa6266ca7
I20251027 21:07:58.789269 9304 catalog_manager.cc:6719] ReplaceTablet: tablet 133e474f9133487f89c44dbfa6266ca7 deleted and replaced by tablet 6cd39721fd28422896d517d349009c66
I20251027 21:07:58.789397 9304 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:41710 (request call id 849) took 1091 ms. Trace:
I20251027 21:07:58.789476 9304 rpcz_store.cc:276] 1027 21:07:57.697649 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.759337 (+1061688us) service_pool.cc:225] Handling call
1027 21:07:58.789348 (+ 30011us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:58.761407 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:07:58.761551 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:07:58.761559 (+ 8us) write_op.cc:435] Acquired schema lock
1027 21:07:58.761563 (+ 4us) tablet.cc:660] Decoding operations
1027 21:07:58.761699 (+ 136us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:07:58.761710 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1027 21:07:58.761716 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:07:58.761786 (+ 70us) tablet.cc:699] Row locks acquired
1027 21:07:58.761789 (+ 3us) write_op.cc:260] PREPARE: finished
1027 21:07:58.761823 (+ 34us) write_op.cc:270] Start()
1027 21:07:58.761849 (+ 26us) write_op.cc:276] Timestamp: P: 1761599278761790 usec, L: 0
1027 21:07:58.761852 (+ 3us) op_driver.cc:348] REPLICATION: starting
1027 21:07:58.762038 (+ 186us) log.cc:844] Serialized 622 byte log entry
1027 21:07:58.787573 (+ 25535us) op_driver.cc:464] REPLICATION: finished
1027 21:07:58.788143 (+ 570us) write_op.cc:301] APPLY: starting
1027 21:07:58.788157 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1027 21:07:58.788209 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
1027 21:07:58.788232 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:07:58.788324 (+ 92us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:07:58.788381 (+ 57us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:07:58.788393 (+ 12us) write_op.cc:312] APPLY: finished
1027 21:07:58.788478 (+ 85us) log.cc:844] Serialized 29 byte log entry
1027 21:07:58.788533 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.788548 (+ 15us) write_op.cc:454] Released schema lock
1027 21:07:58.788554 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":498,"num_ops":2,"prepare.queue_time_us":952,"prepare.run_cpu_time_us":771,"prepare.run_wall_time_us":771,"raft.queue_time_us":4233,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":254,"replication_time_us":25690,"spinlock_wait_cycles":77568}]]}
I20251027 21:07:58.790244 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b5a125355aa246cda7ea2d7a0b2acf62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:58.790743 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5a125355aa246cda7ea2d7a0b2acf62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:58.790843 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 133e474f9133487f89c44dbfa6266ca7 with delete_type TABLET_DATA_DELETED (replaced by tablet 6cd39721fd28422896d517d349009c66) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:58.791630 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 133e474f9133487f89c44dbfa6266ca7 with delete_type TABLET_DATA_DELETED (replaced by tablet 6cd39721fd28422896d517d349009c66) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:58.792128 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 133e474f9133487f89c44dbfa6266ca7 with delete_type TABLET_DATA_DELETED (replaced by tablet 6cd39721fd28422896d517d349009c66) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:58.792963 10029 ts_tablet_manager.cc:1434] T 69da799c886841a19002d3c7c5084e73 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251027 21:07:58.793704 9303 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41710 (request call id 847) took 1134 ms. Trace:
I20251027 21:07:58.793803 9303 rpcz_store.cc:276] 1027 21:07:57.658849 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.745386 (+1086537us) service_pool.cc:225] Handling call
1027 21:07:58.759207 (+ 13821us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x616000051154. stack: 00007f956e0a89f6 00007f956e0a87b0 00007f956d0843ad 0000000000983f04 0000000000983ecd 00007f957d3af6bd 00007f957d3af7b3 00007f9579f76441 00007f9581c309dd 00007f9581dbd263 00007f95788cd7c6 00007f95788cd4f5 00007f957504727e 00007f957504615e 00007f957504c395 00007f957504ccbf
1027 21:07:58.759965 (+ 758us) catalog_manager.cc:2101] Acquired catalog manager lock
1027 21:07:58.760538 (+ 573us) catalog_manager.cc:2147] Created new table and tablet info
1027 21:07:58.793456 (+ 32918us) catalog_manager.cc:2207] Wrote table and tablets to system table
1027 21:07:58.793554 (+ 98us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1027 21:07:58.793639 (+ 85us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:58.762207 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:07:58.762296 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:07:58.762301 (+ 5us) write_op.cc:435] Acquired schema lock
1027 21:07:58.762303 (+ 2us) tablet.cc:660] Decoding operations
1027 21:07:58.762373 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:07:58.762380 (+ 7us) write_op.cc:641] Partition lock acquired for write op
1027 21:07:58.762384 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:07:58.762460 (+ 76us) tablet.cc:699] Row locks acquired
1027 21:07:58.762462 (+ 2us) write_op.cc:260] PREPARE: finished
1027 21:07:58.762490 (+ 28us) write_op.cc:270] Start()
1027 21:07:58.762519 (+ 29us) write_op.cc:276] Timestamp: P: 1761599278762459 usec, L: 0
1027 21:07:58.762522 (+ 3us) op_driver.cc:348] REPLICATION: starting
1027 21:07:58.762759 (+ 237us) log.cc:844] Serialized 433 byte log entry
1027 21:07:58.787645 (+ 24886us) op_driver.cc:464] REPLICATION: finished
1027 21:07:58.792983 (+ 5338us) write_op.cc:301] APPLY: starting
1027 21:07:58.792998 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1027 21:07:58.793034 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
1027 21:07:58.793058 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:07:58.793188 (+ 130us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:07:58.793217 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1027 21:07:58.793223 (+ 6us) write_op.cc:312] APPLY: finished
1027 21:07:58.793293 (+ 70us) log.cc:844] Serialized 29 byte log entry
1027 21:07:58.793351 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.793370 (+ 19us) write_op.cc:454] Released schema lock
1027 21:07:58.793375 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1027 21:07:58.793511 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.793515 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":31076096,"child_traces":[["op",{"apply.queue_time_us":5301,"apply.run_cpu_time_us":460,"apply.run_wall_time_us":460,"num_ops":2,"prepare.queue_time_us":1230,"prepare.run_cpu_time_us":7480,"prepare.run_wall_time_us":31301,"raft.queue_time_us":1784,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":102,"replication_time_us":25104,"spinlock_wait_cycles":59648,"thread_start_us":24030,"threads_started":1}]]}
I20251027 21:07:58.794613 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet a4d26f73b8534f08bcaa386d4de75c01 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:58.797720 10029 tablet_bootstrap.cc:492] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:58.799005 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 23673513189f47638dbbc737da0991d1
I20251027 21:07:58.798029 10043 tablet_replica.cc:333] T 69da799c886841a19002d3c7c5084e73 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:58.799566 9312 catalog_manager.cc:6719] ReplaceTablet: tablet 23673513189f47638dbbc737da0991d1 deleted and replaced by tablet 050e9442041d495496bfa7ce25ac6a88
I20251027 21:07:58.799692 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 23673513189f47638dbbc737da0991d1 with delete_type TABLET_DATA_DELETED (replaced by tablet 050e9442041d495496bfa7ce25ac6a88) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:58.799703 9312 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:41710 (request call id 846) took 1142 ms. Trace:
I20251027 21:07:58.799793 9312 rpcz_store.cc:276] 1027 21:07:57.657192 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.744055 (+1086863us) service_pool.cc:225] Handling call
1027 21:07:58.757592 (+ 13537us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x616000051154. stack: 00007f956e0a89f6 00007f956e0a87b0 00007f956d0843ad 0000000000983f04 0000000000983ecd 00007f957d3af6bd 00007f957d3af7b3 00007f9579f76441 00007f9581c309dd 00007f9581dc9fab 00007f95788da107 00007f95788d9e35 00007f957504727e 00007f957504615e 00007f957504c395 00007f957504ccbf
1027 21:07:58.799656 (+ 42064us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:58.793586 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:07:58.793687 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:07:58.793692 (+ 5us) write_op.cc:435] Acquired schema lock
1027 21:07:58.793694 (+ 2us) tablet.cc:660] Decoding operations
1027 21:07:58.793790 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:07:58.793804 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1027 21:07:58.793808 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:07:58.793877 (+ 69us) tablet.cc:699] Row locks acquired
1027 21:07:58.793880 (+ 3us) write_op.cc:260] PREPARE: finished
1027 21:07:58.793930 (+ 50us) write_op.cc:270] Start()
1027 21:07:58.793964 (+ 34us) write_op.cc:276] Timestamp: P: 1761599278793896 usec, L: 0
1027 21:07:58.793968 (+ 4us) op_driver.cc:348] REPLICATION: starting
1027 21:07:58.794221 (+ 253us) log.cc:844] Serialized 622 byte log entry
1027 21:07:58.796980 (+ 2759us) op_driver.cc:464] REPLICATION: finished
1027 21:07:58.798403 (+ 1423us) write_op.cc:301] APPLY: starting
1027 21:07:58.798416 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
1027 21:07:58.798451 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
1027 21:07:58.798480 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:07:58.798562 (+ 82us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:07:58.798604 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:07:58.798611 (+ 7us) write_op.cc:312] APPLY: finished
1027 21:07:58.798667 (+ 56us) log.cc:844] Serialized 29 byte log entry
1027 21:07:58.798728 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.798745 (+ 17us) write_op.cc:454] Released schema lock
1027 21:07:58.798751 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":30000384,"child_traces":[["op",{"apply.queue_time_us":1376,"apply.run_cpu_time_us":413,"apply.run_wall_time_us":413,"num_ops":2,"prepare.queue_time_us":28750,"raft.queue_time_us":133,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":1878,"replication_time_us":2986}]]}
I20251027 21:07:58.800436 9312 master_service.cc:954] ReplaceTablet: received request to replace tablet 8b30052ff19744a9bf4bfaaa1bc77b36 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:58.801254 10029 tablet_bootstrap.cc:654] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:58.801435 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 23673513189f47638dbbc737da0991d1 with delete_type TABLET_DATA_DELETED (replaced by tablet 050e9442041d495496bfa7ce25ac6a88) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:58.799360 10043 raft_consensus.cc:2243] T 69da799c886841a19002d3c7c5084e73 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:58.804963 10043 raft_consensus.cc:2272] T 69da799c886841a19002d3c7c5084e73 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:58.806084 10043 ts_tablet_manager.cc:1916] T 69da799c886841a19002d3c7c5084e73 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:58.809415 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 90f96c21da9944f1ade05d35787d47e2 consensus state has no leader
I20251027 21:07:58.809581 9311 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:41710 (request call id 850) took 1094 ms. Trace:
I20251027 21:07:58.809684 9311 rpcz_store.cc:276] 1027 21:07:57.714773 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.761553 (+1046780us) service_pool.cc:225] Handling call
1027 21:07:58.761942 (+ 389us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1027 21:07:58.762315 (+ 373us) catalog_manager.cc:3708] Apply alter schema
1027 21:07:58.764865 (+ 2550us) catalog_manager.cc:3740] Acquired catalog manager lock
1027 21:07:58.764974 (+ 109us) catalog_manager.cc:3898] Updating metadata on disk
1027 21:07:58.809205 (+ 44231us) catalog_manager.cc:3929] Committing alterations to in-memory state
1027 21:07:58.809548 (+ 343us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:58.804552 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:07:58.804655 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:07:58.804659 (+ 4us) write_op.cc:435] Acquired schema lock
1027 21:07:58.804661 (+ 2us) tablet.cc:660] Decoding operations
1027 21:07:58.804743 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:07:58.804755 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1027 21:07:58.804760 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1027 21:07:58.804818 (+ 58us) tablet.cc:699] Row locks acquired
1027 21:07:58.804820 (+ 2us) write_op.cc:260] PREPARE: finished
1027 21:07:58.804849 (+ 29us) write_op.cc:270] Start()
1027 21:07:58.804874 (+ 25us) write_op.cc:276] Timestamp: P: 1761599278804816 usec, L: 0
1027 21:07:58.804878 (+ 4us) op_driver.cc:348] REPLICATION: starting
1027 21:07:58.805112 (+ 234us) log.cc:844] Serialized 405 byte log entry
1027 21:07:58.807530 (+ 2418us) op_driver.cc:464] REPLICATION: finished
1027 21:07:58.808790 (+ 1260us) write_op.cc:301] APPLY: starting
1027 21:07:58.808804 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1027 21:07:58.808837 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
1027 21:07:58.808866 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:07:58.808906 (+ 40us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:07:58.808939 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:07:58.808945 (+ 6us) write_op.cc:312] APPLY: finished
1027 21:07:58.809008 (+ 63us) log.cc:844] Serialized 23 byte log entry
1027 21:07:58.809092 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.809116 (+ 24us) write_op.cc:454] Released schema lock
1027 21:07:58.809122 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1027 21:07:58.809539 (+ 417us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.809547 (+ 8us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1213,"apply.run_cpu_time_us":402,"apply.run_wall_time_us":401,"num_ops":1,"prepare.queue_time_us":39153,"prepare.run_cpu_time_us":706,"prepare.run_wall_time_us":4985,"raft.queue_time_us":56,"raft.run_cpu_time_us":384,"raft.run_wall_time_us":3440,"replication_time_us":2628,"spinlock_wait_cycles":70528}]]}
I20251027 21:07:58.810488 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0812ae8e053c4e6f913221ef573f618f" } modify_external_catalogs: true
I20251027 21:07:58.810664 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0812ae8e053c4e6f913221ef573f618f" } modify_external_catalogs: true
I20251027 21:07:58.811165 10032 tablet_bootstrap.cc:492] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:58.811386 10032 ts_tablet_manager.cc:1403] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.000s
I20251027 21:07:58.812201 10032 raft_consensus.cc:359] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.812428 10032 raft_consensus.cc:385] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.812563 10032 raft_consensus.cc:740] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:58.813582 10032 consensus_queue.cc:260] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.814224 10032 ts_tablet_manager.cc:1434] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:07:58.814704 9305 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41710 (request call id 845) took 1157 ms. Trace:
I20251027 21:07:58.814839 9305 rpcz_store.cc:276] 1027 21:07:57.656813 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.743025 (+1086212us) service_pool.cc:225] Handling call
1027 21:07:58.763073 (+ 20048us) spinlock_profiling.cc:243] Waited 19.8 ms on lock 0x616000051154. stack: 00007f956e0a89f6 00007f956e0a87b0 00007f956d0843ad 0000000000983f04 0000000000983ecd 00007f957d3af6bd 00007f957d3af7b3 00007f9579f76441 00007f9581c309dd 00007f9581dbd263 00007f95788cd7c6 00007f95788cd4f5 00007f957504727e 00007f957504615e 00007f957504c395 00007f957504ccbf
1027 21:07:58.766526 (+ 3453us) catalog_manager.cc:2101] Acquired catalog manager lock
1027 21:07:58.766947 (+ 421us) catalog_manager.cc:2147] Created new table and tablet info
1027 21:07:58.814486 (+ 47539us) catalog_manager.cc:2207] Wrote table and tablets to system table
1027 21:07:58.814584 (+ 98us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1027 21:07:58.814660 (+ 76us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:58.809634 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:07:58.809734 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:07:58.809741 (+ 7us) write_op.cc:435] Acquired schema lock
1027 21:07:58.809743 (+ 2us) tablet.cc:660] Decoding operations
1027 21:07:58.809822 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:07:58.809846 (+ 24us) write_op.cc:641] Partition lock acquired for write op
1027 21:07:58.809852 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:07:58.809946 (+ 94us) tablet.cc:699] Row locks acquired
1027 21:07:58.809948 (+ 2us) write_op.cc:260] PREPARE: finished
1027 21:07:58.809992 (+ 44us) write_op.cc:270] Start()
1027 21:07:58.810019 (+ 27us) write_op.cc:276] Timestamp: P: 1761599278809957 usec, L: 0
1027 21:07:58.810022 (+ 3us) op_driver.cc:348] REPLICATION: starting
1027 21:07:58.810209 (+ 187us) log.cc:844] Serialized 433 byte log entry
1027 21:07:58.813730 (+ 3521us) op_driver.cc:464] REPLICATION: finished
1027 21:07:58.814005 (+ 275us) write_op.cc:301] APPLY: starting
1027 21:07:58.814019 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1027 21:07:58.814056 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
1027 21:07:58.814078 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:07:58.814222 (+ 144us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:07:58.814257 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1027 21:07:58.814264 (+ 7us) write_op.cc:312] APPLY: finished
1027 21:07:58.814331 (+ 67us) log.cc:844] Serialized 29 byte log entry
1027 21:07:58.814391 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.814409 (+ 18us) write_op.cc:454] Released schema lock
1027 21:07:58.814415 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1027 21:07:58.814487 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.814491 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":45543296,"child_traces":[["op",{"apply.queue_time_us":223,"apply.run_cpu_time_us":477,"apply.run_wall_time_us":476,"num_ops":2,"prepare.queue_time_us":42326,"prepare.run_cpu_time_us":688,"prepare.run_wall_time_us":2539,"raft.queue_time_us":52,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":1233,"replication_time_us":3684}]]}
I20251027 21:07:58.815707 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_bc5ec745ebe54017b6bc9635581c813d" } modify_external_catalogs: true
I20251027 21:07:58.815927 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_bc5ec745ebe54017b6bc9635581c813d" } modify_external_catalogs: true
I20251027 21:07:58.814755 10032 tablet_bootstrap.cc:492] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:58.819450 9310 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_6b471834498c4afeb57279278d45d9a9 [id=f2edcf33b0b74923b4ddbbc2419e14ca] alter complete (version 1)
I20251027 21:07:58.820897 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6c3cfd4d761643dca83d16b2441b0c64" } modify_external_catalogs: true
I20251027 21:07:58.821055 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6c3cfd4d761643dca83d16b2441b0c64" } modify_external_catalogs: true
I20251027 21:07:58.821256 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 583fba7c750e4ac7b768da447002c211 with delete_type TABLET_DATA_DELETED (replaced by tablet f34512059f0c47fc9af7db2645f86b40) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:58.821372 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet b87e809ff897417c88e5edc19e2d591e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:58.821282 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet cfce0e5c837743ea969374244163be3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:58.823475 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet c400101d2a39490e83722500839198cd with delete_type TABLET_DATA_DELETED (replaced by tablet c9efffa9766e46aca8eb286fc2ad48d2) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:58.823534 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f1129eecfb54448965222269d9c32b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:58.823720 9896 tablet_service.cc:1552] Processing DeleteTablet for tablet 92a2393cff8044028c86c8cd2bccd930 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:58.825311 10032 tablet_bootstrap.cc:654] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:58.827157 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8ec40deea5fd4803a9ee3f6c295a7439" } modify_external_catalogs: true
I20251027 21:07:58.827332 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8ec40deea5fd4803a9ee3f6c295a7439" } modify_external_catalogs: true
I20251027 21:07:58.830543 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 6db0f31c12d14e66a4ec7fc722d62c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:58.830941 9307 catalog_manager.cc:5649] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67). New cstate: current_term: 1 leader_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: UNKNOWN } } }
I20251027 21:07:58.833340 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 9248f458baca409aa3505cc0db321b34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:58.834105 10031 tablet_bootstrap.cc:654] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:07:58.834828 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c3f4e692c08244afa2c79e0a075bbcdb consensus state has no leader
I20251027 21:07:58.834859 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet 23673513189f47638dbbc737da0991d1 with delete_type TABLET_DATA_DELETED (replaced by tablet 050e9442041d495496bfa7ce25ac6a88) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:58.834988 9308 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:41710 (request call id 848) took 1157 ms. Trace:
I20251027 21:07:58.834371 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 779b7f63b0fa4f9ab47cf5799b3e085d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:58.835122 9308 rpcz_store.cc:276] 1027 21:07:57.677715 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.747621 (+1069906us) service_pool.cc:225] Handling call
1027 21:07:58.762738 (+ 15117us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x616000051154. stack: 00007f956e0a89f6 00007f956e0a87b0 00007f956d0843ad 0000000000983f04 0000000000983ecd 00007f957d3af6bd 00007f957d3af7b3 00007f9579f76441 00007f9581c309dd 00007f9581dbf4f6 00007f95788d1146 00007f95788d0e75 00007f957504727e 00007f957504615e 00007f957504c395 00007f957504ccbf
1027 21:07:58.765432 (+ 2694us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1027 21:07:58.765660 (+ 228us) catalog_manager.cc:3708] Apply alter schema
1027 21:07:58.765784 (+ 124us) catalog_manager.cc:3740] Acquired catalog manager lock
1027 21:07:58.765843 (+ 59us) catalog_manager.cc:3898] Updating metadata on disk
1027 21:07:58.834619 (+ 68776us) catalog_manager.cc:3929] Committing alterations to in-memory state
1027 21:07:58.834956 (+ 337us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:58.818345 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:07:58.821744 (+ 3399us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:07:58.821749 (+ 5us) write_op.cc:435] Acquired schema lock
1027 21:07:58.821751 (+ 2us) tablet.cc:660] Decoding operations
1027 21:07:58.821831 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:07:58.821841 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1027 21:07:58.821845 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1027 21:07:58.821891 (+ 46us) tablet.cc:699] Row locks acquired
1027 21:07:58.821894 (+ 3us) write_op.cc:260] PREPARE: finished
1027 21:07:58.821921 (+ 27us) write_op.cc:270] Start()
1027 21:07:58.821945 (+ 24us) write_op.cc:276] Timestamp: P: 1761599278821888 usec, L: 0
1027 21:07:58.821948 (+ 3us) op_driver.cc:348] REPLICATION: starting
1027 21:07:58.824336 (+ 2388us) log.cc:844] Serialized 405 byte log entry
1027 21:07:58.834046 (+ 9710us) op_driver.cc:464] REPLICATION: finished
1027 21:07:58.834213 (+ 167us) write_op.cc:301] APPLY: starting
1027 21:07:58.834225 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
1027 21:07:58.834257 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
1027 21:07:58.834286 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:07:58.834334 (+ 48us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:07:58.834368 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:07:58.834379 (+ 11us) write_op.cc:312] APPLY: finished
1027 21:07:58.834470 (+ 91us) log.cc:844] Serialized 23 byte log entry
1027 21:07:58.834527 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.834542 (+ 15us) write_op.cc:454] Released schema lock
1027 21:07:58.834548 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1027 21:07:58.834614 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.834617 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":34072832,"child_traces":[["op",{"apply.queue_time_us":118,"apply.run_cpu_time_us":394,"apply.run_wall_time_us":394,"num_ops":1,"prepare.queue_time_us":48252,"prepare.run_cpu_time_us":636,"prepare.run_wall_time_us":14968,"raft.queue_time_us":106,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":247,"replication_time_us":12074}]]}
I20251027 21:07:58.839547 10043 ts_tablet_manager.cc:1929] T 69da799c886841a19002d3c7c5084e73 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:58.839672 10043 log.cc:1199] T 69da799c886841a19002d3c7c5084e73 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/69da799c886841a19002d3c7c5084e73
I20251027 21:07:58.840065 10043 ts_tablet_manager.cc:1950] T 69da799c886841a19002d3c7c5084e73 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:58.840912 9309 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41710 (request call id 851) took 1125 ms. Trace:
I20251027 21:07:58.841385 10032 tablet_bootstrap.cc:492] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:58.841539 10032 ts_tablet_manager.cc:1403] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20251027 21:07:58.841025 9309 rpcz_store.cc:276] 1027 21:07:57.714886 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.783335 (+1068449us) service_pool.cc:225] Handling call
1027 21:07:58.787660 (+ 4325us) spinlock_profiling.cc:243] Waited 4.06 ms on lock 0x616000051154. stack: 00007f956e0a89f6 00007f956e0a87b0 00007f956d0843ad 0000000000983f04 0000000000983ecd 00007f957d3af6bd 00007f957d3af7b3 00007f9579f76441 00007f9581c309dd 00007f9581dbd263 00007f95788cd7c6 00007f95788cd4f5 00007f957504727e 00007f957504615e 00007f957504c395 00007f957504ccbf
1027 21:07:58.788353 (+ 693us) catalog_manager.cc:2101] Acquired catalog manager lock
1027 21:07:58.788878 (+ 525us) catalog_manager.cc:2147] Created new table and tablet info
1027 21:07:58.840602 (+ 51724us) catalog_manager.cc:2207] Wrote table and tablets to system table
1027 21:07:58.840722 (+ 120us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1027 21:07:58.840823 (+ 101us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:58.833350 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:07:58.833452 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:07:58.833457 (+ 5us) write_op.cc:435] Acquired schema lock
1027 21:07:58.833459 (+ 2us) tablet.cc:660] Decoding operations
1027 21:07:58.836952 (+ 3493us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:07:58.836961 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1027 21:07:58.836964 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:07:58.837035 (+ 71us) tablet.cc:699] Row locks acquired
1027 21:07:58.837039 (+ 4us) write_op.cc:260] PREPARE: finished
1027 21:07:58.837071 (+ 32us) write_op.cc:270] Start()
1027 21:07:58.837110 (+ 39us) write_op.cc:276] Timestamp: P: 1761599278837036 usec, L: 0
1027 21:07:58.837113 (+ 3us) op_driver.cc:348] REPLICATION: starting
1027 21:07:58.837266 (+ 153us) log.cc:844] Serialized 433 byte log entry
1027 21:07:58.840013 (+ 2747us) op_driver.cc:464] REPLICATION: finished
1027 21:07:58.840163 (+ 150us) write_op.cc:301] APPLY: starting
1027 21:07:58.840177 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1027 21:07:58.840213 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
1027 21:07:58.840236 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:07:58.840327 (+ 91us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:07:58.840363 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1027 21:07:58.840369 (+ 6us) write_op.cc:312] APPLY: finished
1027 21:07:58.840440 (+ 71us) log.cc:844] Serialized 29 byte log entry
1027 21:07:58.840499 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.840515 (+ 16us) write_op.cc:454] Released schema lock
1027 21:07:58.840521 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1027 21:07:58.840588 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.840592 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":9330944,"child_traces":[["op",{"apply.queue_time_us":104,"apply.run_cpu_time_us":418,"apply.run_wall_time_us":418,"num_ops":2,"prepare.queue_time_us":44009,"prepare.run_cpu_time_us":615,"prepare.run_wall_time_us":6064,"raft.queue_time_us":1777,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":1121,"replication_time_us":2876}]]}
I20251027 21:07:58.845867 9304 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41710 (request call id 852) took 1130 ms. Trace:
I20251027 21:07:58.846540 10029 tablet_bootstrap.cc:492] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:58.845980 9304 rpcz_store.cc:276] 1027 21:07:57.715004 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.790012 (+1075008us) service_pool.cc:225] Handling call
1027 21:07:58.790834 (+ 822us) catalog_manager.cc:2101] Acquired catalog manager lock
1027 21:07:58.791275 (+ 441us) catalog_manager.cc:2147] Created new table and tablet info
1027 21:07:58.845611 (+ 54336us) catalog_manager.cc:2207] Wrote table and tablets to system table
1027 21:07:58.845725 (+ 114us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1027 21:07:58.845806 (+ 81us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:58.839448 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:07:58.841819 (+ 2371us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:07:58.841824 (+ 5us) write_op.cc:435] Acquired schema lock
1027 21:07:58.841826 (+ 2us) tablet.cc:660] Decoding operations
1027 21:07:58.841896 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:07:58.841905 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1027 21:07:58.841908 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:07:58.841966 (+ 58us) tablet.cc:699] Row locks acquired
1027 21:07:58.841968 (+ 2us) write_op.cc:260] PREPARE: finished
1027 21:07:58.841996 (+ 28us) write_op.cc:270] Start()
1027 21:07:58.842018 (+ 22us) write_op.cc:276] Timestamp: P: 1761599278841981 usec, L: 0
1027 21:07:58.842021 (+ 3us) op_driver.cc:348] REPLICATION: starting
1027 21:07:58.842206 (+ 185us) log.cc:844] Serialized 433 byte log entry
1027 21:07:58.845065 (+ 2859us) op_driver.cc:464] REPLICATION: finished
1027 21:07:58.845210 (+ 145us) write_op.cc:301] APPLY: starting
1027 21:07:58.845225 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1027 21:07:58.845274 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
1027 21:07:58.845301 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:07:58.845382 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:07:58.845409 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1027 21:07:58.845416 (+ 7us) write_op.cc:312] APPLY: finished
1027 21:07:58.845468 (+ 52us) log.cc:844] Serialized 29 byte log entry
1027 21:07:58.845521 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.845537 (+ 16us) write_op.cc:454] Released schema lock
1027 21:07:58.845543 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":84,"num_ops":2,"prepare.queue_time_us":47113,"prepare.run_cpu_time_us":601,"prepare.run_wall_time_us":4989,"raft.queue_time_us":2182,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":176,"replication_time_us":2997}]]}
I20251027 21:07:58.846747 10029 ts_tablet_manager.cc:1403] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.002s
I20251027 21:07:58.847651 10029 raft_consensus.cc:359] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.847911 10029 raft_consensus.cc:385] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.848030 10029 raft_consensus.cc:740] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:58.848340 10029 consensus_queue.cc:260] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.852545 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet a4d26f73b8534f08bcaa386d4de75c01
I20251027 21:07:58.853094 9303 catalog_manager.cc:6719] ReplaceTablet: tablet a4d26f73b8534f08bcaa386d4de75c01 deleted and replaced by tablet db2735604dc14039810df4ecd330c671
I20251027 21:07:58.853205 9303 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:41710 (request call id 853) took 1130 ms. Trace:
I20251027 21:07:58.853477 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet a4d26f73b8534f08bcaa386d4de75c01 with delete_type TABLET_DATA_DELETED (replaced by tablet db2735604dc14039810df4ecd330c671) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:58.853307 9303 rpcz_store.cc:276] 1027 21:07:57.722609 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.794531 (+1071922us) service_pool.cc:225] Handling call
1027 21:07:58.853172 (+ 58641us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:58.844473 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:07:58.844558 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:07:58.844563 (+ 5us) write_op.cc:435] Acquired schema lock
1027 21:07:58.844565 (+ 2us) tablet.cc:660] Decoding operations
1027 21:07:58.848525 (+ 3960us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:07:58.848535 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1027 21:07:58.848540 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:07:58.848607 (+ 67us) tablet.cc:699] Row locks acquired
1027 21:07:58.848609 (+ 2us) write_op.cc:260] PREPARE: finished
1027 21:07:58.848638 (+ 29us) write_op.cc:270] Start()
1027 21:07:58.848665 (+ 27us) write_op.cc:276] Timestamp: P: 1761599278848622 usec, L: 0
1027 21:07:58.848669 (+ 4us) op_driver.cc:348] REPLICATION: starting
1027 21:07:58.848799 (+ 130us) log.cc:844] Serialized 622 byte log entry
1027 21:07:58.851826 (+ 3027us) op_driver.cc:464] REPLICATION: finished
1027 21:07:58.851952 (+ 126us) write_op.cc:301] APPLY: starting
1027 21:07:58.851965 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
1027 21:07:58.852000 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
1027 21:07:58.852029 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:07:58.852125 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:07:58.852172 (+ 47us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:07:58.852178 (+ 6us) write_op.cc:312] APPLY: finished
1027 21:07:58.852242 (+ 64us) log.cc:844] Serialized 29 byte log entry
1027 21:07:58.852291 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.852309 (+ 18us) write_op.cc:454] Released schema lock
1027 21:07:58.852314 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1027 21:07:58.852383 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.852387 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":86,"apply.run_cpu_time_us":423,"apply.run_wall_time_us":422,"num_ops":2,"prepare.queue_time_us":48943,"prepare.run_cpu_time_us":586,"prepare.run_wall_time_us":6732,"raft.queue_time_us":1941,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":596,"replication_time_us":3134}]]}
I20251027 21:07:58.854449 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet ed5619d295a246379b625f35692203e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:58.854477 9896 tablet_service.cc:1552] Processing DeleteTablet for tablet a4d26f73b8534f08bcaa386d4de75c01 with delete_type TABLET_DATA_DELETED (replaced by tablet db2735604dc14039810df4ecd330c671) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:58.854777 10029 ts_tablet_manager.cc:1434] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251027 21:07:58.855590 10040 ts_tablet_manager.cc:1929] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:58.855705 10040 log.cc:1199] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/019a892bfdd54e62bc3560a341d5c2da
I20251027 21:07:58.856395 10040 ts_tablet_manager.cc:1950] T 019a892bfdd54e62bc3560a341d5c2da P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:58.856662 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 382) took 4942 ms. Trace:
I20251027 21:07:58.857663 10040 rpcz_store.cc:276] 1027 21:07:53.914464 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:53.914527 (+ 63us) service_pool.cc:225] Handling call
1027 21:07:58.706381 (+4791854us) ts_tablet_manager.cc:1151] Deleting tablet 019a892bfdd54e62bc3560a341d5c2da
1027 21:07:58.706390 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.813136 (+106746us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.855219 (+ 42083us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.856636 (+ 1417us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4791640}
I20251027 21:07:58.858098 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet a4d26f73b8534f08bcaa386d4de75c01 with delete_type TABLET_DATA_DELETED (replaced by tablet db2735604dc14039810df4ecd330c671) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:58.859510 10029 tablet_bootstrap.cc:492] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:58.861428 10040 tablet_replica.cc:333] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:58.861634 10040 raft_consensus.cc:2243] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:58.861804 10040 raft_consensus.cc:2272] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:58.862349 10040 ts_tablet_manager.cc:1916] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:58.862687 10032 raft_consensus.cc:359] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.862964 10032 raft_consensus.cc:385] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.865262 10032 raft_consensus.cc:740] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:58.865384 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 8b30052ff19744a9bf4bfaaa1bc77b36
I20251027 21:07:58.865459 10029 tablet_bootstrap.cc:654] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:58.865950 9312 catalog_manager.cc:6719] ReplaceTablet: tablet 8b30052ff19744a9bf4bfaaa1bc77b36 deleted and replaced by tablet 283b72f626644ac2b9b61186387b3839
I20251027 21:07:58.866070 9312 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:41710 (request call id 854) took 1132 ms. Trace:
I20251027 21:07:58.866336 9896 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b30052ff19744a9bf4bfaaa1bc77b36 with delete_type TABLET_DATA_DELETED (replaced by tablet 283b72f626644ac2b9b61186387b3839) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:58.866623 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b30052ff19744a9bf4bfaaa1bc77b36 with delete_type TABLET_DATA_DELETED (replaced by tablet 283b72f626644ac2b9b61186387b3839) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:58.866631 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b30052ff19744a9bf4bfaaa1bc77b36 with delete_type TABLET_DATA_DELETED (replaced by tablet 283b72f626644ac2b9b61186387b3839) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:58.866775 10032 consensus_queue.cc:260] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.866468 9312 rpcz_store.cc:276] 1027 21:07:57.733352 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.800361 (+1067009us) service_pool.cc:225] Handling call
1027 21:07:58.866036 (+ 65675us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:07:58.851241 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:07:58.855898 (+ 4657us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:07:58.855903 (+ 5us) write_op.cc:435] Acquired schema lock
1027 21:07:58.855905 (+ 2us) tablet.cc:660] Decoding operations
1027 21:07:58.855997 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:07:58.856007 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1027 21:07:58.856012 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:07:58.856067 (+ 55us) tablet.cc:699] Row locks acquired
1027 21:07:58.856069 (+ 2us) write_op.cc:260] PREPARE: finished
1027 21:07:58.856108 (+ 39us) write_op.cc:270] Start()
1027 21:07:58.856132 (+ 24us) write_op.cc:276] Timestamp: P: 1761599278856082 usec, L: 0
1027 21:07:58.856135 (+ 3us) op_driver.cc:348] REPLICATION: starting
1027 21:07:58.856295 (+ 160us) log.cc:844] Serialized 622 byte log entry
1027 21:07:58.860386 (+ 4091us) op_driver.cc:464] REPLICATION: finished
1027 21:07:58.864653 (+ 4267us) write_op.cc:301] APPLY: starting
1027 21:07:58.864668 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1027 21:07:58.864716 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1027 21:07:58.864765 (+ 49us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:07:58.864867 (+ 102us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:07:58.864936 (+ 69us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:07:58.864945 (+ 9us) write_op.cc:312] APPLY: finished
1027 21:07:58.865017 (+ 72us) log.cc:844] Serialized 29 byte log entry
1027 21:07:58.865113 (+ 96us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.865130 (+ 17us) write_op.cc:454] Released schema lock
1027 21:07:58.865136 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1027 21:07:58.865209 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:07:58.865213 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4214,"apply.run_cpu_time_us":553,"apply.run_wall_time_us":552,"num_ops":2,"prepare.queue_time_us":49924,"prepare.run_cpu_time_us":618,"prepare.run_wall_time_us":7936,"raft.queue_time_us":3061,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":207,"replication_time_us":4228}]]}
I20251027 21:07:58.867322 10032 ts_tablet_manager.cc:1434] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20251027 21:07:58.868468 9896 tablet_service.cc:1552] Processing DeleteTablet for tablet 288b88e13fdf4427aae53f64b87733bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:58.869447 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet b719dc40d7e348608cdfc78401c19824 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:58.870695 10515 raft_consensus.cc:493] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:58.870899 10515 raft_consensus.cc:515] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.871275 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 533f231e3ff64531ae2331409a961425 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:58.871558 10515 leader_election.cc:290] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:58.872393 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6db0f31c12d14e66a4ec7fc722d62c3d" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:58.872815 9566 leader_election.cc:343] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.875200 10041 ts_tablet_manager.cc:1929] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:58.875324 10041 log.cc:1199] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/8fc2d7d6601d4cc2849064f4cdddc90d
I20251027 21:07:58.875661 10041 ts_tablet_manager.cc:1950] T 8fc2d7d6601d4cc2849064f4cdddc90d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:58.877241 10032 tablet_bootstrap.cc:492] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:07:58.878626 9568 leader_election.cc:343] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 6db0f31c12d14e66a4ec7fc722d62c3d
I20251027 21:07:58.878749 9568 leader_election.cc:304] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:58.879093 10628 raft_consensus.cc:493] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:58.879272 10515 raft_consensus.cc:2749] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:58.879675 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 614) took 2080 ms. Trace:
I20251027 21:07:58.879797 10043 rpcz_store.cc:276] 1027 21:07:56.799481 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.487573 (+688092us) service_pool.cc:225] Handling call
1027 21:07:58.797943 (+1310370us) ts_tablet_manager.cc:1151] Deleting tablet 69da799c886841a19002d3c7c5084e73
1027 21:07:58.797952 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.825762 (+ 27810us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.839420 (+ 13658us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.879637 (+ 40217us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1292230}
I20251027 21:07:58.879432 10628 raft_consensus.cc:515] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.880255 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 69da799c886841a19002d3c7c5084e73 (table default.Table_c2e56e5918cb4007b681c71977325233 [id=0cf4bae5f10d45c1a41880b45fa0b0f0]) successfully deleted
I20251027 21:07:58.880818 10628 leader_election.cc:290] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:58.881088 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "995284883d834bbb9ef583b736ca8402" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:58.881291 9632 raft_consensus.cc:2468] T 995284883d834bbb9ef583b736ca8402 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 0.
I20251027 21:07:58.881573 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 628) took 1534 ms. Trace:
I20251027 21:07:58.881700 10043 rpcz_store.cc:276] 1027 21:07:57.347068 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.608578 (+261510us) service_pool.cc:225] Handling call
1027 21:07:58.880853 (+1272275us) ts_tablet_manager.cc:1151] Deleting tablet 9b9ffa18afa54f2c9ffd7b5a03a3a324
1027 21:07:58.880865 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.881533 (+ 668us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1025501}
W20251027 21:07:58.882225 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 9b9ffa18afa54f2c9ffd7b5a03a3a324 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b9ffa18afa54f2c9ffd7b5a03a3a324 already in progress: creating tablet
I20251027 21:07:58.882316 9860 leader_election.cc:304] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:07:58.882373 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet fe24af9838954a9bb680474800642e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:58.882596 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 630) took 1535 ms. Trace:
I20251027 21:07:58.882709 10043 rpcz_store.cc:276] 1027 21:07:57.347272 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.635089 (+287817us) service_pool.cc:225] Handling call
1027 21:07:58.882473 (+1247384us) ts_tablet_manager.cc:1151] Deleting tablet 1329ff0f8ebe4da8a6462a574ede98d4
1027 21:07:58.882482 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.882564 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1021280}
I20251027 21:07:58.882958 10628 raft_consensus.cc:2804] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:58.883055 10628 raft_consensus.cc:493] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251027 21:07:58.883064 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 1329ff0f8ebe4da8a6462a574ede98d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1329ff0f8ebe4da8a6462a574ede98d4 already in progress: creating tablet
I20251027 21:07:58.883176 10628 raft_consensus.cc:3060] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:58.883428 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 629) took 1536 ms. Trace:
I20251027 21:07:58.883535 10043 rpcz_store.cc:276] 1027 21:07:57.347139 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.635033 (+287894us) service_pool.cc:225] Handling call
1027 21:07:58.883276 (+1248243us) ts_tablet_manager.cc:1151] Deleting tablet b3abc5f78c4e41e29128e2d790319c75
1027 21:07:58.883284 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.883399 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1020003}
W20251027 21:07:58.883929 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet b3abc5f78c4e41e29128e2d790319c75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3abc5f78c4e41e29128e2d790319c75 already in progress: creating tablet
I20251027 21:07:58.884198 10043 tablet_replica.cc:333] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:58.884402 10043 raft_consensus.cc:2243] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:58.884568 10043 raft_consensus.cc:2272] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:58.886021 10628 raft_consensus.cc:515] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.887728 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "995284883d834bbb9ef583b736ca8402" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:58.888188 9859 leader_election.cc:343] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.888398 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "995284883d834bbb9ef583b736ca8402" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:58.888571 10628 leader_election.cc:290] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:58.888780 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "995284883d834bbb9ef583b736ca8402" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
W20251027 21:07:58.888847 9859 leader_election.cc:343] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.888988 9632 raft_consensus.cc:3060] T 995284883d834bbb9ef583b736ca8402 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:58.889663 9733 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:40722 (request call id 112) took 1007 ms. Trace:
I20251027 21:07:58.889803 9733 rpcz_store.cc:276] 1027 21:07:57.882557 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.882775 (+ 218us) service_pool.cc:225] Handling call
1027 21:07:58.733236 (+850461us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x616000102d54. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d8fee1200 00007f8d93d9166f 00007f8d8e1b7c36 00007f8d8e1b7965 00007f8d8afe627e 00007f8d8afe515e 00007f8d8afeb395 00007f8d8afebcbf 00007f8d8afebae8 0000000000a8867c
1027 21:07:58.889632 (+156396us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28049280}
I20251027 21:07:58.891779 9632 raft_consensus.cc:2468] T 995284883d834bbb9ef583b736ca8402 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 1.
I20251027 21:07:58.892690 9860 leader_election.cc:304] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:58.893622 10628 raft_consensus.cc:2804] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:58.894544 10628 raft_consensus.cc:697] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Becoming Leader. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Running, Role: LEADER
I20251027 21:07:58.894903 10628 consensus_queue.cc:237] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.898649 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_526d1cbb28784221b43a6719685c5e2a" } modify_external_catalogs: true
I20251027 21:07:58.898882 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_526d1cbb28784221b43a6719685c5e2a" } modify_external_catalogs: true
I20251027 21:07:58.900432 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 38f2133010b14aad88b550fb12c966a5
I20251027 21:07:58.900774 10029 tablet_bootstrap.cc:492] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:58.900988 10029 ts_tablet_manager.cc:1403] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.002s
I20251027 21:07:58.901265 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 38f2133010b14aad88b550fb12c966a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:58.901527 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 38f2133010b14aad88b550fb12c966a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:58.901890 10029 raft_consensus.cc:359] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.902050 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 38f2133010b14aad88b550fb12c966a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:58.902112 10029 raft_consensus.cc:385] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.902326 10029 raft_consensus.cc:740] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:58.902915 10029 consensus_queue.cc:260] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.905328 10029 ts_tablet_manager.cc:1434] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251027 21:07:58.905827 10029 tablet_bootstrap.cc:492] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:58.907953 10029 tablet_bootstrap.cc:654] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:58.879176 10032 tablet_bootstrap.cc:654] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:58.911463 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 4d88aacbf47144cb837b99d68f6331e2
I20251027 21:07:58.912774 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d88aacbf47144cb837b99d68f6331e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:58.912777 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d88aacbf47144cb837b99d68f6331e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:58.914929 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 628) took 2103 ms. Trace:
I20251027 21:07:58.920073 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 6210d3d10fc44b9e9e2b0780883186ce
I20251027 21:07:58.921052 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 6210d3d10fc44b9e9e2b0780883186ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:58.921219 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 8ae222307a374a91916f560a0fbe8d8f
I20251027 21:07:58.920905 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 6210d3d10fc44b9e9e2b0780883186ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:58.922222 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ae222307a374a91916f560a0fbe8d8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:58.922387 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ae222307a374a91916f560a0fbe8d8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:58.925387 10040 ts_tablet_manager.cc:1929] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:58.925531 10040 log.cc:1199] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/efc0019840244d14bb95ae581793e74b
I20251027 21:07:58.925879 10040 ts_tablet_manager.cc:1950] T efc0019840244d14bb95ae581793e74b P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:58.921416 10041 rpcz_store.cc:276] 1027 21:07:56.811562 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.065837 (+254275us) service_pool.cc:225] Handling call
1027 21:07:58.741622 (+1675785us) ts_tablet_manager.cc:1151] Deleting tablet 8fc2d7d6601d4cc2849064f4cdddc90d
1027 21:07:58.741632 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.783668 (+ 42036us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.875070 (+ 91402us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.914875 (+ 39805us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1666579}
I20251027 21:07:58.927160 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 8fc2d7d6601d4cc2849064f4cdddc90d (table default.Table_162a38d7fe14491d8dae6d2e097d3e3a [id=6f39910d23dd455f87a4ac847917b87f]) successfully deleted
I20251027 21:07:58.909850 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 8fc2d7d6601d4cc2849064f4cdddc90d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:58.913161 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d88aacbf47144cb837b99d68f6331e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:58.928439 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet a8328318a4f14b38a4ce36d183e99dfb
I20251027 21:07:58.914408 10043 ts_tablet_manager.cc:1916] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:58.929517 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet a8328318a4f14b38a4ce36d183e99dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:58.921713 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 6210d3d10fc44b9e9e2b0780883186ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:58.922839 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ae222307a374a91916f560a0fbe8d8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:58.925351 9601 tablet_service.cc:1505] Processing CreateTablet for tablet d6608ae767ae42b883e482eec46fefdb (DEFAULT_TABLE table=default.Table_8d267a7e37b447158fe1ab962b3283d3 [id=0f52927fb2a34aff935e641e5835334e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.930547 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6608ae767ae42b883e482eec46fefdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.930787 9736 tablet_service.cc:1505] Processing CreateTablet for tablet d6608ae767ae42b883e482eec46fefdb (DEFAULT_TABLE table=default.Table_8d267a7e37b447158fe1ab962b3283d3 [id=0f52927fb2a34aff935e641e5835334e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.931142 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6608ae767ae42b883e482eec46fefdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.934966 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet a8328318a4f14b38a4ce36d183e99dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:58.935122 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet a8328318a4f14b38a4ce36d183e99dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:58.938122 9893 tablet_service.cc:1505] Processing CreateTablet for tablet d6608ae767ae42b883e482eec46fefdb (DEFAULT_TABLE table=default.Table_8d267a7e37b447158fe1ab962b3283d3 [id=0f52927fb2a34aff935e641e5835334e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.938521 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6608ae767ae42b883e482eec46fefdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.938666 9601 tablet_service.cc:1505] Processing CreateTablet for tablet cdec0ffb1531436f8b6400e960ef2a65 (DEFAULT_TABLE table=default.Table_5a7e103ca3a64375a49f7c4efdfb045a [id=8c41d1076a10483f9733109ead9913f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.938988 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdec0ffb1531436f8b6400e960ef2a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.943159 9896 tablet_service.cc:1505] Processing CreateTablet for tablet cdec0ffb1531436f8b6400e960ef2a65 (DEFAULT_TABLE table=default.Table_5a7e103ca3a64375a49f7c4efdfb045a [id=8c41d1076a10483f9733109ead9913f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.943550 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdec0ffb1531436f8b6400e960ef2a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.944180 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_11d4ee667e1c4d9badc00349e1c61463" } new_table_name: "default.Table_d7530773668f42c5a07a921df02a95ab" modify_external_catalogs: true
I20251027 21:07:58.945919 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_08ca8383428c4a25bc4ab94c823d0fb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:58.946007 9755 tablet_service.cc:1505] Processing CreateTablet for tablet cdec0ffb1531436f8b6400e960ef2a65 (DEFAULT_TABLE table=default.Table_5a7e103ca3a64375a49f7c4efdfb045a [id=8c41d1076a10483f9733109ead9913f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.946373 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdec0ffb1531436f8b6400e960ef2a65. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:58.946501 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08ca8383428c4a25bc4ab94c823d0fb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:58.949735 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8291d411936d4f4e8d267fdc28efc533" } new_table_name: "default.Table_5a979a51f58942db97c3897dcd5feb08" modify_external_catalogs: true
I20251027 21:07:58.950192 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_645fb813ce0a4245963a221252ac8d87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:58.950639 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_645fb813ce0a4245963a221252ac8d87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:58.950934 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5a9b8f6ad9a64835ab66cfce7bcd144e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:58.951367 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a9b8f6ad9a64835ab66cfce7bcd144e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:58.952862 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_17c224e94fc94d99b0d23c088cef1820"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:58.953145 10043 ts_tablet_manager.cc:1929] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:58.953269 10043 log.cc:1199] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/49d9e5c5960a4534a6603875006e6266
W20251027 21:07:58.953325 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17c224e94fc94d99b0d23c088cef1820 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:58.953621 10043 ts_tablet_manager.cc:1950] T 49d9e5c5960a4534a6603875006e6266 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:58.955345 10029 tablet_bootstrap.cc:492] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:58.955557 10029 ts_tablet_manager.cc:1403] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.050s user 0.001s sys 0.003s
I20251027 21:07:58.955583 9594 tablet_service.cc:1505] Processing CreateTablet for tablet cf66e5a18be84f9d8469c5fd46da6f2f (DEFAULT_TABLE table=default.Table_352d903f119f41d3b405c925eaf2ebaa [id=9463e0b2a20040219af1ad2a1e858632]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.955940 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf66e5a18be84f9d8469c5fd46da6f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.956548 10029 raft_consensus.cc:359] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.956799 10029 raft_consensus.cc:385] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.956983 10029 raft_consensus.cc:740] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:58.957278 10029 consensus_queue.cc:260] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.957556 9312 master_service.cc:954] ReplaceTablet: received request to replace tablet bbf92e29993b46389a69efd9e9a2341f from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:58.960047 10032 tablet_bootstrap.cc:492] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:58.960108 10031 tablet_bootstrap.cc:492] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:58.960178 10032 ts_tablet_manager.cc:1403] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.002s
I20251027 21:07:58.957799 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 640) took 1647 ms. Trace:
I20251027 21:07:58.960386 10041 rpcz_store.cc:276] 1027 21:07:57.310256 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.310536 (+ 280us) service_pool.cc:225] Handling call
1027 21:07:58.957642 (+1647106us) ts_tablet_manager.cc:1151] Deleting tablet 9b9ffa18afa54f2c9ffd7b5a03a3a324
1027 21:07:58.957652 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.957759 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1646870}
W20251027 21:07:58.960857 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 9b9ffa18afa54f2c9ffd7b5a03a3a324 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b9ffa18afa54f2c9ffd7b5a03a3a324 already in progress: creating tablet
I20251027 21:07:58.961108 10032 raft_consensus.cc:359] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.961475 10032 raft_consensus.cc:385] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.961628 10032 raft_consensus.cc:740] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:58.961658 10031 ts_tablet_manager.cc:1403] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.199s user 0.006s sys 0.007s
I20251027 21:07:58.961923 10032 consensus_queue.cc:260] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.962044 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 78884c2be75f45f881f5424d3b3f4e08 with delete_type TABLET_DATA_DELETED (replaced by tablet 4996016747cd484cbda5f11e40e269b8) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:58.962731 10029 ts_tablet_manager.cc:1434] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251027 21:07:58.964358 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0efd5aacf5b54ded90984a7c702d12d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:58.964600 10029 tablet_bootstrap.cc:492] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
W20251027 21:07:58.964783 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0efd5aacf5b54ded90984a7c702d12d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:58.966358 10032 ts_tablet_manager.cc:1434] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:58.966639 10032 tablet_bootstrap.cc:492] T 343eaeca24dc4816ae032c7e674e9412 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:58.966724 10031 raft_consensus.cc:359] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.966938 10031 raft_consensus.cc:385] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.967007 10031 raft_consensus.cc:740] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:58.967168 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 641) took 1646 ms. Trace:
I20251027 21:07:58.967327 10041 rpcz_store.cc:276] 1027 21:07:57.314821 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.315122 (+ 301us) service_pool.cc:225] Handling call
1027 21:07:58.961432 (+1646310us) ts_tablet_manager.cc:1151] Deleting tablet b3abc5f78c4e41e29128e2d790319c75
1027 21:07:58.961443 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.961556 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1646088}
I20251027 21:07:58.967720 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_fe401505cb114943863339485e5ff2f5" } new_table_name: "default.Table_61920d45ff1042008086598b29d483af" modify_external_catalogs: true
I20251027 21:07:58.967782 10041 tablet_replica.cc:333] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:58.967998 10041 raft_consensus.cc:2243] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:58.968149 10041 raft_consensus.cc:2272] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:58.968348 10032 tablet_bootstrap.cc:654] T 343eaeca24dc4816ae032c7e674e9412 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:58.968606 10041 ts_tablet_manager.cc:1916] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:58.969811 10029 tablet_bootstrap.cc:654] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:58.972863 9303 catalog_manager.cc:5649] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67). New cstate: current_term: 1 leader_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: UNKNOWN } } }
I20251027 21:07:58.973277 9303 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 995284883d834bbb9ef583b736ca8402 (table default.Table_140424c161ea4581bed312cf5deb5ab5 [id=186c9cce53d74040b9799ea386fdffd7]). Expected version 4 got 0
I20251027 21:07:58.974592 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 642) took 1425 ms. Trace:
I20251027 21:07:58.974689 10043 rpcz_store.cc:276] 1027 21:07:57.548941 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.861218 (+312277us) service_pool.cc:225] Handling call
1027 21:07:58.884141 (+1022923us) ts_tablet_manager.cc:1151] Deleting tablet 49d9e5c5960a4534a6603875006e6266
1027 21:07:58.884150 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.932488 (+ 48338us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.953049 (+ 20561us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.974537 (+ 21488us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":991962}
I20251027 21:07:58.975159 10043 tablet_replica.cc:333] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:58.975332 10043 raft_consensus.cc:2243] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:58.975487 10043 raft_consensus.cc:2272] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:58.975956 10043 ts_tablet_manager.cc:1916] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:58.977730 10628 raft_consensus.cc:493] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:58.977959 10628 raft_consensus.cc:515] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:58.978284 10515 raft_consensus.cc:493] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251027 21:07:58.978789 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6600774e4c7b48479a2fdb02614ba505 consensus state has no leader
I20251027 21:07:58.979477 10628 leader_election.cc:290] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:58.979535 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9339c6b1154400eace53d765528fdf9" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:58.978749 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet b3abc5f78c4e41e29128e2d790319c75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3abc5f78c4e41e29128e2d790319c75 already in progress: creating tablet
W20251027 21:07:58.980046 9859 leader_election.cc:343] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:58.980278 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 49d9e5c5960a4534a6603875006e6266 (table default.Table_52c53bafe66b4c66b99e5b26e4df7ef5 [id=27dc835f316f41e091be9b2b1f560bbf]) successfully deleted
W20251027 21:07:58.983834 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0820648719a34f379507efced9cc434b consensus state has no leader
I20251027 21:07:58.985370 10032 tablet_bootstrap.cc:492] T 343eaeca24dc4816ae032c7e674e9412 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:58.985536 10032 ts_tablet_manager.cc:1403] T 343eaeca24dc4816ae032c7e674e9412 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.002s
I20251027 21:07:58.985766 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5dee7205c58b4b70beadf92f948aea58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:58.977545 10031 consensus_queue.cc:260] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
W20251027 21:07:58.986375 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5dee7205c58b4b70beadf92f948aea58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:58.986745 10031 ts_tablet_manager.cc:1434] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20251027 21:07:58.988838 9736 tablet_service.cc:1505] Processing CreateTablet for tablet cf66e5a18be84f9d8469c5fd46da6f2f (DEFAULT_TABLE table=default.Table_352d903f119f41d3b405c925eaf2ebaa [id=9463e0b2a20040219af1ad2a1e858632]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.989797 10031 tablet_bootstrap.cc:492] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:58.988886 10032 raft_consensus.cc:359] T 343eaeca24dc4816ae032c7e674e9412 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:58.990046 10032 raft_consensus.cc:385] T 343eaeca24dc4816ae032c7e674e9412 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:58.988981 9893 tablet_service.cc:1505] Processing CreateTablet for tablet cf66e5a18be84f9d8469c5fd46da6f2f (DEFAULT_TABLE table=default.Table_352d903f119f41d3b405c925eaf2ebaa [id=9463e0b2a20040219af1ad2a1e858632]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.990126 10032 raft_consensus.cc:740] T 343eaeca24dc4816ae032c7e674e9412 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:58.990406 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf66e5a18be84f9d8469c5fd46da6f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.990478 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf66e5a18be84f9d8469c5fd46da6f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.990777 10032 consensus_queue.cc:260] T 343eaeca24dc4816ae032c7e674e9412 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:58.991407 10032 ts_tablet_manager.cc:1434] T 343eaeca24dc4816ae032c7e674e9412 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251027 21:07:58.991784 10032 tablet_bootstrap.cc:492] T abd3b8e71d3041b4b9e5f6663841bfcf P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:58.993492 10032 tablet_bootstrap.cc:654] T abd3b8e71d3041b4b9e5f6663841bfcf P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:58.994257 10515 raft_consensus.cc:515] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:58.994880 10515 leader_election.cc:290] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:58.994951 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 606526e392e44e0aae6f05f57f23e8b2 (DEFAULT_TABLE table=default.Table_d898f3a5d8ad4819a75089ec91846c21 [id=cc1308ec3673467b9efdea3d76c91f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.997196 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 606526e392e44e0aae6f05f57f23e8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:58.998739 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 606526e392e44e0aae6f05f57f23e8b2 (DEFAULT_TABLE table=default.Table_d898f3a5d8ad4819a75089ec91846c21 [id=cc1308ec3673467b9efdea3d76c91f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:58.999511 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1776590d8387494da5dfbd08db17f4b1" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:58.999692 9771 raft_consensus.cc:2468] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:59.000245 10560 raft_consensus.cc:493] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.000368 10560 raft_consensus.cc:515] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.000965 10560 leader_election.cc:290] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:59.002094 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c8fded5f6e8b4798a55ec84954d75574"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.002635 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8fded5f6e8b4798a55ec84954d75574 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.004470 10032 tablet_bootstrap.cc:492] T abd3b8e71d3041b4b9e5f6663841bfcf P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:59.004601 10032 ts_tablet_manager.cc:1403] T abd3b8e71d3041b4b9e5f6663841bfcf P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20251027 21:07:59.005378 10032 raft_consensus.cc:359] T abd3b8e71d3041b4b9e5f6663841bfcf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.005563 10032 raft_consensus.cc:385] T abd3b8e71d3041b4b9e5f6663841bfcf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.005649 10032 raft_consensus.cc:740] T abd3b8e71d3041b4b9e5f6663841bfcf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.005890 10032 consensus_queue.cc:260] T abd3b8e71d3041b4b9e5f6663841bfcf P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.011523 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 606526e392e44e0aae6f05f57f23e8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.011570 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_019b51c3ff544dae94c53821dc97ab9f" } new_table_name: "default.Table_c9999fc8fde44b77aecd45e35a92bdbc" modify_external_catalogs: true
I20251027 21:07:59.012240 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1f99ca89f924b58b1416d2945a6ba40" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:59.017199 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_8fee41d5402246f2b96068387e60cc49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.017644 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fee41d5402246f2b96068387e60cc49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.018204 9895 tablet_service.cc:1505] Processing CreateTablet for tablet be33f1825a4d433ea9ed859f37fe208e (DEFAULT_TABLE table=default.Table_86b3f1d7a96f4c788e7ab443853fd5cd [id=70eb7e3452dc438cbdd9f12917bb8c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.018581 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be33f1825a4d433ea9ed859f37fe208e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:59.019240 9859 leader_election.cc:343] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.022468 10032 ts_tablet_manager.cc:1434] T abd3b8e71d3041b4b9e5f6663841bfcf P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20251027 21:07:59.022905 10032 tablet_bootstrap.cc:492] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:59.028479 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 383) took 5036 ms. Trace:
I20251027 21:07:59.028577 10040 rpcz_store.cc:276] 1027 21:07:53.992268 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.004773 (+ 12505us) service_pool.cc:225] Handling call
1027 21:07:58.861360 (+4856587us) ts_tablet_manager.cc:1151] Deleting tablet efc0019840244d14bb95ae581793e74b
1027 21:07:58.861369 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:58.865493 (+ 4124us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:58.925257 (+ 59764us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.028423 (+103166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4856353}
I20251027 21:07:59.028924 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet efc0019840244d14bb95ae581793e74b (table default.Table_0432e7918b494dccb2f6b921657c0d81 [id=f773d65a07a4473ab8b87cd82ac50bcf]) successfully deleted
I20251027 21:07:59.029083 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 397) took 4826 ms. Trace:
I20251027 21:07:59.029153 10040 rpcz_store.cc:276] 1027 21:07:54.202954 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.203021 (+ 67us) service_pool.cc:225] Handling call
1027 21:07:59.028943 (+4825922us) ts_tablet_manager.cc:1151] Deleting tablet 22d861810ed846c3b77a452d05e93d72
1027 21:07:59.028953 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.029035 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4825300}
I20251027 21:07:59.029548 10040 tablet_replica.cc:333] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:59.029717 10040 raft_consensus.cc:2243] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251027 21:07:59.029798 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 22d861810ed846c3b77a452d05e93d72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22d861810ed846c3b77a452d05e93d72 already in progress: creating tablet
I20251027 21:07:59.029892 10040 raft_consensus.cc:2272] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.030573 10040 ts_tablet_manager.cc:1916] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.031454 10032 tablet_bootstrap.cc:654] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.034596 10032 tablet_bootstrap.cc:492] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:59.034757 10032 ts_tablet_manager.cc:1403] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20251027 21:07:59.034684 9755 tablet_service.cc:1505] Processing CreateTablet for tablet be33f1825a4d433ea9ed859f37fe208e (DEFAULT_TABLE table=default.Table_86b3f1d7a96f4c788e7ab443853fd5cd [id=70eb7e3452dc438cbdd9f12917bb8c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.035056 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be33f1825a4d433ea9ed859f37fe208e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.036998 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_274350c56d3a43a4b3738ced49d24150" } new_table_name: "default.Table_4b0bc60a7c5e4888b79210296b7dfa20" modify_external_catalogs: true
I20251027 21:07:59.039414 10032 raft_consensus.cc:359] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.039611 10032 raft_consensus.cc:385] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.039696 10032 raft_consensus.cc:740] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
W20251027 21:07:59.041858 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9a1a3394d53419ebfe327a58388fc25 consensus state has no leader
I20251027 21:07:59.044474 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5291bf5018e443118fabd8171b612b07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.044917 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5291bf5018e443118fabd8171b612b07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.046301 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9339c6b1154400eace53d765528fdf9" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:59.046881 9860 leader_election.cc:343] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.047045 9860 leader_election.cc:304] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:59.047439 10560 raft_consensus.cc:2749] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:59.047634 10031 tablet_bootstrap.cc:654] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.048205 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3de9fdb7f17a46e0a0560f82afaef828"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:59.048296 10529 raft_consensus.cc:493] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.048432 10529 raft_consensus.cc:515] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:59.048669 9568 leader_election.cc:343] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 1776590d8387494da5dfbd08db17f4b1
W20251027 21:07:59.048697 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3de9fdb7f17a46e0a0560f82afaef828 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.050005 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1f99ca89f924b58b1416d2945a6ba40" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:59.049983 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 606526e392e44e0aae6f05f57f23e8b2 (DEFAULT_TABLE table=default.Table_d898f3a5d8ad4819a75089ec91846c21 [id=cc1308ec3673467b9efdea3d76c91f9e]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:59.050526 9860 leader_election.cc:343] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.050658 9860 leader_election.cc:304] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:59.051029 10560 raft_consensus.cc:2749] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:07:59.051452 9568 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 52062.1us
I20251027 21:07:59.052918 9566 leader_election.cc:304] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:07:59.053283 10529 leader_election.cc:290] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:59.053570 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 606526e392e44e0aae6f05f57f23e8b2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:59.054035 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 53249.6us
I20251027 21:07:59.054580 10515 raft_consensus.cc:2804] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:59.054678 10515 raft_consensus.cc:493] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:59.054893 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78884c2be75f45f881f5424d3b3f4e08" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:59.058849 9303 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet abd3b8e71d3041b4b9e5f6663841bfcf (table default.Table_e18ff5a3d21f467db8e09554b370ef9d [id=f3e3147411e5488bae105ed755f5b50f]). Expected version 1 got 0
I20251027 21:07:59.060283 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 343eaeca24dc4816ae032c7e674e9412 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:59.060837 10515 raft_consensus.cc:3060] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:07:59.060875 9568 leader_election.cc:343] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 78884c2be75f45f881f5424d3b3f4e08
W20251027 21:07:59.061046 9566 leader_election.cc:343] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.061254 9566 leader_election.cc:304] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:59.061621 10529 raft_consensus.cc:2749] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:59.063624 10515 raft_consensus.cc:515] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.064199 10515 leader_election.cc:290] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:59.064564 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1776590d8387494da5dfbd08db17f4b1" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:59.064765 9771 raft_consensus.cc:3060] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:07:59.066835 9568 leader_election.cc:343] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 1776590d8387494da5dfbd08db17f4b1
I20251027 21:07:59.071367 9896 tablet_service.cc:1552] Processing DeleteTablet for tablet a287439ea6c9469da5fb56d84ee68048 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:59.067976 9771 raft_consensus.cc:2468] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:59.072662 9566 leader_election.cc:304] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:07:59.072969 10029 tablet_bootstrap.cc:492] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:59.073107 10029 ts_tablet_manager.cc:1403] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.109s user 0.003s sys 0.011s
I20251027 21:07:59.074278 10029 raft_consensus.cc:359] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.074820 10029 raft_consensus.cc:385] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.075134 10029 raft_consensus.cc:740] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.075001 10515 raft_consensus.cc:2804] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:59.076009 10515 raft_consensus.cc:697] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:07:59.076493 10515 consensus_queue.cc:237] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.079891 10041 ts_tablet_manager.cc:1929] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.080024 10041 log.cc:1199] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/ccaf5c2f8e544842880ba8ba27300ae6
I20251027 21:07:59.080322 10041 ts_tablet_manager.cc:1950] T ccaf5c2f8e544842880ba8ba27300ae6 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:59.080864 9601 tablet_service.cc:1505] Processing CreateTablet for tablet be33f1825a4d433ea9ed859f37fe208e (DEFAULT_TABLE table=default.Table_86b3f1d7a96f4c788e7ab443853fd5cd [id=70eb7e3452dc438cbdd9f12917bb8c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.081044 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 642) took 1702 ms. Trace:
I20251027 21:07:59.081132 10041 rpcz_store.cc:276] 1027 21:07:57.378433 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.488958 (+110525us) service_pool.cc:225] Handling call
1027 21:07:58.967702 (+1478744us) ts_tablet_manager.cc:1151] Deleting tablet ccaf5c2f8e544842880ba8ba27300ae6
1027 21:07:58.967711 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.021970 (+ 54259us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.079768 (+ 57798us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.081026 (+ 1258us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1478522}
I20251027 21:07:59.081599 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet ccaf5c2f8e544842880ba8ba27300ae6 (table default.Table_9c6b1e43fc5f4f4389620f1ee244db12 [id=2b0d4e085a8e42879c67963231a24b03]) successfully deleted
I20251027 21:07:59.081895 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be33f1825a4d433ea9ed859f37fe208e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:59.083336 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ccd419ff40a94f9587b024ec87a2517e consensus state has no leader
I20251027 21:07:59.089834 10560 raft_consensus.cc:493] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.089986 10560 raft_consensus.cc:515] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.090694 10560 leader_election.cc:290] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:59.091323 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b719dc40d7e348608cdfc78401c19824" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:59.091555 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b719dc40d7e348608cdfc78401c19824" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:59.091850 9859 leader_election.cc:343] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:07:59.092224 9860 leader_election.cc:343] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.092309 9860 leader_election.cc:304] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:59.092818 10560 raft_consensus.cc:2749] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:59.093214 10029 consensus_queue.cc:260] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.094910 10029 ts_tablet_manager.cc:1434] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20251027 21:07:59.098057 10029 tablet_bootstrap.cc:492] T f1aa5f7f76cb4a1ba15f016942f17067 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:59.099814 10029 tablet_bootstrap.cc:654] T f1aa5f7f76cb4a1ba15f016942f17067 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.039929 10032 consensus_queue.cc:260] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.100710 10032 ts_tablet_manager.cc:1434] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.066s user 0.016s sys 0.000s
I20251027 21:07:59.101529 10560 raft_consensus.cc:493] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.101636 10560 raft_consensus.cc:515] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:59.102213 10560 leader_election.cc:290] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:59.102761 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "288b88e13fdf4427aae53f64b87733bf" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:59.102758 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_08ca8383428c4a25bc4ab94c823d0fb4" } modify_external_catalogs: true
I20251027 21:07:59.102986 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_08ca8383428c4a25bc4ab94c823d0fb4" } modify_external_catalogs: true
W20251027 21:07:59.103322 9859 leader_election.cc:343] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.104914 9308 master_service.cc:954] ReplaceTablet: received request to replace tablet 49b3d23798b740008c98cd0754ae33f4 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:59.103281 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "288b88e13fdf4427aae53f64b87733bf" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:59.108330 9860 leader_election.cc:343] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.108428 9860 leader_election.cc:304] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:59.108860 10560 raft_consensus.cc:2749] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:59.109617 10040 ts_tablet_manager.cc:1929] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.109757 10040 log.cc:1199] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/49b3d23798b740008c98cd0754ae33f4
I20251027 21:07:59.111609 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 663) took 1249 ms. Trace:
I20251027 21:07:59.111685 10041 rpcz_store.cc:276] 1027 21:07:57.862214 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.892082 (+ 29868us) service_pool.cc:225] Handling call
1027 21:07:59.081399 (+1189317us) ts_tablet_manager.cc:1151] Deleting tablet 7427bbc47a8643c19fb543626c6aea2a
1027 21:07:59.081407 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.111592 (+ 30185us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1184933}
W20251027 21:07:59.113032 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 7427bbc47a8643c19fb543626c6aea2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7427bbc47a8643c19fb543626c6aea2a already in progress: creating tablet
I20251027 21:07:59.115432 10041 tablet_replica.cc:333] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:59.115622 10041 raft_consensus.cc:2243] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.115794 10041 raft_consensus.cc:2272] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.115934 10032 tablet_bootstrap.cc:492] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:59.116348 10041 ts_tablet_manager.cc:1916] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.117496 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2a9fa2fa9615471691f07230a08e1399" } modify_external_catalogs: true
I20251027 21:07:59.117641 10032 tablet_bootstrap.cc:654] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.117666 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2a9fa2fa9615471691f07230a08e1399" } modify_external_catalogs: true
W20251027 21:07:59.123029 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 76bee322d0b94abe9166170733608853 consensus state has no leader
I20251027 21:07:59.125360 10041 ts_tablet_manager.cc:1929] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.125478 9896 tablet_service.cc:1552] Processing DeleteTablet for tablet 132a01405a63447ab27b2137b803224f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:59.125530 10041 log.cc:1199] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/6bf9c2a78aac42528d67846dc58adc40
I20251027 21:07:59.126102 10041 ts_tablet_manager.cc:1950] T 6bf9c2a78aac42528d67846dc58adc40 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:59.126312 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_562d4900bd0c4a43b93862a97e5d6561" } modify_external_catalogs: true
I20251027 21:07:59.126516 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_562d4900bd0c4a43b93862a97e5d6561" } modify_external_catalogs: true
I20251027 21:07:59.127035 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 664) took 1264 ms. Trace:
I20251027 21:07:59.127163 10041 rpcz_store.cc:276] 1027 21:07:57.862340 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.896488 (+ 34148us) service_pool.cc:225] Handling call
1027 21:07:59.111906 (+1215418us) ts_tablet_manager.cc:1151] Deleting tablet 6bf9c2a78aac42528d67846dc58adc40
1027 21:07:59.115372 (+ 3466us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.121630 (+ 6258us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.125287 (+ 3657us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.127018 (+ 1731us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1214132}
I20251027 21:07:59.127656 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 6bf9c2a78aac42528d67846dc58adc40 (table default.Table_e9fd9bee6a2a40d7ad92d2d31b03afce [id=af6cb904f8cd4a929be7a978f9bc2001]) successfully deleted
I20251027 21:07:59.127772 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1a51e839945b4912a5519bb1f21a6b75" } modify_external_catalogs: true
I20251027 21:07:59.127930 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1a51e839945b4912a5519bb1f21a6b75" } modify_external_catalogs: true
I20251027 21:07:59.127988 10041 tablet_replica.cc:333] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:59.128165 10041 raft_consensus.cc:2243] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.128288 10041 raft_consensus.cc:2272] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.128800 10041 ts_tablet_manager.cc:1916] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.130244 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d5b9049d02b8474aafbf1e27d9f103ec" } modify_external_catalogs: true
I20251027 21:07:59.130407 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d5b9049d02b8474aafbf1e27d9f103ec" } modify_external_catalogs: true
I20251027 21:07:59.131755 10043 ts_tablet_manager.cc:1929] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.134754 10032 tablet_bootstrap.cc:492] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:59.134881 10032 ts_tablet_manager.cc:1403] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20251027 21:07:59.135383 10041 ts_tablet_manager.cc:1929] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.135535 10041 log.cc:1199] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/9b04349b15f3494bb0933db56b7b24bc
I20251027 21:07:59.135890 10041 ts_tablet_manager.cc:1950] T 9b04349b15f3494bb0933db56b7b24bc P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:59.136686 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 665) took 1274 ms. Trace:
I20251027 21:07:59.136822 10041 rpcz_store.cc:276] 1027 21:07:57.862429 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.897797 (+ 35368us) service_pool.cc:225] Handling call
1027 21:07:59.127949 (+1230152us) ts_tablet_manager.cc:1151] Deleting tablet 9b04349b15f3494bb0933db56b7b24bc
1027 21:07:59.127958 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.132227 (+ 4269us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.135309 (+ 3082us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.136665 (+ 1356us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1229940}
I20251027 21:07:59.137373 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 9b04349b15f3494bb0933db56b7b24bc (table default.Table_2ff5a5b701c540b8a9eb8b87aace0e77 [id=69f10c0f3f3a4227ab4c35e939085634]) successfully deleted
I20251027 21:07:59.137622 9304 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 7aa8c538b6c849a28a28ffe706306891 (table default.Table_b779f2756a924ab6ba1bc1702a412e06 [id=8ed572891f9b41cba8ed984911fa6b11]). Expected version 1 got 0
I20251027 21:07:59.137717 10041 tablet_replica.cc:333] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:59.137904 10041 raft_consensus.cc:2243] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.138031 10041 raft_consensus.cc:2272] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.138037 10032 raft_consensus.cc:359] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.138247 10032 raft_consensus.cc:385] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.138302 10032 raft_consensus.cc:740] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.138495 10041 ts_tablet_manager.cc:1916] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.138473 10032 consensus_queue.cc:260] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.139262 10032 ts_tablet_manager.cc:1434] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251027 21:07:59.139511 10032 tablet_bootstrap.cc:492] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:59.145018 10032 tablet_bootstrap.cc:654] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.145484 10041 ts_tablet_manager.cc:1929] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.145634 10041 log.cc:1199] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/d8e008c868d04281beba0e5b204eb69a
I20251027 21:07:59.145972 10041 ts_tablet_manager.cc:1950] T d8e008c868d04281beba0e5b204eb69a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:59.146746 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 666) took 1284 ms. Trace:
I20251027 21:07:59.146871 10041 rpcz_store.cc:276] 1027 21:07:57.862567 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.898025 (+ 35458us) service_pool.cc:225] Handling call
1027 21:07:59.137676 (+1239651us) ts_tablet_manager.cc:1151] Deleting tablet d8e008c868d04281beba0e5b204eb69a
1027 21:07:59.137684 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.142027 (+ 4343us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.145408 (+ 3381us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.146730 (+ 1322us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1190488}
I20251027 21:07:59.147009 9307 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet f2773409e2ac447e89a5d91dc354be0a
I20251027 21:07:59.147069 10043 log.cc:1199] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/ae124514964440c5b88b16ac9f8a2567
I20251027 21:07:59.147377 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 667) took 1269 ms. Trace:
I20251027 21:07:59.147485 10043 ts_tablet_manager.cc:1950] T ae124514964440c5b88b16ac9f8a2567 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:59.147490 10041 rpcz_store.cc:276] 1027 21:07:57.881485 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.947230 (+ 65745us) service_pool.cc:225] Handling call
1027 21:07:59.147262 (+1200032us) ts_tablet_manager.cc:1151] Deleting tablet d1f24d21bb954ac9b62ce458c6a22462
1027 21:07:59.147273 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.147359 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1194174}
I20251027 21:07:59.147809 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet d8e008c868d04281beba0e5b204eb69a (table default.Table_04ef6b3afcc34d84804cd6e9613d7bc2 [id=8a1cda11e1b3436cb8ffa2616d543a29]) successfully deleted
I20251027 21:07:59.148036 10515 raft_consensus.cc:493] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251027 21:07:59.148193 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet d1f24d21bb954ac9b62ce458c6a22462 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1f24d21bb954ac9b62ce458c6a22462 already in progress: creating tablet
I20251027 21:07:59.148212 10515 raft_consensus.cc:515] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.148509 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 668) took 1266 ms. Trace:
I20251027 21:07:59.148609 10041 rpcz_store.cc:276] 1027 21:07:57.881648 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.953122 (+ 71474us) service_pool.cc:225] Handling call
1027 21:07:59.148420 (+1195298us) ts_tablet_manager.cc:1151] Deleting tablet de2c033a064643b386ee84085cbdad64
1027 21:07:59.148428 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.148494 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1192606}
I20251027 21:07:59.148906 9896 tablet_service.cc:1552] Processing DeleteTablet for tablet 533f34d4872444278477abcce89da9ec with delete_type TABLET_DATA_DELETED (replaced by tablet 5054bbc3e27243298b4df6b0a87af8be) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:59.149044 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet de2c033a064643b386ee84085cbdad64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de2c033a064643b386ee84085cbdad64 already in progress: creating tablet
I20251027 21:07:59.149367 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 669) took 1267 ms. Trace:
I20251027 21:07:59.149477 10041 rpcz_store.cc:276] 1027 21:07:57.881746 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.955839 (+ 74093us) service_pool.cc:225] Handling call
1027 21:07:59.149280 (+1193441us) ts_tablet_manager.cc:1151] Deleting tablet 2532bf7190064443a898e47e843fd9ac
1027 21:07:59.149288 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.149355 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1186768}
W20251027 21:07:59.149715 9568 leader_election.cc:343] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 583fba7c750e4ac7b768da447002c211
I20251027 21:07:59.149902 10515 leader_election.cc:290] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:59.150451 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 641) took 1609 ms. Trace:
I20251027 21:07:59.150621 10043 rpcz_store.cc:276] 1027 21:07:57.541443 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.855375 (+313932us) service_pool.cc:225] Handling call
1027 21:07:58.975066 (+1119691us) ts_tablet_manager.cc:1151] Deleting tablet ae124514964440c5b88b16ac9f8a2567
1027 21:07:58.975075 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.083124 (+108049us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.131635 (+ 48511us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.150417 (+ 18782us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2127,"tablet-delete.queue_time_us":1063237}
I20251027 21:07:59.152455 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 654) took 1259 ms. Trace:
I20251027 21:07:59.152608 10043 rpcz_store.cc:276] 1027 21:07:57.892752 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.117881 (+225129us) service_pool.cc:225] Handling call
1027 21:07:59.152266 (+1034385us) ts_tablet_manager.cc:1151] Deleting tablet 7427bbc47a8643c19fb543626c6aea2a
1027 21:07:59.152334 (+ 68us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.152425 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":750435}
W20251027 21:07:59.150130 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 2532bf7190064443a898e47e843fd9ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2532bf7190064443a898e47e843fd9ac already in progress: creating tablet
I20251027 21:07:59.152179 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet ae124514964440c5b88b16ac9f8a2567 (table default.Table_11fbadd9eed14cdba7664715e389f7e5 [id=83ec44dba5c444aba48846811a5f5fc9]) successfully deleted
I20251027 21:07:59.154753 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 655) took 1235 ms. Trace:
W20251027 21:07:59.154824 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 7427bbc47a8643c19fb543626c6aea2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7427bbc47a8643c19fb543626c6aea2a already in progress: creating tablet
I20251027 21:07:59.154872 10043 rpcz_store.cc:276] 1027 21:07:57.918933 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.118781 (+199848us) service_pool.cc:225] Handling call
1027 21:07:59.154567 (+1035786us) ts_tablet_manager.cc:1151] Deleting tablet d1f24d21bb954ac9b62ce458c6a22462
1027 21:07:59.154575 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.154719 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":752164}
I20251027 21:07:59.155323 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 656) took 1235 ms. Trace:
I20251027 21:07:59.155432 10043 rpcz_store.cc:276] 1027 21:07:57.920063 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.122913 (+202850us) service_pool.cc:225] Handling call
1027 21:07:59.155206 (+1032293us) ts_tablet_manager.cc:1151] Deleting tablet de2c033a064643b386ee84085cbdad64
1027 21:07:59.155215 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.155293 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":752169}
W20251027 21:07:59.155570 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet d1f24d21bb954ac9b62ce458c6a22462 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1f24d21bb954ac9b62ce458c6a22462 already in progress: creating tablet
W20251027 21:07:59.155824 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet de2c033a064643b386ee84085cbdad64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de2c033a064643b386ee84085cbdad64 already in progress: creating tablet
I20251027 21:07:59.155915 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 657) took 1233 ms. Trace:
I20251027 21:07:59.156006 10043 rpcz_store.cc:276] 1027 21:07:57.922446 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.172212 (+249766us) service_pool.cc:225] Handling call
1027 21:07:59.155768 (+983556us) ts_tablet_manager.cc:1151] Deleting tablet 2532bf7190064443a898e47e843fd9ac
1027 21:07:59.155774 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.155887 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":751927}
W20251027 21:07:59.156370 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 2532bf7190064443a898e47e843fd9ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2532bf7190064443a898e47e843fd9ac already in progress: creating tablet
W20251027 21:07:59.157398 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 2752b86151a6428c88bfaac1ff72cfb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2752b86151a6428c88bfaac1ff72cfb2 already in progress: creating tablet
I20251027 21:07:59.157727 10043 tablet_replica.cc:333] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:59.157797 10041 tablet_replica.cc:333] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
W20251027 21:07:59.157769 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 2752b86151a6428c88bfaac1ff72cfb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2752b86151a6428c88bfaac1ff72cfb2 already in progress: creating tablet
I20251027 21:07:59.157948 10043 raft_consensus.cc:2243] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:07:59.157977 10041 raft_consensus.cc:2243] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.158110 10032 tablet_bootstrap.cc:492] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:59.158190 10043 pending_rounds.cc:70] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14: Trying to abort 1 pending ops.
I20251027 21:07:59.158274 10041 raft_consensus.cc:2272] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.158269 10043 pending_rounds.cc:77] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510647100805120 op_type: NO_OP noop_request { }
I20251027 21:07:59.158284 10032 ts_tablet_manager.cc:1403] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20251027 21:07:59.158371 10043 raft_consensus.cc:2889] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:07:59.158480 10043 raft_consensus.cc:2272] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.158932 10041 ts_tablet_manager.cc:1916] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.159051 10032 raft_consensus.cc:359] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.159209 10032 raft_consensus.cc:385] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.159261 10032 raft_consensus.cc:740] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.159487 10032 consensus_queue.cc:260] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.159983 10032 ts_tablet_manager.cc:1434] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:07:59.159978 10040 ts_tablet_manager.cc:1950] T 49b3d23798b740008c98cd0754ae33f4 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:59.160295 10032 tablet_bootstrap.cc:492] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:59.160377 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 398) took 4867 ms. Trace:
I20251027 21:07:59.161935 10032 tablet_bootstrap.cc:654] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.166946 10041 ts_tablet_manager.cc:1929] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.167097 10041 log.cc:1199] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/4e8c83b501d8426fa0cad6a89f9a0b9b
I20251027 21:07:59.167404 10041 ts_tablet_manager.cc:1950] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:59.167562 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "583fba7c750e4ac7b768da447002c211" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:59.168587 10610 raft_consensus.cc:493] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.167603 10040 rpcz_store.cc:276] 1027 21:07:54.293178 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.297489 (+ 4311us) service_pool.cc:225] Handling call
1027 21:07:59.029476 (+4731987us) ts_tablet_manager.cc:1151] Deleting tablet 49b3d23798b740008c98cd0754ae33f4
1027 21:07:59.029485 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.033532 (+ 4047us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.109482 (+ 75950us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.160338 (+ 50856us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4700421}
W20251027 21:07:59.169050 9566 leader_election.cc:343] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.169150 10040 tablet_replica.cc:333] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:59.169211 9566 leader_election.cc:304] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:59.169564 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 49b3d23798b740008c98cd0754ae33f4 (table default.Table_5d5f327d728f4250841a9f4f10a2b533 [id=33dbf5fe986c415cb54fbf1b574651ac]) successfully deleted
I20251027 21:07:59.170002 10031 tablet_bootstrap.cc:492] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:59.170192 10031 ts_tablet_manager.cc:1403] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.181s user 0.002s sys 0.003s
I20251027 21:07:59.171151 10031 raft_consensus.cc:359] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.171352 10031 raft_consensus.cc:385] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.171463 10031 raft_consensus.cc:740] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.171703 10031 consensus_queue.cc:260] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.172290 10040 raft_consensus.cc:2243] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.172511 10040 raft_consensus.cc:2272] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.173221 10040 ts_tablet_manager.cc:1916] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.169808 10610 raft_consensus.cc:515] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.174636 10515 raft_consensus.cc:2749] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:59.174918 10032 tablet_bootstrap.cc:492] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:59.175047 10032 ts_tablet_manager.cc:1403] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.003s
I20251027 21:07:59.175307 10029 tablet_bootstrap.cc:492] T f1aa5f7f76cb4a1ba15f016942f17067 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:59.175516 10029 ts_tablet_manager.cc:1403] T f1aa5f7f76cb4a1ba15f016942f17067 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.078s user 0.000s sys 0.003s
I20251027 21:07:59.175704 10032 raft_consensus.cc:359] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.175868 10032 raft_consensus.cc:385] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.175935 10032 raft_consensus.cc:740] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.176133 10032 consensus_queue.cc:260] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.176573 10032 ts_tablet_manager.cc:1434] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:07:59.176819 10032 tablet_bootstrap.cc:492] T 9882b234e79940088be5f604c41576b7 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:59.176959 10029 raft_consensus.cc:359] T f1aa5f7f76cb4a1ba15f016942f17067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.177208 10029 raft_consensus.cc:385] T f1aa5f7f76cb4a1ba15f016942f17067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.177367 10029 raft_consensus.cc:740] T f1aa5f7f76cb4a1ba15f016942f17067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.177670 10029 consensus_queue.cc:260] T f1aa5f7f76cb4a1ba15f016942f17067 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.178619 10029 ts_tablet_manager.cc:1434] T f1aa5f7f76cb4a1ba15f016942f17067 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251027 21:07:59.178988 10029 tablet_bootstrap.cc:492] T 6bf9c2a78aac42528d67846dc58adc40 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:59.178771 10032 tablet_bootstrap.cc:654] T 9882b234e79940088be5f604c41576b7 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.182567 10029 tablet_bootstrap.cc:654] T 6bf9c2a78aac42528d67846dc58adc40 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.182932 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 4e8c83b501d8426fa0cad6a89f9a0b9b (table default.Table_4f6664b7160e4549a50589454ff10313 [id=67b71acdadc64445bb2492e5d424cb2d]) successfully deleted
I20251027 21:07:59.183454 10041 tablet_replica.cc:333] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:59.183820 10041 raft_consensus.cc:2243] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.184294 10041 raft_consensus.cc:2272] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.184698 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9d77e1ff90f047748d72fe89fbcc0c49" } new_table_name: "default.Table_4fab264efb244206a3717a69e90455a6" modify_external_catalogs: true
I20251027 21:07:59.184927 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 57de49dacafa497dad79ab96540b5888
I20251027 21:07:59.186033 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet fc2cf41a7b474f7d86a46af76cef4100
I20251027 21:07:59.186777 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 9239f74d725649cab5ea1b0077bccef3
I20251027 21:07:59.186849 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet fc2cf41a7b474f7d86a46af76cef4100 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:59.187137 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_559e10c49ef84eeb960dcdb1ebddf817" } new_table_name: "default.Table_32eeb5fb70df48eb8c27a0a3bc95cee8" modify_external_catalogs: true
I20251027 21:07:59.187726 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_fcb69a1b675e4af592b2ce080f423003"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.188263 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fcb69a1b675e4af592b2ce080f423003 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.188597 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 9239f74d725649cab5ea1b0077bccef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:59.189181 10041 ts_tablet_manager.cc:1916] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.191712 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 21a8fc4a69f3486e92b1a68d10faed38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:59.192137 9896 tablet_service.cc:1552] Processing DeleteTablet for tablet 57de49dacafa497dad79ab96540b5888 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:59.176976 10610 leader_election.cc:290] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
W20251027 21:07:59.194693 9711 leader_election.cc:343] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 533f231e3ff64531ae2331409a961425
I20251027 21:07:59.195760 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet 9239f74d725649cab5ea1b0077bccef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:59.196818 10560 raft_consensus.cc:493] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.196861 9896 tablet_service.cc:1552] Processing DeleteTablet for tablet f34512059f0c47fc9af7db2645f86b40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:59.196950 10032 tablet_bootstrap.cc:492] T 9882b234e79940088be5f604c41576b7 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:59.196952 10560 raft_consensus.cc:515] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.197080 10032 ts_tablet_manager.cc:1403] T 9882b234e79940088be5f604c41576b7 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.001s
I20251027 21:07:59.197526 10560 leader_election.cc:290] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:59.197898 10032 raft_consensus.cc:359] T 9882b234e79940088be5f604c41576b7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.198069 10032 raft_consensus.cc:385] T 9882b234e79940088be5f604c41576b7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.198129 10032 raft_consensus.cc:740] T 9882b234e79940088be5f604c41576b7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.198364 10032 consensus_queue.cc:260] T 9882b234e79940088be5f604c41576b7 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.199391 10032 ts_tablet_manager.cc:1434] T 9882b234e79940088be5f604c41576b7 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:59.199671 10032 tablet_bootstrap.cc:492] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:59.199307 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3ad4d44124674e9aae353b207951534e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.200238 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ad4d44124674e9aae353b207951534e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.201474 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet fc2cf41a7b474f7d86a46af76cef4100 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:59.202199 10032 tablet_bootstrap.cc:654] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.193023 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet fc2cf41a7b474f7d86a46af76cef4100 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:59.193166 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 9239f74d725649cab5ea1b0077bccef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:59.193189 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet f1aa5f7f76cb4a1ba15f016942f17067 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:59.198561 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a287439ea6c9469da5fb56d84ee68048" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:59.205753 9860 leader_election.cc:343] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.208796 9896 tablet_service.cc:1552] Processing DeleteTablet for tablet 9882b234e79940088be5f604c41576b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:59.209214 10041 ts_tablet_manager.cc:1929] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251027 21:07:59.209683 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c25be263facd42c5bb6fa421160fbef0 consensus state has no leader
I20251027 21:07:59.210390 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 57de49dacafa497dad79ab96540b5888 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:59.210574 9710 leader_election.cc:343] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 533f231e3ff64531ae2331409a961425
I20251027 21:07:59.198779 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet 57de49dacafa497dad79ab96540b5888 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:59.211010 10041 log.cc:1199] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/f22f013b25074d4db9c7f13caf5b9b42
I20251027 21:07:59.211474 10041 ts_tablet_manager.cc:1950] T f22f013b25074d4db9c7f13caf5b9b42 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:59.212095 10043 ts_tablet_manager.cc:1916] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.213161 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a287439ea6c9469da5fb56d84ee68048" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:59.213498 9710 leader_election.cc:304] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:59.213657 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_131a037349c446e4a99258abc4815b26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.214088 9859 leader_election.cc:343] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.214223 9859 leader_election.cc:304] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:07:59.214319 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_131a037349c446e4a99258abc4815b26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.214565 10560 raft_consensus.cc:2749] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:59.214716 10041 tablet_replica.cc:333] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:59.214872 10041 raft_consensus.cc:2243] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.215000 10041 raft_consensus.cc:2272] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.215283 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet f22f013b25074d4db9c7f13caf5b9b42 (table default.Table_d601b22c11c34ebd8559009cd144affe [id=b96a43739ca54dac9e598e703186606b]) successfully deleted
I20251027 21:07:59.215476 10041 ts_tablet_manager.cc:1916] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:59.215521 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 133e474f9133487f89c44dbfa6266ca7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 133e474f9133487f89c44dbfa6266ca7 already in progress: creating tablet
W20251027 21:07:59.215778 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 23673513189f47638dbbc737da0991d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23673513189f47638dbbc737da0991d1 already in progress: creating tablet
I20251027 21:07:59.216337 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_ec17a31b5e2948e2b6e7f1ffee9901c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.216861 9296 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 6210d3d10fc44b9e9e2b0780883186ce (table default.Table_6c3cfd4d761643dca83d16b2441b0c64 [id=20a2daa7746f40e69803fc010dc4bcbe], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6210d3d10fc44b9e9e2b0780883186ce consensus state has no leader
W20251027 21:07:59.216899 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec17a31b5e2948e2b6e7f1ffee9901c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.218691 10610 raft_consensus.cc:2749] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:07:59.221920 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3c0d0c5263140728f3b8f0b11f46aab has no consensus state
I20251027 21:07:59.230263 10040 ts_tablet_manager.cc:1929] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.231191 10040 log.cc:1199] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/2b247829a11349b6835a2f18906e20a9
I20251027 21:07:59.231774 10040 ts_tablet_manager.cc:1950] T 2b247829a11349b6835a2f18906e20a9 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:59.235668 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c34dc2324989467984ece8cf4853bb34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.236407 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c34dc2324989467984ece8cf4853bb34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.237958 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 399) took 4879 ms. Trace:
I20251027 21:07:59.238058 10040 rpcz_store.cc:276] 1027 21:07:54.354196 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.354240 (+ 44us) service_pool.cc:225] Handling call
1027 21:07:59.169075 (+4814835us) ts_tablet_manager.cc:1151] Deleting tablet 2b247829a11349b6835a2f18906e20a9
1027 21:07:59.169086 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.220293 (+ 51207us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.230127 (+ 9834us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.233611 (+ 3484us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4814604}
I20251027 21:07:59.238353 10628 raft_consensus.cc:493] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.238459 10031 ts_tablet_manager.cc:1434] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.068s user 0.008s sys 0.009s
I20251027 21:07:59.238530 10628 raft_consensus.cc:515] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.239269 10628 leader_election.cc:290] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:59.239871 10031 tablet_bootstrap.cc:492] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:59.240106 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21a8fc4a69f3486e92b1a68d10faed38" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:59.240504 9859 leader_election.cc:343] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.241427 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21a8fc4a69f3486e92b1a68d10faed38" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:59.241900 9860 leader_election.cc:343] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.242019 9860 leader_election.cc:304] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:59.242441 10560 raft_consensus.cc:2749] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:59.242726 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 400) took 4774 ms. Trace:
I20251027 21:07:59.242839 10040 rpcz_store.cc:276] 1027 21:07:54.466600 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.466665 (+ 65us) service_pool.cc:225] Handling call
1027 21:07:59.240961 (+4774296us) ts_tablet_manager.cc:1151] Deleting tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05
1027 21:07:59.240969 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.241388 (+ 419us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4774036}
W20251027 21:07:59.243301 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 already in progress: creating tablet
I20251027 21:07:59.243381 10031 tablet_bootstrap.cc:654] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.243716 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_219373cdafa946c68b0330d75aa69cc8" } new_table_name: "default.Table_c40e78267cfd4b84b63987b1c910818f" modify_external_catalogs: true
I20251027 21:07:59.244437 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 402) took 4745 ms. Trace:
I20251027 21:07:59.244550 10040 rpcz_store.cc:276] 1027 21:07:54.497760 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.497814 (+ 54us) service_pool.cc:225] Handling call
1027 21:07:59.243268 (+4745454us) ts_tablet_manager.cc:1151] Deleting tablet a033551a591146039b017b4193b221ca
1027 21:07:59.243276 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.243595 (+ 319us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4745188}
W20251027 21:07:59.245007 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet a033551a591146039b017b4193b221ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a033551a591146039b017b4193b221ca already in progress: creating tablet
I20251027 21:07:59.247403 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 403) took 4747 ms. Trace:
I20251027 21:07:59.247500 10040 rpcz_store.cc:276] 1027 21:07:54.497916 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.497967 (+ 51us) service_pool.cc:225] Handling call
1027 21:07:59.244884 (+4746917us) ts_tablet_manager.cc:1151] Deleting tablet 9ed4f9c61ad54e1292281b30bcd19533
1027 21:07:59.244894 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.245050 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4746638}
I20251027 21:07:59.249454 10032 tablet_bootstrap.cc:492] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:59.249627 10032 ts_tablet_manager.cc:1403] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.015s
I20251027 21:07:59.250420 10032 raft_consensus.cc:359] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:07:59.250754 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 9ed4f9c61ad54e1292281b30bcd19533 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ed4f9c61ad54e1292281b30bcd19533 already in progress: creating tablet
I20251027 21:07:59.250808 10560 raft_consensus.cc:493] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.250927 10560 raft_consensus.cc:515] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.251546 10560 leader_election.cc:290] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:59.252123 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92a2393cff8044028c86c8cd2bccd930" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:59.252614 9860 leader_election.cc:343] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.252874 10043 ts_tablet_manager.cc:1929] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:07:59.252990 10043 log.cc:1199] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/1776590d8387494da5dfbd08db17f4b1
I20251027 21:07:59.253357 10043 ts_tablet_manager.cc:1950] T 1776590d8387494da5dfbd08db17f4b1 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:59.250991 10032 raft_consensus.cc:385] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.254755 10032 raft_consensus.cc:740] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.255062 10032 consensus_queue.cc:260] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.255535 10032 ts_tablet_manager.cc:1434] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251027 21:07:59.257316 10032 tablet_bootstrap.cc:492] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:59.258363 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet 0a58cd07e1bb4ffdbf39e7e16a46022a from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:59.261082 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 404) took 4736 ms. Trace:
I20251027 21:07:59.262876 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92a2393cff8044028c86c8cd2bccd930" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:59.262955 9896 tablet_service.cc:1552] Processing DeleteTablet for tablet c5f2e60caa824b5db2330b423221873c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:59.262979 10040 rpcz_store.cc:276] 1027 21:07:54.511720 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.512204 (+ 484us) service_pool.cc:225] Handling call
1027 21:07:59.247710 (+4735506us) ts_tablet_manager.cc:1151] Deleting tablet e7df6f0acec34d399aa25b78c3ed7c7a
1027 21:07:59.247717 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.247784 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4735274}
W20251027 21:07:59.263809 9859 leader_election.cc:343] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.263880 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 405) took 4751 ms. Trace:
I20251027 21:07:59.263918 9859 leader_election.cc:304] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:59.263964 10040 rpcz_store.cc:276] 1027 21:07:54.511873 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.512478 (+ 605us) service_pool.cc:225] Handling call
1027 21:07:59.263218 (+4750740us) ts_tablet_manager.cc:1151] Deleting tablet f1aa5f7f76cb4a1ba15f016942f17067
1027 21:07:59.263226 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.263318 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4750587}
I20251027 21:07:59.264350 10560 raft_consensus.cc:2749] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:59.264570 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 406) took 4750 ms. Trace:
I20251027 21:07:59.264673 10040 rpcz_store.cc:276] 1027 21:07:54.514182 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.514463 (+ 281us) service_pool.cc:225] Handling call
1027 21:07:59.264166 (+4749703us) ts_tablet_manager.cc:1151] Deleting tablet f775a7e42bdc4910bdb604b26a28a7f4
1027 21:07:59.264174 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.264260 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4749447}
W20251027 21:07:59.265097 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet e7df6f0acec34d399aa25b78c3ed7c7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7df6f0acec34d399aa25b78c3ed7c7a already in progress: creating tablet
I20251027 21:07:59.265134 10040 tablet_replica.cc:333] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
W20251027 21:07:59.265300 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet f1aa5f7f76cb4a1ba15f016942f17067 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1aa5f7f76cb4a1ba15f016942f17067 already in progress: creating tablet
I20251027 21:07:59.265348 10040 raft_consensus.cc:2243] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.265580 10040 raft_consensus.cc:2272] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.266242 10040 ts_tablet_manager.cc:1916] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.268978 10031 tablet_bootstrap.cc:492] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:59.269981 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet ae124514964440c5b88b16ac9f8a2567 with delete_type TABLET_DATA_DELETED (replaced by tablet 01e9db9eaf8e41c6a2ee13b9ed127421) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:59.270502 10032 tablet_bootstrap.cc:654] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.271047 10031 ts_tablet_manager.cc:1403] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.001s
I20251027 21:07:59.275513 10041 ts_tablet_manager.cc:1929] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.275789 10041 log.cc:1199] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/c400101d2a39490e83722500839198cd
I20251027 21:07:59.276180 10031 raft_consensus.cc:359] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.276921 10041 ts_tablet_manager.cc:1950] T c400101d2a39490e83722500839198cd P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:59.277391 10043 tablet_replica.cc:333] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:59.277619 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 1776590d8387494da5dfbd08db17f4b1 (table default.Table_b779f2756a924ab6ba1bc1702a412e06 [id=8ed572891f9b41cba8ed984911fa6b11]) successfully deleted
I20251027 21:07:59.277716 10031 raft_consensus.cc:385] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.277611 10043 raft_consensus.cc:2243] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.277833 10031 raft_consensus.cc:740] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.278021 10043 raft_consensus.cc:2272] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:07:59.278079 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 133e474f9133487f89c44dbfa6266ca7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 133e474f9133487f89c44dbfa6266ca7 already in progress: creating tablet
I20251027 21:07:59.278247 10031 consensus_queue.cc:260] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
W20251027 21:07:59.278544 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 23673513189f47638dbbc737da0991d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23673513189f47638dbbc737da0991d1 already in progress: creating tablet
I20251027 21:07:59.279235 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6666bff9087744179f5eb067a74af9f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.279748 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6666bff9087744179f5eb067a74af9f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.283018 10041 tablet_replica.cc:333] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:59.283144 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet c400101d2a39490e83722500839198cd (table default.Table_97b01c1840c543838263ca4222695f54 [id=bea51a58fda64d40916f306c4aa8ab4e]) successfully deleted
I20251027 21:07:59.283195 10041 raft_consensus.cc:2243] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.283337 10041 raft_consensus.cc:2272] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.283782 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e6dc3cc1aeef48a48107dc699cbb2238"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.284299 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6dc3cc1aeef48a48107dc699cbb2238 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.284786 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_fa60bcd5277c4bf1bbf49d0e971340ba" } new_table_name: "default.Table_a04bcb2698ec467e8f85f4408ae2ca01" modify_external_catalogs: true
I20251027 21:07:59.285007 10040 ts_tablet_manager.cc:1929] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.285126 10040 log.cc:1199] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/c49775486ff24686b0f45c47e2757aba
I20251027 21:07:59.285553 10040 ts_tablet_manager.cc:1950] T c49775486ff24686b0f45c47e2757aba P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:59.285636 10041 ts_tablet_manager.cc:1916] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.285842 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 408) took 4753 ms. Trace:
I20251027 21:07:59.286335 10040 rpcz_store.cc:276] 1027 21:07:54.532300 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.532462 (+ 162us) service_pool.cc:225] Handling call
1027 21:07:59.264908 (+4732446us) ts_tablet_manager.cc:1151] Deleting tablet c49775486ff24686b0f45c47e2757aba
1027 21:07:59.264916 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.275422 (+ 10506us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.284313 (+ 8891us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.285813 (+ 1500us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4732237}
I20251027 21:07:59.286726 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet c49775486ff24686b0f45c47e2757aba (table default.Table_b534fdb9e3084d6f9656dfdbb0e01750 [id=dcefc291ce454343964e99be8876837c]) successfully deleted
I20251027 21:07:59.290732 10043 ts_tablet_manager.cc:1916] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.291647 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet a9cc64ad85334daa9a09538e24ebd8a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:59.293114 10031 ts_tablet_manager.cc:1434] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20251027 21:07:59.296345 10031 tablet_bootstrap.cc:492] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:59.297695 10029 tablet_bootstrap.cc:492] T 6bf9c2a78aac42528d67846dc58adc40 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:59.297928 10029 ts_tablet_manager.cc:1403] T 6bf9c2a78aac42528d67846dc58adc40 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.119s user 0.003s sys 0.001s
I20251027 21:07:59.299355 10029 raft_consensus.cc:359] T 6bf9c2a78aac42528d67846dc58adc40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.299579 10029 raft_consensus.cc:385] T 6bf9c2a78aac42528d67846dc58adc40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.299666 10032 tablet_bootstrap.cc:492] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:59.299700 10029 raft_consensus.cc:740] T 6bf9c2a78aac42528d67846dc58adc40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.299821 10032 ts_tablet_manager.cc:1403] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.000s
I20251027 21:07:59.300035 10029 consensus_queue.cc:260] T 6bf9c2a78aac42528d67846dc58adc40 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.300908 10032 raft_consensus.cc:359] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.301074 10032 raft_consensus.cc:385] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.301123 10032 raft_consensus.cc:740] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.301488 10029 ts_tablet_manager.cc:1434] T 6bf9c2a78aac42528d67846dc58adc40 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:07:59.301463 10032 consensus_queue.cc:260] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.301788 10031 tablet_bootstrap.cc:654] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.302405 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 410) took 4756 ms. Trace:
I20251027 21:07:59.302501 10040 rpcz_store.cc:276] 1027 21:07:54.545814 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.545865 (+ 51us) service_pool.cc:225] Handling call
1027 21:07:59.302263 (+4756398us) ts_tablet_manager.cc:1151] Deleting tablet 53cac7817db440e79e7416d625a9b2c8
1027 21:07:59.302272 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.302373 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4756202}
I20251027 21:07:59.302966 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 411) took 4744 ms. Trace:
I20251027 21:07:59.303061 10040 rpcz_store.cc:276] 1027 21:07:54.558904 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.558959 (+ 55us) service_pool.cc:225] Handling call
1027 21:07:59.302836 (+4743877us) ts_tablet_manager.cc:1151] Deleting tablet a2d26784bbb24ae3893103802c41affb
1027 21:07:59.302845 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.302936 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4724274}
W20251027 21:07:59.303459 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 53cac7817db440e79e7416d625a9b2c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53cac7817db440e79e7416d625a9b2c8 already in progress: creating tablet
W20251027 21:07:59.303710 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet a2d26784bbb24ae3893103802c41affb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2d26784bbb24ae3893103802c41affb already in progress: creating tablet
I20251027 21:07:59.305934 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bf9c2a78aac42528d67846dc58adc40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:59.306502 10029 tablet_bootstrap.cc:492] T 9c239216cb8846f0b1aaa360ddb83321 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:59.307861 9896 tablet_service.cc:1552] Processing DeleteTablet for tablet d52d71f4976143e4a514350621ebe309 with delete_type TABLET_DATA_DELETED (replaced by tablet e419e588b31d48c8bc19d7948081cc81) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:59.308938 10560 raft_consensus.cc:493] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.309053 10560 raft_consensus.cc:515] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.309500 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 419) took 4615 ms. Trace:
I20251027 21:07:59.309549 10032 ts_tablet_manager.cc:1434] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251027 21:07:59.309595 10560 leader_election.cc:290] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:59.309594 10040 rpcz_store.cc:276] 1027 21:07:54.693758 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.693808 (+ 50us) service_pool.cc:225] Handling call
1027 21:07:59.309389 (+4615581us) ts_tablet_manager.cc:1151] Deleting tablet 630ae4d66f99474382814cc32cc60ea0
1027 21:07:59.309398 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.309472 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4614970}
I20251027 21:07:59.310199 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe24af9838954a9bb680474800642e62" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:59.310426 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 630ae4d66f99474382814cc32cc60ea0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 630ae4d66f99474382814cc32cc60ea0 already in progress: creating tablet
W20251027 21:07:59.310657 9859 leader_election.cc:343] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.313120 10032 tablet_bootstrap.cc:492] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:59.314041 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 420) took 4598 ms. Trace:
I20251027 21:07:59.314205 10040 rpcz_store.cc:276] 1027 21:07:54.715598 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.715705 (+ 107us) service_pool.cc:225] Handling call
1027 21:07:59.313361 (+4597656us) ts_tablet_manager.cc:1151] Deleting tablet cef2bfee222f45e6b6f20c6d63ff44d4
1027 21:07:59.313369 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.314010 (+ 641us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4597426}
W20251027 21:07:59.314611 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet cef2bfee222f45e6b6f20c6d63ff44d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cef2bfee222f45e6b6f20c6d63ff44d4 already in progress: creating tablet
I20251027 21:07:59.315167 10668 raft_consensus.cc:493] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.315289 10668 raft_consensus.cc:515] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.315922 10668 leader_election.cc:290] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:59.316699 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9c8bc163f60741c3bf7fed269c243545"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.317315 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c8bc163f60741c3bf7fed269c243545 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:59.317684 9710 leader_election.cc:343] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: e07b8aa0a41241d79254c4dae8738e79
W20251027 21:07:59.318045 9711 leader_election.cc:343] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: e07b8aa0a41241d79254c4dae8738e79
I20251027 21:07:59.318189 9711 leader_election.cc:304] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:59.309909 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe24af9838954a9bb680474800642e62" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:59.318711 9860 leader_election.cc:343] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.318791 10668 raft_consensus.cc:2749] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:59.318816 9860 leader_election.cc:304] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:59.319134 10032 tablet_bootstrap.cc:654] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.319378 10560 raft_consensus.cc:2749] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:59.319922 10040 tablet_replica.cc:333] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:59.320132 10040 raft_consensus.cc:2243] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.320295 10040 raft_consensus.cc:2272] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.320451 10029 tablet_bootstrap.cc:654] T 9c239216cb8846f0b1aaa360ddb83321 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.320777 10040 ts_tablet_manager.cc:1916] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.323522 10041 ts_tablet_manager.cc:1929] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.323648 10041 log.cc:1199] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/6f1129eecfb54448965222269d9c32b7
I20251027 21:07:59.324985 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_735a5a1d675c4c44a98e6793ab7f21fd" } new_table_name: "default.Table_e7d7ba7c45a54852b436abe87effeaa0" modify_external_catalogs: true
W20251027 21:07:59.327906 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 95446c4333424f10a71829ce32d857e4 consensus state has no leader
I20251027 21:07:59.330024 10031 tablet_bootstrap.cc:492] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:59.330199 10031 ts_tablet_manager.cc:1403] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.001s
I20251027 21:07:59.330996 10031 raft_consensus.cc:359] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.331182 10031 raft_consensus.cc:385] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.331274 10031 raft_consensus.cc:740] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.331490 10031 consensus_queue.cc:260] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.331991 10041 ts_tablet_manager.cc:1950] T 6f1129eecfb54448965222269d9c32b7 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:59.332888 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_56aebdec46ea4837a4a42ff7c45aebba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:59.334167 10031 ts_tablet_manager.cc:1434] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:07:59.335673 10031 tablet_bootstrap.cc:492] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:59.338081 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 6f1129eecfb54448965222269d9c32b7 (table default.Table_f16bc07fcb4c48b7969d13b8341aefdc [id=b5f4e2be1a5e42f58d98b64b8700ef1a]) successfully deleted
I20251027 21:07:59.338929 10032 tablet_bootstrap.cc:492] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:59.338972 10041 tablet_replica.cc:333] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:59.339262 10043 ts_tablet_manager.cc:1929] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.339076 10032 ts_tablet_manager.cc:1403] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.004s
I20251027 21:07:59.340063 10043 log.cc:1199] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/583fba7c750e4ac7b768da447002c211
I20251027 21:07:59.340467 10041 raft_consensus.cc:2243] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.341002 10032 raft_consensus.cc:359] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.341172 10032 raft_consensus.cc:385] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.341229 10032 raft_consensus.cc:740] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.341539 10032 consensus_queue.cc:260] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.342010 10032 ts_tablet_manager.cc:1434] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:59.341595 10041 raft_consensus.cc:2272] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.343571 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2da8f285aede4af7b444347f7183293f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.345187 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56aebdec46ea4837a4a42ff7c45aebba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:07:59.347460 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2da8f285aede4af7b444347f7183293f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.349895 9896 tablet_service.cc:1552] Processing DeleteTablet for tablet abc7458fcfb94985acf76d77c0292b08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:59.349917 10032 tablet_bootstrap.cc:492] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:59.351972 10031 tablet_bootstrap.cc:654] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.354526 10041 ts_tablet_manager.cc:1916] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.354724 10043 ts_tablet_manager.cc:1950] T 583fba7c750e4ac7b768da447002c211 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
W20251027 21:07:59.356796 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93a3b22cdc1543aebcc7fe72f2f2c0ae consensus state has no leader
I20251027 21:07:59.357793 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c34dc2324989467984ece8cf4853bb34" } modify_external_catalogs: true
I20251027 21:07:59.358014 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c34dc2324989467984ece8cf4853bb34" } modify_external_catalogs: true
I20251027 21:07:59.360965 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 94e94cd50c91429d8da9804149501c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:59.355247 10032 tablet_bootstrap.cc:654] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.370661 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 583fba7c750e4ac7b768da447002c211 (table default.Table_660eb77e69884fe5bd5e65b47ea91dde [id=14daa3e199fe4fa3ba2de06dd6be4330]) successfully deleted
I20251027 21:07:59.371061 10043 tablet_replica.cc:333] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:59.371326 10043 raft_consensus.cc:2243] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.371546 10040 ts_tablet_manager.cc:1929] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.371563 10043 raft_consensus.cc:2272] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.371697 10040 log.cc:1199] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/e07b8aa0a41241d79254c4dae8738e79
I20251027 21:07:59.372114 10040 ts_tablet_manager.cc:1950] T e07b8aa0a41241d79254c4dae8738e79 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:59.372455 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 421) took 4636 ms. Trace:
I20251027 21:07:59.372555 10040 rpcz_store.cc:276] 1027 21:07:54.736220 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.736294 (+ 74us) service_pool.cc:225] Handling call
1027 21:07:59.319862 (+4583568us) ts_tablet_manager.cc:1151] Deleting tablet e07b8aa0a41241d79254c4dae8738e79
1027 21:07:59.319871 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.336615 (+ 16744us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.371429 (+ 34814us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.372400 (+ 971us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4583343}
I20251027 21:07:59.372938 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet e07b8aa0a41241d79254c4dae8738e79 (table default.Table_e68b56d3ddd94f308243ec6a03ea9e6e [id=0a94e88b69094ffe8be3367470be86da]) successfully deleted
I20251027 21:07:59.373513 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 424) took 4611 ms. Trace:
I20251027 21:07:59.373716 10043 ts_tablet_manager.cc:1916] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.373616 10040 rpcz_store.cc:276] 1027 21:07:54.761644 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.790187 (+ 28543us) service_pool.cc:225] Handling call
1027 21:07:59.372833 (+4582646us) ts_tablet_manager.cc:1151] Deleting tablet 815e12edcd3847ff8485ba768291a824
1027 21:07:59.372850 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.372946 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4582427}
I20251027 21:07:59.374009 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 433) took 4519 ms. Trace:
W20251027 21:07:59.374039 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 815e12edcd3847ff8485ba768291a824 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 815e12edcd3847ff8485ba768291a824 already in progress: creating tablet
I20251027 21:07:59.382607 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet d2203ee62f7e404c95c5f86b5ae85feb from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:59.384439 10031 tablet_bootstrap.cc:492] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:59.384598 10031 ts_tablet_manager.cc:1403] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.049s user 0.005s sys 0.004s
I20251027 21:07:59.385222 10041 ts_tablet_manager.cc:1929] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.385746 10031 raft_consensus.cc:359] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.385967 10031 raft_consensus.cc:385] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.386051 10031 raft_consensus.cc:740] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.386404 10668 raft_consensus.cc:493] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.386375 10031 consensus_queue.cc:260] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.386637 10041 log.cc:1199] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/92a2393cff8044028c86c8cd2bccd930
I20251027 21:07:59.386574 10668 raft_consensus.cc:515] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.386924 10031 ts_tablet_manager.cc:1434] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251027 21:07:59.387236 10031 tablet_bootstrap.cc:492] T 7f7ad06a15c04efbb701634e14904351 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:59.387264 10668 leader_election.cc:290] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:59.389107 10031 tablet_bootstrap.cc:654] T 7f7ad06a15c04efbb701634e14904351 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.387260 10041 ts_tablet_manager.cc:1950] T 92a2393cff8044028c86c8cd2bccd930 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
W20251027 21:07:59.389307 9711 leader_election.cc:343] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 779b7f63b0fa4f9ab47cf5799b3e085d
I20251027 21:07:59.389322 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b6905d2ba9e649178b7b4cb0a420a516" } modify_external_catalogs: true
I20251027 21:07:59.389544 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b6905d2ba9e649178b7b4cb0a420a516" } modify_external_catalogs: true
W20251027 21:07:59.389992 9710 leader_election.cc:343] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 779b7f63b0fa4f9ab47cf5799b3e085d
I20251027 21:07:59.390103 9710 leader_election.cc:304] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:59.390242 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 92a2393cff8044028c86c8cd2bccd930 (table default.Table_1d766c40686340d59cc4680fbf8a2dbb [id=e5fba763a46b488cb19c738cee468ad1]) successfully deleted
I20251027 21:07:59.390384 10665 raft_consensus.cc:2749] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:59.390420 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_41ae08dcac5e4355b09bd4d15a35c229" } modify_external_catalogs: true
I20251027 21:07:59.390571 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_41ae08dcac5e4355b09bd4d15a35c229" } modify_external_catalogs: true
I20251027 21:07:59.390749 10515 raft_consensus.cc:493] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.390893 10515 raft_consensus.cc:515] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.391590 10515 leader_election.cc:290] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:59.391589 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5a979a51f58942db97c3897dcd5feb08" } modify_external_catalogs: true
I20251027 21:07:59.391736 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5a979a51f58942db97c3897dcd5feb08" } modify_external_catalogs: true
I20251027 21:07:59.392231 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7aa8c538b6c849a28a28ffe706306891" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:59.392763 9566 leader_election.cc:343] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.393633 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 77797eea3e9c4f8099dc9d58f116200b with delete_type TABLET_DATA_DELETED (replaced by tablet 631ff72613734913818a3cad24f20dbc) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:59.393779 10032 tablet_bootstrap.cc:492] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:59.393747 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7aa8c538b6c849a28a28ffe706306891" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:07:59.393880 10032 ts_tablet_manager.cc:1403] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.007s
I20251027 21:07:59.393904 9924 raft_consensus.cc:2468] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:07:59.394454 9568 leader_election.cc:304] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:59.394927 10515 raft_consensus.cc:2804] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:07:59.394941 10032 raft_consensus.cc:359] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:59.395038 10515 raft_consensus.cc:493] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:07:59.395110 10032 raft_consensus.cc:385] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.395145 10515 raft_consensus.cc:3060] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:59.395172 10032 raft_consensus.cc:740] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.395373 10032 consensus_queue.cc:260] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:59.399214 10515 raft_consensus.cc:515] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.399484 10032 ts_tablet_manager.cc:1434] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251027 21:07:59.374114 10040 rpcz_store.cc:276] 1027 21:07:54.854900 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.855474 (+ 574us) service_pool.cc:225] Handling call
1027 21:07:59.373873 (+4518399us) ts_tablet_manager.cc:1151] Deleting tablet cdbde6272af147829ecadd67d634f049
1027 21:07:59.373882 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.373958 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4518206}
W20251027 21:07:59.401563 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 29693cf06bc24930870dc60457126519 consensus state has no leader
W20251027 21:07:59.401994 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet cdbde6272af147829ecadd67d634f049 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cdbde6272af147829ecadd67d634f049 already in progress: creating tablet
I20251027 21:07:59.402019 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 432) took 4547 ms. Trace:
I20251027 21:07:59.402105 10040 rpcz_store.cc:276] 1027 21:07:54.854780 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.856748 (+ 1968us) service_pool.cc:225] Handling call
1027 21:07:59.401830 (+4545082us) ts_tablet_manager.cc:1151] Deleting tablet 42371496a7494e77a7bf1169de5b083a
1027 21:07:59.401850 (+ 20us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.401964 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4544890}
W20251027 21:07:59.402513 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 42371496a7494e77a7bf1169de5b083a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42371496a7494e77a7bf1169de5b083a already in progress: creating tablet
I20251027 21:07:59.403272 10032 tablet_bootstrap.cc:492] T dc589852710342f7bd017193e4075303 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:59.403600 10040 tablet_replica.cc:333] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:59.403671 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7aa8c538b6c849a28a28ffe706306891" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:07:59.403789 10040 raft_consensus.cc:2243] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.403812 9924 raft_consensus.cc:3060] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:07:59.403611 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7aa8c538b6c849a28a28ffe706306891" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:07:59.403985 10040 raft_consensus.cc:2272] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:07:59.404386 9566 leader_election.cc:343] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.404946 10040 ts_tablet_manager.cc:1916] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.405576 10032 tablet_bootstrap.cc:654] T dc589852710342f7bd017193e4075303 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.407871 9924 raft_consensus.cc:2468] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:07:59.408407 10029 tablet_bootstrap.cc:492] T 9c239216cb8846f0b1aaa360ddb83321 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:59.408511 9568 leader_election.cc:304] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:59.408648 10029 ts_tablet_manager.cc:1403] T 9c239216cb8846f0b1aaa360ddb83321 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.102s user 0.003s sys 0.001s
I20251027 21:07:59.408931 10529 raft_consensus.cc:2804] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:07:59.409430 10029 raft_consensus.cc:359] T 9c239216cb8846f0b1aaa360ddb83321 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.409708 10029 raft_consensus.cc:385] T 9c239216cb8846f0b1aaa360ddb83321 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.409827 10029 raft_consensus.cc:740] T 9c239216cb8846f0b1aaa360ddb83321 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.410162 10029 consensus_queue.cc:260] T 9c239216cb8846f0b1aaa360ddb83321 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.410997 10515 leader_election.cc:290] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:59.411397 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4fab264efb244206a3717a69e90455a6" } new_table_name: "default.Table_573e41ad7d2845c8a4af93a93ed20525" modify_external_catalogs: true
I20251027 21:07:59.414569 10560 raft_consensus.cc:493] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.416241 10560 raft_consensus.cc:515] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.416962 10560 leader_election.cc:290] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:59.417631 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "533f34d4872444278477abcce89da9ec" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:59.418639 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "533f34d4872444278477abcce89da9ec" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:59.418761 9860 leader_election.cc:343] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:07:59.419322 9859 leader_election.cc:343] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.419438 9859 leader_election.cc:304] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:59.420365 10560 raft_consensus.cc:2749] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:07:59.421612 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet a4d26f73b8534f08bcaa386d4de75c01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4d26f73b8534f08bcaa386d4de75c01 already in progress: creating tablet
I20251027 21:07:59.422775 10043 ts_tablet_manager.cc:1929] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251027 21:07:59.423954 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 8b30052ff19744a9bf4bfaaa1bc77b36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b30052ff19744a9bf4bfaaa1bc77b36 already in progress: creating tablet
I20251027 21:07:59.424718 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5639561ca4aa45e5bc117ecc76d943d9" } new_table_name: "default.Table_9406473db81e445ab8be74949a2545c3" modify_external_catalogs: true
I20251027 21:07:59.426760 9303 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet f830eb2edd704cc0a400e8e4a3682a2c
I20251027 21:07:59.427692 10041 tablet_replica.cc:333] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:59.428371 10041 raft_consensus.cc:2243] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.428701 10043 log.cc:1199] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/b87e809ff897417c88e5edc19e2d591e
I20251027 21:07:59.429560 10043 ts_tablet_manager.cc:1950] T b87e809ff897417c88e5edc19e2d591e P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:59.429917 10041 raft_consensus.cc:2272] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.430969 10041 ts_tablet_manager.cc:1916] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.434170 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet b87e809ff897417c88e5edc19e2d591e (table default.Table_d2bf88aca4cc44df99d13783ea2722f8 [id=7de1278fe67640ff8540392837b2eeae]) successfully deleted
I20251027 21:07:59.435084 10529 raft_consensus.cc:697] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:07:59.435667 10529 consensus_queue.cc:237] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.438182 10040 ts_tablet_manager.cc:1929] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.438338 10040 log.cc:1199] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/1e5f5b57f5d948f3b2352451ccaacb09
I20251027 21:07:59.442662 10029 ts_tablet_manager.cc:1434] T 9c239216cb8846f0b1aaa360ddb83321 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20251027 21:07:59.443085 10029 tablet_bootstrap.cc:492] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:59.444471 10043 tablet_replica.cc:333] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:59.444833 10043 raft_consensus.cc:2243] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251027 21:07:59.445451 9298 catalog_manager.cc:4952] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet cfce0e5c837743ea969374244163be3e because the tablet was not found. No further retry: Not found: Tablet not found: cfce0e5c837743ea969374244163be3e
W20251027 21:07:59.447047 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c25be263facd42c5bb6fa421160fbef0 consensus state has no leader
I20251027 21:07:59.447572 10043 raft_consensus.cc:2272] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.455693 9310 catalog_manager.cc:5649] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20251027 21:07:59.456099 9310 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 7aa8c538b6c849a28a28ffe706306891 (table default.Table_b779f2756a924ab6ba1bc1702a412e06 [id=8ed572891f9b41cba8ed984911fa6b11]). Expected version 1 got 0
I20251027 21:07:59.457031 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7714994006124254a5feb37d68ce5808"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.457616 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7714994006124254a5feb37d68ce5808 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.459733 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 23f10b4258c8416cb5eba963f2ca4452 (DEFAULT_TABLE table=default.Table_645fb813ce0a4245963a221252ac8d87 [id=3f768fdddb604293ad8cf4a39ed9b7d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.459867 9306 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet b8856c5dc8f144358445165324bfb477 (table default.Table_203191e192be4e3a9a7477b4313ef4f4 [id=5da6bd4463e24c03accd9985f72b6ebb]). Expected version 3 got 0
I20251027 21:07:59.460582 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 23f10b4258c8416cb5eba963f2ca4452 (DEFAULT_TABLE table=default.Table_645fb813ce0a4245963a221252ac8d87 [id=3f768fdddb604293ad8cf4a39ed9b7d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.461098 10040 ts_tablet_manager.cc:1950] T 1e5f5b57f5d948f3b2352451ccaacb09 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:59.463647 9736 tablet_service.cc:1505] Processing CreateTablet for tablet d08b3bf501a441b5b7bf6a273f5e4e37 (DEFAULT_TABLE table=default.Table_b5a125355aa246cda7ea2d7a0b2acf62 [id=75bc9e4eb8954685b3940b81d3e0afd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.463120 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 437) took 4562 ms. Trace:
I20251027 21:07:59.464439 9306 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet b8856c5dc8f144358445165324bfb477 (table default.Table_203191e192be4e3a9a7477b4313ef4f4 [id=5da6bd4463e24c03accd9985f72b6ebb]). Expected version 3 got 0
I20251027 21:07:59.466493 9893 tablet_service.cc:1505] Processing CreateTablet for tablet d08b3bf501a441b5b7bf6a273f5e4e37 (DEFAULT_TABLE table=default.Table_b5a125355aa246cda7ea2d7a0b2acf62 [id=75bc9e4eb8954685b3940b81d3e0afd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.467401 10031 tablet_bootstrap.cc:492] T 7f7ad06a15c04efbb701634e14904351 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:59.467407 10043 ts_tablet_manager.cc:1916] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.467659 10031 ts_tablet_manager.cc:1403] T 7f7ad06a15c04efbb701634e14904351 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.081s user 0.006s sys 0.003s
I20251027 21:07:59.467999 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a5ded5d12b9d45128034b8672f42f808"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.468629 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5ded5d12b9d45128034b8672f42f808 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.469182 10031 raft_consensus.cc:359] T 7f7ad06a15c04efbb701634e14904351 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.469455 10031 raft_consensus.cc:385] T 7f7ad06a15c04efbb701634e14904351 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.469635 10031 raft_consensus.cc:740] T 7f7ad06a15c04efbb701634e14904351 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
W20251027 21:07:59.470485 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3b2c479d11794a6a99b5e05d3954ad54 consensus state has no leader
I20251027 21:07:59.470599 9744 tablet_service.cc:1505] Processing CreateTablet for tablet 211ac01a37f8434f95aa6a63970296fa (DEFAULT_TABLE table=default.Table_0efd5aacf5b54ded90984a7c702d12d6 [id=23a8799add7949e8948c2088f3172684]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.459578 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 23f10b4258c8416cb5eba963f2ca4452 (DEFAULT_TABLE table=default.Table_645fb813ce0a4245963a221252ac8d87 [id=3f768fdddb604293ad8cf4a39ed9b7d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.462870 9602 tablet_service.cc:1505] Processing CreateTablet for tablet d08b3bf501a441b5b7bf6a273f5e4e37 (DEFAULT_TABLE table=default.Table_b5a125355aa246cda7ea2d7a0b2acf62 [id=75bc9e4eb8954685b3940b81d3e0afd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.445140 10029 tablet_bootstrap.cc:654] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.469990 10031 consensus_queue.cc:260] T 7f7ad06a15c04efbb701634e14904351 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.471267 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23f10b4258c8416cb5eba963f2ca4452. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.471467 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d08b3bf501a441b5b7bf6a273f5e4e37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.469022 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 211ac01a37f8434f95aa6a63970296fa (DEFAULT_TABLE table=default.Table_0efd5aacf5b54ded90984a7c702d12d6 [id=23a8799add7949e8948c2088f3172684]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.471979 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 211ac01a37f8434f95aa6a63970296fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.473553 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23f10b4258c8416cb5eba963f2ca4452. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.473630 10031 ts_tablet_manager.cc:1434] T 7f7ad06a15c04efbb701634e14904351 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251027 21:07:59.474007 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d08b3bf501a441b5b7bf6a273f5e4e37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.474252 10031 tablet_bootstrap.cc:492] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:59.464636 10040 rpcz_store.cc:276] 1027 21:07:54.900359 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.910190 (+ 9831us) service_pool.cc:225] Handling call
1027 21:07:59.403498 (+4493308us) ts_tablet_manager.cc:1151] Deleting tablet 1e5f5b57f5d948f3b2352451ccaacb09
1027 21:07:59.403508 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.413102 (+ 9594us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.438016 (+ 24914us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.462579 (+ 24563us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4493064}
I20251027 21:07:59.477629 10529 raft_consensus.cc:493] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.478694 10032 tablet_bootstrap.cc:492] T dc589852710342f7bd017193e4075303 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:59.478857 10032 ts_tablet_manager.cc:1403] T dc589852710342f7bd017193e4075303 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.076s user 0.014s sys 0.040s
I20251027 21:07:59.479753 10032 raft_consensus.cc:359] T dc589852710342f7bd017193e4075303 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.479952 10032 raft_consensus.cc:385] T dc589852710342f7bd017193e4075303 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.480020 10032 raft_consensus.cc:740] T dc589852710342f7bd017193e4075303 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.479619 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 442) took 4541 ms. Trace:
I20251027 21:07:59.480252 10032 consensus_queue.cc:260] T dc589852710342f7bd017193e4075303 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.481540 10032 ts_tablet_manager.cc:1434] T dc589852710342f7bd017193e4075303 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:07:59.482913 10032 tablet_bootstrap.cc:492] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:59.484490 10032 tablet_bootstrap.cc:654] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.482493 10040 rpcz_store.cc:276] 1027 21:07:54.937511 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:54.938040 (+ 529us) service_pool.cc:225] Handling call
1027 21:07:59.478201 (+4540161us) ts_tablet_manager.cc:1151] Deleting tablet 7292bc6c22ca4ab5b8f53a69520833ba
1027 21:07:59.478210 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.479310 (+ 1100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4538430}
I20251027 21:07:59.486652 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 455) took 4479 ms. Trace:
I20251027 21:07:59.486821 10040 rpcz_store.cc:276] 1027 21:07:55.007449 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.007793 (+ 344us) service_pool.cc:225] Handling call
1027 21:07:59.486408 (+4478615us) ts_tablet_manager.cc:1151] Deleting tablet 27bd3b2a011648b2b8f2f92944513751
1027 21:07:59.486416 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.486592 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4255893}
W20251027 21:07:59.487331 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 27bd3b2a011648b2b8f2f92944513751 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27bd3b2a011648b2b8f2f92944513751 already in progress: creating tablet
I20251027 21:07:59.487627 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 456) took 4480 ms. Trace:
I20251027 21:07:59.487818 10040 rpcz_store.cc:276] 1027 21:07:55.007536 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.007911 (+ 375us) service_pool.cc:225] Handling call
1027 21:07:59.487379 (+4479468us) ts_tablet_manager.cc:1151] Deleting tablet 533f34d4872444278477abcce89da9ec
1027 21:07:59.487391 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.487569 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4255234}
I20251027 21:07:59.488348 9308 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 7f7ad06a15c04efbb701634e14904351 (table default.Table_318d41ccfb2c4d2c99bafcea875143a1 [id=2bc39b5bc02645ef830eee14e92707db]). Expected version 1 got 0
I20251027 21:07:59.489130 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 0820648719a34f379507efced9cc434b
I20251027 21:07:59.488654 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 457) took 4480 ms. Trace:
I20251027 21:07:59.489611 10040 rpcz_store.cc:276] 1027 21:07:55.007645 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.007812 (+ 167us) service_pool.cc:225] Handling call
1027 21:07:59.488460 (+4480648us) ts_tablet_manager.cc:1151] Deleting tablet 61ba8d7df6fd45abb6f5231ee8c4ec38
1027 21:07:59.488469 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.488598 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4256108}
I20251027 21:07:59.490044 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 0820648719a34f379507efced9cc434b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:59.490479 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 454) took 4483 ms. Trace:
I20251027 21:07:59.490677 10040 rpcz_store.cc:276] 1027 21:07:55.007346 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.007772 (+ 426us) service_pool.cc:225] Handling call
1027 21:07:59.490241 (+4482469us) ts_tablet_manager.cc:1151] Deleting tablet 0a58cd07e1bb4ffdbf39e7e16a46022a
1027 21:07:59.490250 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.490388 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4257709}
I20251027 21:07:59.491051 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_066671f8fa8b403386935bcda7a8d095"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:59.491181 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet dc589852710342f7bd017193e4075303 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:59.491575 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_066671f8fa8b403386935bcda7a8d095 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.492300 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 211ac01a37f8434f95aa6a63970296fa (DEFAULT_TABLE table=default.Table_0efd5aacf5b54ded90984a7c702d12d6 [id=23a8799add7949e8948c2088f3172684]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.492715 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 211ac01a37f8434f95aa6a63970296fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.492952 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 0820648719a34f379507efced9cc434b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:59.492976 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 61ba8d7df6fd45abb6f5231ee8c4ec38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61ba8d7df6fd45abb6f5231ee8c4ec38 already in progress: creating tablet
I20251027 21:07:59.496747 10529 raft_consensus.cc:515] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.497488 10529 leader_election.cc:290] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:07:59.497839 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 0820648719a34f379507efced9cc434b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:59.498764 9568 leader_election.cc:343] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 8fc2d7d6601d4cc2849064f4cdddc90d
I20251027 21:07:59.499948 9742 tablet_service.cc:1505] Processing CreateTablet for tablet c5bc4ee5ec5f4a039356ac8841f5b836 (DEFAULT_TABLE table=default.Table_f4c9a8d0ca5f4277b9ff989e1b832985 [id=80426e5961ba4c5cb081587277d89891]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.504118 9308 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 7f7ad06a15c04efbb701634e14904351 (table default.Table_318d41ccfb2c4d2c99bafcea875143a1 [id=2bc39b5bc02645ef830eee14e92707db]). Expected version 1 got 0
I20251027 21:07:59.507093 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_918531cae34b4834ba3017dacf1181d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:59.509296 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 113a5d30d4b3490187681383d4fef325
I20251027 21:07:59.509665 10031 tablet_bootstrap.cc:654] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.510461 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet 113a5d30d4b3490187681383d4fef325 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:07:59.510607 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_918531cae34b4834ba3017dacf1181d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.512177 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 113a5d30d4b3490187681383d4fef325 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:59.513142 9600 tablet_service.cc:1505] Processing CreateTablet for tablet c5bc4ee5ec5f4a039356ac8841f5b836 (DEFAULT_TABLE table=default.Table_f4c9a8d0ca5f4277b9ff989e1b832985 [id=80426e5961ba4c5cb081587277d89891]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.513684 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5bc4ee5ec5f4a039356ac8841f5b836. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.514372 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 113a5d30d4b3490187681383d4fef325 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:59.516245 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet d2203ee62f7e404c95c5f86b5ae85feb
I20251027 21:07:59.516870 9311 catalog_manager.cc:6719] ReplaceTablet: tablet d2203ee62f7e404c95c5f86b5ae85feb deleted and replaced by tablet 00bfb0bdf2ca4b288eac0c1dbee5765a
I20251027 21:07:59.516777 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 458) took 4422 ms. Trace:
I20251027 21:07:59.517303 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8fc2d7d6601d4cc2849064f4cdddc90d" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:59.517832 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 849b3e9fc34045939a7a678fa8269d7f
I20251027 21:07:59.517896 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet d2203ee62f7e404c95c5f86b5ae85feb with delete_type TABLET_DATA_DELETED (replaced by tablet 00bfb0bdf2ca4b288eac0c1dbee5765a) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:59.517210 10040 rpcz_store.cc:276] 1027 21:07:55.094527 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:55.137912 (+ 43385us) service_pool.cc:225] Handling call
1027 21:07:59.491310 (+4353398us) ts_tablet_manager.cc:1151] Deleting tablet d872cf3ef39042ff9a615b55e7195afa
1027 21:07:59.516250 (+ 24940us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.516715 (+ 465us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4252044}
I20251027 21:07:59.518586 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e2ff7fc3c1fa4dbd9a5c38d3219b03fa
I20251027 21:07:59.519127 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 473) took 3456 ms. Trace:
I20251027 21:07:59.519342 10040 rpcz_store.cc:276] 1027 21:07:56.062696 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.063421 (+ 725us) service_pool.cc:225] Handling call
1027 21:07:59.518758 (+3455337us) ts_tablet_manager.cc:1151] Deleting tablet 4bbb69d62b9a41e3b88a5527f7549881
1027 21:07:59.518768 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.519042 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3455131}
W20251027 21:07:59.520249 9566 leader_election.cc:343] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.520263 9897 tablet_service.cc:1552] Processing DeleteTablet for tablet 849b3e9fc34045939a7a678fa8269d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:59.520471 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet e2ff7fc3c1fa4dbd9a5c38d3219b03fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:59.520654 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet d2203ee62f7e404c95c5f86b5ae85feb with delete_type TABLET_DATA_DELETED (replaced by tablet 00bfb0bdf2ca4b288eac0c1dbee5765a) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:59.520783 9566 leader_election.cc:304] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
W20251027 21:07:59.521036 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet d872cf3ef39042ff9a615b55e7195afa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d872cf3ef39042ff9a615b55e7195afa already in progress: creating tablet
W20251027 21:07:59.521281 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 4bbb69d62b9a41e3b88a5527f7549881 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4bbb69d62b9a41e3b88a5527f7549881 already in progress: creating tablet
I20251027 21:07:59.521636 9745 tablet_service.cc:1552] Processing DeleteTablet for tablet e2ff7fc3c1fa4dbd9a5c38d3219b03fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:59.521860 10529 raft_consensus.cc:2749] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:59.522118 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet 849b3e9fc34045939a7a678fa8269d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:59.522254 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet d2203ee62f7e404c95c5f86b5ae85feb with delete_type TABLET_DATA_DELETED (replaced by tablet 00bfb0bdf2ca4b288eac0c1dbee5765a) from {username='slave'} at 127.0.0.1:59712
W20251027 21:07:59.525027 9859 consensus_peers.cc:597] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:07:59.531788 10031 tablet_bootstrap.cc:492] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:59.532188 10031 ts_tablet_manager.cc:1403] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.058s user 0.005s sys 0.001s
I20251027 21:07:59.534307 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet 849b3e9fc34045939a7a678fa8269d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:59.534828 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet e2ff7fc3c1fa4dbd9a5c38d3219b03fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:59.535498 9895 tablet_service.cc:1505] Processing CreateTablet for tablet c5bc4ee5ec5f4a039356ac8841f5b836 (DEFAULT_TABLE table=default.Table_f4c9a8d0ca5f4277b9ff989e1b832985 [id=80426e5961ba4c5cb081587277d89891]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.535583 10031 raft_consensus.cc:359] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.535892 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5bc4ee5ec5f4a039356ac8841f5b836. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.535986 10031 raft_consensus.cc:385] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.536123 10031 raft_consensus.cc:740] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.536993 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 478) took 3473 ms. Trace:
I20251027 21:07:59.536821 10031 consensus_queue.cc:260] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.537735 10040 rpcz_store.cc:276] 1027 21:07:56.063182 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.067445 (+ 4263us) service_pool.cc:225] Handling call
1027 21:07:59.519998 (+3452553us) ts_tablet_manager.cc:1151] Deleting tablet ccaf5c2f8e544842880ba8ba27300ae6
1027 21:07:59.520045 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.536720 (+ 16675us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3452225}
W20251027 21:07:59.541136 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet ccaf5c2f8e544842880ba8ba27300ae6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ccaf5c2f8e544842880ba8ba27300ae6 already in progress: creating tablet
I20251027 21:07:59.543493 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 474) took 3480 ms. Trace:
I20251027 21:07:59.544807 10031 ts_tablet_manager.cc:1434] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.012s user 0.001s sys 0.002s
I20251027 21:07:59.546120 9739 tablet_service.cc:1505] Processing CreateTablet for tablet a3c0d0c5263140728f3b8f0b11f46aab (DEFAULT_TABLE table=default.Table_32eeb5fb70df48eb8c27a0a3bc95cee8 [id=3272f8f4808e4796afe01074ed0e955b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.544763 10040 rpcz_store.cc:276] 1027 21:07:56.062804 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.075922 (+ 13118us) service_pool.cc:225] Handling call
1027 21:07:59.542083 (+3466161us) ts_tablet_manager.cc:1151] Deleting tablet 28a2e85021394f9d8a9ea524959b52a1
1027 21:07:59.542094 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.543149 (+ 1055us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3465578}
W20251027 21:07:59.550064 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 28a2e85021394f9d8a9ea524959b52a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28a2e85021394f9d8a9ea524959b52a1 already in progress: creating tablet
I20251027 21:07:59.549892 10031 tablet_bootstrap.cc:492] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:59.552148 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 514) took 3138 ms. Trace:
I20251027 21:07:59.554359 10040 rpcz_store.cc:276] 1027 21:07:56.412813 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.539854 (+127041us) service_pool.cc:225] Handling call
1027 21:07:59.549736 (+3009882us) ts_tablet_manager.cc:1151] Deleting tablet 874d5974698c448ab79f19812cb0e886
1027 21:07:59.550266 (+ 530us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.551721 (+ 1455us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2970945}
I20251027 21:07:59.557582 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23f10b4258c8416cb5eba963f2ca4452. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.558298 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5bc4ee5ec5f4a039356ac8841f5b836. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.559293 9896 tablet_service.cc:1505] Processing CreateTablet for tablet a3c0d0c5263140728f3b8f0b11f46aab (DEFAULT_TABLE table=default.Table_32eeb5fb70df48eb8c27a0a3bc95cee8 [id=3272f8f4808e4796afe01074ed0e955b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.559666 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3c0d0c5263140728f3b8f0b11f46aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.566823 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d08b3bf501a441b5b7bf6a273f5e4e37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.567292 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 211ac01a37f8434f95aa6a63970296fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.568543 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3c0d0c5263140728f3b8f0b11f46aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.568987 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 6fe4618b580745428d8809010d4e64fe (DEFAULT_TABLE table=default.Table_93bb0a51ff97439e8ebcb08268bd2462 [id=4ca37f21cc99498dbdf575a68b84741b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.569365 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fe4618b580745428d8809010d4e64fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.571381 9601 tablet_service.cc:1505] Processing CreateTablet for tablet a3c0d0c5263140728f3b8f0b11f46aab (DEFAULT_TABLE table=default.Table_32eeb5fb70df48eb8c27a0a3bc95cee8 [id=3272f8f4808e4796afe01074ed0e955b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.571830 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3c0d0c5263140728f3b8f0b11f46aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.572338 10041 ts_tablet_manager.cc:1929] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.586237 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 283b72f626644ac2b9b61186387b3839 (DEFAULT_TABLE table=default.Table_5dc88891b75b4f1ca7b3ffba0d2b1fda [id=b46f1e4f1e6d4bd8b4a3edd1d6787612]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.586930 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 283b72f626644ac2b9b61186387b3839. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:59.589862 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 874d5974698c448ab79f19812cb0e886 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 874d5974698c448ab79f19812cb0e886 already in progress: creating tablet
I20251027 21:07:59.591055 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c239216cb8846f0b1aaa360ddb83321 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:59.598181 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c51243cadc5c403d84acc583c03edaba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:59.598417 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ec17a31b5e2948e2b6e7f1ffee9901c7" } new_table_name: "default.Table_3478938a9d2b44e7875de2ae7a4d471b" modify_external_catalogs: true
I20251027 21:07:59.598780 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_ab6ea9b86ffd4160a20af2687c8ef0ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.599351 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab6ea9b86ffd4160a20af2687c8ef0ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.599692 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_537e495061924a4da35379e43642675a" } new_table_name: "default.Table_3ef4dcb270d64cbbb4ad719b91ab1f10" modify_external_catalogs: true
I20251027 21:07:59.600590 9304 master_service.cc:954] ReplaceTablet: received request to replace tablet e477ca4b69c84327b04618d90406e19a from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:59.601723 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_ef8a1ea1e15249948b532d0bd39d5bc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.602252 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef8a1ea1e15249948b532d0bd39d5bc8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.578155 10032 tablet_bootstrap.cc:492] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:59.579026 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 6fe4618b580745428d8809010d4e64fe (DEFAULT_TABLE table=default.Table_93bb0a51ff97439e8ebcb08268bd2462 [id=4ca37f21cc99498dbdf575a68b84741b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.603075 10032 ts_tablet_manager.cc:1403] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.120s user 0.027s sys 0.029s
I20251027 21:07:59.603192 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fe4618b580745428d8809010d4e64fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.603876 10032 raft_consensus.cc:359] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:07:59.598786 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c51243cadc5c403d84acc583c03edaba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.604051 10032 raft_consensus.cc:385] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.604102 10032 raft_consensus.cc:740] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.604327 10032 consensus_queue.cc:260] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:59.605405 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b98328c296ab4b6ab486c3477c221544"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.605917 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b98328c296ab4b6ab486c3477c221544 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.608031 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 6fe4618b580745428d8809010d4e64fe (DEFAULT_TABLE table=default.Table_93bb0a51ff97439e8ebcb08268bd2462 [id=4ca37f21cc99498dbdf575a68b84741b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.608467 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fe4618b580745428d8809010d4e64fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.611011 10029 tablet_bootstrap.cc:492] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:59.611196 10029 ts_tablet_manager.cc:1403] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.168s user 0.004s sys 0.007s
I20251027 21:07:59.612507 10041 log.cc:1199] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/288b88e13fdf4427aae53f64b87733bf
I20251027 21:07:59.612985 10041 ts_tablet_manager.cc:1950] T 288b88e13fdf4427aae53f64b87733bf P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:59.613494 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 516) took 3178 ms. Trace:
I20251027 21:07:59.613600 10040 rpcz_store.cc:276] 1027 21:07:56.412986 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.588088 (+175102us) service_pool.cc:225] Handling call
1027 21:07:59.589440 (+3001352us) ts_tablet_manager.cc:1151] Deleting tablet 479d2cc8b8ce4c26841f50eea7c0e09a
1027 21:07:59.589450 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.591589 (+ 2139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2973184}
W20251027 21:07:59.614029 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 479d2cc8b8ce4c26841f50eea7c0e09a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 479d2cc8b8ce4c26841f50eea7c0e09a already in progress: creating tablet
I20251027 21:07:59.614154 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 288b88e13fdf4427aae53f64b87733bf (table default.Table_a061b11818fc4a2db61e51a5972d69db [id=92ff9768853d403fb15db41e79fe77bc]) successfully deleted
I20251027 21:07:59.614388 10041 tablet_replica.cc:333] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:59.614590 10041 raft_consensus.cc:2243] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.614745 10041 raft_consensus.cc:2272] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.615252 10041 ts_tablet_manager.cc:1916] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.614658 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 520) took 3201 ms. Trace:
I20251027 21:07:59.615478 10040 rpcz_store.cc:276] 1027 21:07:56.413323 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.616341 (+203018us) service_pool.cc:225] Handling call
1027 21:07:59.614489 (+2998148us) ts_tablet_manager.cc:1151] Deleting tablet 631ff72613734913818a3cad24f20dbc
1027 21:07:59.614518 (+ 29us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.614639 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2971778}
W20251027 21:07:59.616137 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 631ff72613734913818a3cad24f20dbc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 631ff72613734913818a3cad24f20dbc already in progress: creating tablet
I20251027 21:07:59.616530 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 521) took 3203 ms. Trace:
I20251027 21:07:59.617023 10040 rpcz_store.cc:276] 1027 21:07:56.413393 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.642198 (+228805us) service_pool.cc:225] Handling call
1027 21:07:59.616409 (+2974211us) ts_tablet_manager.cc:1151] Deleting tablet 37ae3b6bf590473e8e09092fc5f7df1b
1027 21:07:59.616417 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.616513 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2952069}
I20251027 21:07:59.617491 10032 ts_tablet_manager.cc:1434] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
W20251027 21:07:59.617995 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 37ae3b6bf590473e8e09092fc5f7df1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37ae3b6bf590473e8e09092fc5f7df1b already in progress: creating tablet
I20251027 21:07:59.616966 10031 tablet_bootstrap.cc:654] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.620208 9890 tablet_service.cc:1505] Processing CreateTablet for tablet db2735604dc14039810df4ecd330c671 (DEFAULT_TABLE table=default.Table_a6cf2819ed9241cea09738477423c84a [id=f5ef094809e24355a7e021408e2ddd20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.620563 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db2735604dc14039810df4ecd330c671. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.611992 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 283b72f626644ac2b9b61186387b3839 (DEFAULT_TABLE table=default.Table_5dc88891b75b4f1ca7b3ffba0d2b1fda [id=b46f1e4f1e6d4bd8b4a3edd1d6787612]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.621645 9594 tablet_service.cc:1505] Processing CreateTablet for tablet db2735604dc14039810df4ecd330c671 (DEFAULT_TABLE table=default.Table_a6cf2819ed9241cea09738477423c84a [id=f5ef094809e24355a7e021408e2ddd20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.622031 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 0f127882b9d346ba99d767dbb591d0be (DEFAULT_TABLE table=default.Table_5a9b8f6ad9a64835ab66cfce7bcd144e [id=8e7b5d0535094704817974d65722e7e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.622324 10029 raft_consensus.cc:359] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.622629 10029 raft_consensus.cc:385] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.622864 10029 raft_consensus.cc:740] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.623294 10029 consensus_queue.cc:260] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.626157 9748 tablet_service.cc:1505] Processing CreateTablet for tablet db2735604dc14039810df4ecd330c671 (DEFAULT_TABLE table=default.Table_a6cf2819ed9241cea09738477423c84a [id=f5ef094809e24355a7e021408e2ddd20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.627066 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 0f127882b9d346ba99d767dbb591d0be (DEFAULT_TABLE table=default.Table_5a9b8f6ad9a64835ab66cfce7bcd144e [id=8e7b5d0535094704817974d65722e7e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.627408 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 283b72f626644ac2b9b61186387b3839. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.627677 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 0f127882b9d346ba99d767dbb591d0be (DEFAULT_TABLE table=default.Table_5a9b8f6ad9a64835ab66cfce7bcd144e [id=8e7b5d0535094704817974d65722e7e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.627754 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db2735604dc14039810df4ecd330c671. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.627998 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f127882b9d346ba99d767dbb591d0be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.628619 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f127882b9d346ba99d767dbb591d0be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.598176 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 283b72f626644ac2b9b61186387b3839 (DEFAULT_TABLE table=default.Table_5dc88891b75b4f1ca7b3ffba0d2b1fda [id=b46f1e4f1e6d4bd8b4a3edd1d6787612]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.630414 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 283b72f626644ac2b9b61186387b3839. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.630376 10560 raft_consensus.cc:493] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.630510 10560 raft_consensus.cc:515] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.631040 10560 leader_election.cc:290] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:59.631749 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d52d71f4976143e4a514350621ebe309" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:59.632223 9860 leader_election.cc:343] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.636930 10529 raft_consensus.cc:493] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.637084 10529 raft_consensus.cc:515] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.637573 10529 leader_election.cc:290] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:07:59.638862 9568 leader_election.cc:343] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: ed5619d295a246379b625f35692203e4
I20251027 21:07:59.639024 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d52d71f4976143e4a514350621ebe309" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:59.639250 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db2735604dc14039810df4ecd330c671. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.639060 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed5619d295a246379b625f35692203e4" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:59.639457 9859 leader_election.cc:343] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.639561 9859 leader_election.cc:304] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:59.639930 10560 raft_consensus.cc:2749] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:59.640985 10040 tablet_replica.cc:333] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:59.641167 10040 raft_consensus.cc:2243] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.641319 10040 raft_consensus.cc:2272] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.641347 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_cd4849448781434f836c682ba134ce00" } new_table_name: "default.Table_e53258cb321b4ae099133b8176b9c0a1" modify_external_catalogs: true
I20251027 21:07:59.641825 10040 ts_tablet_manager.cc:1916] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:59.642275 9566 leader_election.cc:343] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.642397 9566 leader_election.cc:304] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:59.642921 10529 raft_consensus.cc:2749] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:59.646576 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a58cd07e1bb4ffdbf39e7e16a46022a with delete_type TABLET_DATA_DELETED (replaced by tablet 4d88aacbf47144cb837b99d68f6331e2) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:59.646782 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f127882b9d346ba99d767dbb591d0be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.647701 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_190963122a7245dfa310ab8c69edf80f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.648265 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_190963122a7245dfa310ab8c69edf80f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.649864 10032 tablet_bootstrap.cc:492] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:59.650239 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_011a37833a9142ad95940903f3185dfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.650767 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_011a37833a9142ad95940903f3185dfb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.651499 10032 tablet_bootstrap.cc:654] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.651438 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 6cd39721fd28422896d517d349009c66 (DEFAULT_TABLE table=default.Table_537e495061924a4da35379e43642675a [id=3b80e606aa374fd9ad1e7b0c5c20bb6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.651816 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cd39721fd28422896d517d349009c66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.651899 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b462e4bf95dc4fc4ad3fd23b236fffa7" } modify_external_catalogs: true
I20251027 21:07:59.652051 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b462e4bf95dc4fc4ad3fd23b236fffa7" } modify_external_catalogs: true
I20251027 21:07:59.657987 9609 tablet_service.cc:1505] Processing CreateTablet for tablet d27f9c498eca4e5f8d2bfe9c805d06b3 (DEFAULT_TABLE table=default.Table_17c224e94fc94d99b0d23c088cef1820 [id=e649157c74aa4e18b4f3b9bf810fff2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.658386 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d27f9c498eca4e5f8d2bfe9c805d06b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.663039 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 050e9442041d495496bfa7ce25ac6a88 (DEFAULT_TABLE table=default.Table_bae8db615e424af398f1f9baec11cb76 [id=cc1da530aa95455e90889dd5ea3768c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.663429 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 050e9442041d495496bfa7ce25ac6a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.666450 9739 tablet_service.cc:1505] Processing CreateTablet for tablet d27f9c498eca4e5f8d2bfe9c805d06b3 (DEFAULT_TABLE table=default.Table_17c224e94fc94d99b0d23c088cef1820 [id=e649157c74aa4e18b4f3b9bf810fff2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.666774 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d27f9c498eca4e5f8d2bfe9c805d06b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.668133 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 050e9442041d495496bfa7ce25ac6a88 (DEFAULT_TABLE table=default.Table_bae8db615e424af398f1f9baec11cb76 [id=cc1da530aa95455e90889dd5ea3768c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.668557 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 050e9442041d495496bfa7ce25ac6a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.670715 9611 tablet_service.cc:1505] Processing CreateTablet for tablet d7955e31f10449718964a8ced3c18672 (DEFAULT_TABLE table=default.Table_5cd332429fd147e1b99e2fed13e0a0e7 [id=593dff7ab03d4186b1e58a46ff1db6fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.671113 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7955e31f10449718964a8ced3c18672. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.673130 9744 tablet_service.cc:1505] Processing CreateTablet for tablet d7955e31f10449718964a8ced3c18672 (DEFAULT_TABLE table=default.Table_5cd332429fd147e1b99e2fed13e0a0e7 [id=593dff7ab03d4186b1e58a46ff1db6fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.673746 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7955e31f10449718964a8ced3c18672. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.681520 10665 raft_consensus.cc:493] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.681645 10665 raft_consensus.cc:515] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.682353 10665 leader_election.cc:290] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
W20251027 21:07:59.682961 9711 leader_election.cc:343] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: ae124514964440c5b88b16ac9f8a2567
W20251027 21:07:59.683606 9710 leader_election.cc:343] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: ae124514964440c5b88b16ac9f8a2567
I20251027 21:07:59.683703 9710 leader_election.cc:304] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:59.683918 10665 raft_consensus.cc:2749] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:59.684623 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 6cd39721fd28422896d517d349009c66 (DEFAULT_TABLE table=default.Table_537e495061924a4da35379e43642675a [id=3b80e606aa374fd9ad1e7b0c5c20bb6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.684957 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cd39721fd28422896d517d349009c66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.662127 10041 ts_tablet_manager.cc:1929] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251027 21:07:59.667991 9859 consensus_peers.cc:597] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:07:59.689054 10041 log.cc:1199] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/b719dc40d7e348608cdfc78401c19824
I20251027 21:07:59.689383 10041 ts_tablet_manager.cc:1950] T b719dc40d7e348608cdfc78401c19824 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:59.691219 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet b719dc40d7e348608cdfc78401c19824 (table default.Table_41078339d0834d069db2e8ae485bcc89 [id=f641ffdf5fd7497096643ea3278c49d0]) successfully deleted
W20251027 21:07:59.692703 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 196a594025ad4564b028b39f1afae48f has no consensus state
I20251027 21:07:59.693056 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d0d99e11752e4faba5d28e5f7211c121" } new_table_name: "default.Table_fbcff2da42d54b38a090a042559bad38" modify_external_catalogs: true
I20251027 21:07:59.694465 10040 ts_tablet_manager.cc:1929] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.694555 10040 log.cc:1199] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/cfce0e5c837743ea969374244163be3e
I20251027 21:07:59.694861 10040 ts_tablet_manager.cc:1950] T cfce0e5c837743ea969374244163be3e P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:59.694891 10628 raft_consensus.cc:493] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.695022 10628 raft_consensus.cc:515] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.695169 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 526) took 3218 ms. Trace:
I20251027 21:07:59.695238 10040 rpcz_store.cc:276] 1027 21:07:56.476631 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.706541 (+229910us) service_pool.cc:225] Handling call
1027 21:07:59.618344 (+2911803us) ts_tablet_manager.cc:1151] Deleting tablet cfce0e5c837743ea969374244163be3e
1027 21:07:59.618353 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.667372 (+ 49019us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.694388 (+ 27016us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.695153 (+ 765us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2911593}
I20251027 21:07:59.695647 10628 leader_election.cc:290] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:59.695808 10628 consensus_queue.cc:1048] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:07:59.696154 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet cfce0e5c837743ea969374244163be3e (table default.Table_4568794575f345c08a45afeb2d4f9648 [id=13a25b18846c4580a337b91cee116a3d]) successfully deleted
I20251027 21:07:59.668394 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 050e9442041d495496bfa7ce25ac6a88 (DEFAULT_TABLE table=default.Table_bae8db615e424af398f1f9baec11cb76 [id=cc1da530aa95455e90889dd5ea3768c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.697965 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 050e9442041d495496bfa7ce25ac6a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.698426 10041 tablet_replica.cc:333] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:59.698681 10041 raft_consensus.cc:2243] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.698891 10041 raft_consensus.cc:2272] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.699572 10041 ts_tablet_manager.cc:1916] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.700351 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7c426eeeaca14f65ab36fef3162d4a3a" } new_table_name: "default.Table_c97bb7def2964a3aacb4b0627420bf5a" modify_external_catalogs: true
W20251027 21:07:59.700532 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6cd39721fd28422896d517d349009c66 consensus state has no leader
I20251027 21:07:59.700862 9308 master_service.cc:954] ReplaceTablet: received request to replace tablet f1aa5f7f76cb4a1ba15f016942f17067 from {username='slave'} at 127.0.0.1:41710
I20251027 21:07:59.671294 9897 tablet_service.cc:1505] Processing CreateTablet for tablet d7955e31f10449718964a8ced3c18672 (DEFAULT_TABLE table=default.Table_5cd332429fd147e1b99e2fed13e0a0e7 [id=593dff7ab03d4186b1e58a46ff1db6fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.701702 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7955e31f10449718964a8ced3c18672. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:59.702394 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 217fbfb1f58f420f8f258b4178901370 consensus state has no leader
I20251027 21:07:59.706302 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abc7458fcfb94985acf76d77c0292b08" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:59.672331 10560 raft_consensus.cc:493] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.706995 10560 raft_consensus.cc:515] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.707719 10560 leader_election.cc:290] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:59.708432 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c4996d8fae094db8a96599b9057f1649
I20251027 21:07:59.709781 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet c4996d8fae094db8a96599b9057f1649 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:59.709951 9896 tablet_service.cc:1552] Processing DeleteTablet for tablet c4996d8fae094db8a96599b9057f1649 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:07:59.710310 9859 leader_election.cc:343] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.710601 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f34512059f0c47fc9af7db2645f86b40" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:59.711359 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 523) took 3297 ms. Trace:
W20251027 21:07:59.711457 9859 leader_election.cc:343] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.711438 10040 rpcz_store.cc:276] 1027 21:07:56.476332 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.654651 (+178319us) service_pool.cc:225] Handling call
1027 21:07:59.711241 (+3056590us) ts_tablet_manager.cc:1151] Deleting tablet 92a2393cff8044028c86c8cd2bccd930
1027 21:07:59.711251 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.711342 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2983572}
W20251027 21:07:59.711822 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 92a2393cff8044028c86c8cd2bccd930 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92a2393cff8044028c86c8cd2bccd930 already in progress: creating tablet
I20251027 21:07:59.711915 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 537) took 3097 ms. Trace:
I20251027 21:07:59.711973 10040 rpcz_store.cc:276] 1027 21:07:56.614926 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.767834 (+152908us) service_pool.cc:225] Handling call
1027 21:07:59.711799 (+2943965us) ts_tablet_manager.cc:1151] Deleting tablet 95f9595784df434dbb0dd29d9a05511e
1027 21:07:59.711810 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.711899 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2943792}
W20251027 21:07:59.712320 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 95f9595784df434dbb0dd29d9a05511e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95f9595784df434dbb0dd29d9a05511e already in progress: creating tablet
I20251027 21:07:59.712388 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 538) took 3075 ms. Trace:
I20251027 21:07:59.712442 10040 rpcz_store.cc:276] 1027 21:07:56.636457 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.768038 (+131581us) service_pool.cc:225] Handling call
1027 21:07:59.712290 (+2944252us) ts_tablet_manager.cc:1151] Deleting tablet 11627a84ff9b478c87c323617a09570b
1027 21:07:59.712297 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.712373 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2936972}
W20251027 21:07:59.712774 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 11627a84ff9b478c87c323617a09570b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11627a84ff9b478c87c323617a09570b already in progress: creating tablet
I20251027 21:07:59.712877 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 542) took 3044 ms. Trace:
I20251027 21:07:59.712937 10040 rpcz_store.cc:276] 1027 21:07:56.668454 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.975323 (+306869us) service_pool.cc:225] Handling call
1027 21:07:59.712757 (+2737434us) ts_tablet_manager.cc:1151] Deleting tablet 5054bbc3e27243298b4df6b0a87af8be
1027 21:07:59.712767 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.712863 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2737221}
W20251027 21:07:59.713310 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 5054bbc3e27243298b4df6b0a87af8be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5054bbc3e27243298b4df6b0a87af8be already in progress: creating tablet
I20251027 21:07:59.713768 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 544) took 2991 ms. Trace:
I20251027 21:07:59.713857 10040 rpcz_store.cc:276] 1027 21:07:56.721881 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.975551 (+253670us) service_pool.cc:225] Handling call
1027 21:07:59.713680 (+2738129us) ts_tablet_manager.cc:1151] Deleting tablet c8f0cd910b9d4daea287b3b192a81bd2
1027 21:07:59.713688 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.713752 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2737930}
I20251027 21:07:59.714012 10665 raft_consensus.cc:493] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.714164 10665 raft_consensus.cc:515] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.714711 10665 leader_election.cc:290] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
W20251027 21:07:59.715233 9711 leader_election.cc:343] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: cb1686fc7c3a4996aa4462884d30c41f
W20251027 21:07:59.715664 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c8f0cd910b9d4daea287b3b192a81bd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8f0cd910b9d4daea287b3b192a81bd2 already in progress: creating tablet
W20251027 21:07:59.716032 9710 leader_election.cc:343] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: cb1686fc7c3a4996aa4462884d30c41f
I20251027 21:07:59.716145 9710 leader_election.cc:304] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:59.716339 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e6e6313ef68948c79fc2f7d96afae64b" } modify_external_catalogs: true
I20251027 21:07:59.716477 10665 raft_consensus.cc:2749] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:59.716498 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e6e6313ef68948c79fc2f7d96afae64b" } modify_external_catalogs: true
I20251027 21:07:59.674542 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 6cd39721fd28422896d517d349009c66 (DEFAULT_TABLE table=default.Table_537e495061924a4da35379e43642675a [id=3b80e606aa374fd9ad1e7b0c5c20bb6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.682476 9895 tablet_service.cc:1505] Processing CreateTablet for tablet d27f9c498eca4e5f8d2bfe9c805d06b3 (DEFAULT_TABLE table=default.Table_17c224e94fc94d99b0d23c088cef1820 [id=e649157c74aa4e18b4f3b9bf810fff2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.724395 10029 ts_tablet_manager.cc:1434] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.113s user 0.001s sys 0.001s
I20251027 21:07:59.728552 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet c4996d8fae094db8a96599b9057f1649 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:59.729281 10029 tablet_bootstrap.cc:492] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:59.731561 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 545) took 2998 ms. Trace:
I20251027 21:07:59.731669 10040 rpcz_store.cc:276] 1027 21:07:56.739014 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.975763 (+236749us) service_pool.cc:225] Handling call
1027 21:07:59.731409 (+2755646us) ts_tablet_manager.cc:1151] Deleting tablet e6ed9d7a700446edafcd0daa02aa7ef6
1027 21:07:59.731420 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.731522 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2755494}
I20251027 21:07:59.731823 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cd39721fd28422896d517d349009c66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.732163 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 546) took 2993 ms. Trace:
I20251027 21:07:59.732259 10040 rpcz_store.cc:276] 1027 21:07:56.739175 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.975927 (+236752us) service_pool.cc:225] Handling call
1027 21:07:59.732060 (+2756133us) ts_tablet_manager.cc:1151] Deleting tablet c3b9a461567a48fcb4af514ecdf618b4
1027 21:07:59.732069 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.732147 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2755960}
W20251027 21:07:59.732460 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet e6ed9d7a700446edafcd0daa02aa7ef6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6ed9d7a700446edafcd0daa02aa7ef6 already in progress: creating tablet
I20251027 21:07:59.732663 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 547) took 2993 ms. Trace:
W20251027 21:07:59.732722 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c3b9a461567a48fcb4af514ecdf618b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3b9a461567a48fcb4af514ecdf618b4 already in progress: creating tablet
I20251027 21:07:59.732735 10040 rpcz_store.cc:276] 1027 21:07:56.739265 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.976115 (+236850us) service_pool.cc:225] Handling call
1027 21:07:59.732576 (+2756461us) ts_tablet_manager.cc:1151] Deleting tablet c9efffa9766e46aca8eb286fc2ad48d2
1027 21:07:59.732584 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.732649 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2756306}
W20251027 21:07:59.733131 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c9efffa9766e46aca8eb286fc2ad48d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9efffa9766e46aca8eb286fc2ad48d2 already in progress: creating tablet
I20251027 21:07:59.733271 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 548) took 2993 ms. Trace:
I20251027 21:07:59.733350 10040 rpcz_store.cc:276] 1027 21:07:56.739342 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.976286 (+236944us) service_pool.cc:225] Handling call
1027 21:07:59.733180 (+2756894us) ts_tablet_manager.cc:1151] Deleting tablet aabaa34787cf4d179822269b7f323314
1027 21:07:59.733190 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.733257 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2756737}
I20251027 21:07:59.734587 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 549) took 2995 ms. Trace:
I20251027 21:07:59.734649 10040 rpcz_store.cc:276] 1027 21:07:56.739437 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.976458 (+237021us) service_pool.cc:225] Handling call
1027 21:07:59.734503 (+2758045us) ts_tablet_manager.cc:1151] Deleting tablet e10ae34eaaf3476594813de8e96c7e77
1027 21:07:59.734511 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.734573 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2757910}
I20251027 21:07:59.735306 10029 tablet_bootstrap.cc:654] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.735698 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 550) took 2996 ms. Trace:
I20251027 21:07:59.735761 10040 rpcz_store.cc:276] 1027 21:07:56.739558 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.976601 (+237043us) service_pool.cc:225] Handling call
1027 21:07:59.735609 (+2759008us) ts_tablet_manager.cc:1151] Deleting tablet 371aa8a57aee466ab01652d6061db4c2
1027 21:07:59.735617 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.735683 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2758789}
I20251027 21:07:59.736171 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abc7458fcfb94985acf76d77c0292b08" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:59.736907 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f34512059f0c47fc9af7db2645f86b40" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:07:59.737223 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet aabaa34787cf4d179822269b7f323314 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aabaa34787cf4d179822269b7f323314 already in progress: creating tablet
W20251027 21:07:59.737419 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet e10ae34eaaf3476594813de8e96c7e77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e10ae34eaaf3476594813de8e96c7e77 already in progress: creating tablet
W20251027 21:07:59.737625 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 371aa8a57aee466ab01652d6061db4c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 371aa8a57aee466ab01652d6061db4c2 already in progress: creating tablet
I20251027 21:07:59.741312 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d27f9c498eca4e5f8d2bfe9c805d06b3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:59.742483 9860 leader_election.cc:343] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.742930 9860 leader_election.cc:304] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:59.743489 10628 raft_consensus.cc:2749] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:07:59.743901 9860 leader_election.cc:343] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.744415 9860 leader_election.cc:304] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:07:59.744947 10628 raft_consensus.cc:2749] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:59.746393 10041 ts_tablet_manager.cc:1929] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.746556 10041 log.cc:1199] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/fe24af9838954a9bb680474800642e62
W20251027 21:07:59.746649 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3ee02a05b5ae46e7bf5cf9fdd3e9a80f consensus state has no leader
I20251027 21:07:59.746907 10041 ts_tablet_manager.cc:1950] T fe24af9838954a9bb680474800642e62 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:59.751711 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 551) took 3012 ms. Trace:
W20251027 21:07:59.751852 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7c15b74f53bc45e1af33b043f2cf225f consensus state has no leader
I20251027 21:07:59.751824 10040 rpcz_store.cc:276] 1027 21:07:56.739641 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.976827 (+237186us) service_pool.cc:225] Handling call
1027 21:07:59.751162 (+2774335us) ts_tablet_manager.cc:1151] Deleting tablet 4d2c40f983e743b0b53f987b6a7ba437
1027 21:07:59.751172 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.751681 (+ 509us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2774175}
I20251027 21:07:59.752034 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 83a040332f994055a9ecf4c66720b7e1
W20251027 21:07:59.752959 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 4d2c40f983e743b0b53f987b6a7ba437 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d2c40f983e743b0b53f987b6a7ba437 already in progress: creating tablet
I20251027 21:07:59.753149 9745 tablet_service.cc:1552] Processing DeleteTablet for tablet 83a040332f994055a9ecf4c66720b7e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:59.754575 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 552) took 3014 ms. Trace:
I20251027 21:07:59.755149 10040 rpcz_store.cc:276] 1027 21:07:56.739721 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.976998 (+237277us) service_pool.cc:225] Handling call
1027 21:07:59.753695 (+2776697us) ts_tablet_manager.cc:1151] Deleting tablet 618f1657a10947fd98976820564c20b3
1027 21:07:59.753707 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.754553 (+ 846us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2776538}
W20251027 21:07:59.756145 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 618f1657a10947fd98976820564c20b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 618f1657a10947fd98976820564c20b3 already in progress: creating tablet
I20251027 21:07:59.756294 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 543) took 3045 ms. Trace:
I20251027 21:07:59.756382 10040 rpcz_store.cc:276] 1027 21:07:56.711264 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.978244 (+266980us) service_pool.cc:225] Handling call
1027 21:07:59.756175 (+2777931us) ts_tablet_manager.cc:1151] Deleting tablet bc9322398d024e629a3c2aae7b6a77be
1027 21:07:59.756186 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.756275 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2777710}
W20251027 21:07:59.756745 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet bc9322398d024e629a3c2aae7b6a77be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc9322398d024e629a3c2aae7b6a77be already in progress: creating tablet
I20251027 21:07:59.757331 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 83a040332f994055a9ecf4c66720b7e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:07:59.757354 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 83a040332f994055a9ecf4c66720b7e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:59.757776 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 554) took 3017 ms. Trace:
I20251027 21:07:59.757900 10040 rpcz_store.cc:276] 1027 21:07:56.740763 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.978479 (+237716us) service_pool.cc:225] Handling call
1027 21:07:59.757689 (+2779210us) ts_tablet_manager.cc:1151] Deleting tablet 64e39cc210664c3d83eaf1fc576f83ce
1027 21:07:59.757697 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.757762 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2779040}
W20251027 21:07:59.758239 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 64e39cc210664c3d83eaf1fc576f83ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64e39cc210664c3d83eaf1fc576f83ce already in progress: creating tablet
I20251027 21:07:59.758394 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet fe24af9838954a9bb680474800642e62 (table default.Table_fc695aa989384c1ba355a09381428926 [id=dae1114f6b50443f8f6b79d18619d868]) successfully deleted
W20251027 21:07:59.759109 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 38f2133010b14aad88b550fb12c966a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38f2133010b14aad88b550fb12c966a5 already in progress: creating tablet
I20251027 21:07:59.759167 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 555) took 3018 ms. Trace:
I20251027 21:07:59.759225 10040 rpcz_store.cc:276] 1027 21:07:56.740872 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.978740 (+237868us) service_pool.cc:225] Handling call
1027 21:07:59.759073 (+2780333us) ts_tablet_manager.cc:1151] Deleting tablet ed5619d295a246379b625f35692203e4
1027 21:07:59.759081 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.759153 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2780133}
W20251027 21:07:59.759562 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet ed5619d295a246379b625f35692203e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed5619d295a246379b625f35692203e4 already in progress: creating tablet
W20251027 21:07:59.759840 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 4d88aacbf47144cb837b99d68f6331e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d88aacbf47144cb837b99d68f6331e2 already in progress: creating tablet
I20251027 21:07:59.760385 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 562) took 2990 ms. Trace:
W20251027 21:07:59.760484 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 6210d3d10fc44b9e9e2b0780883186ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6210d3d10fc44b9e9e2b0780883186ce already in progress: creating tablet
I20251027 21:07:59.760458 10040 rpcz_store.cc:276] 1027 21:07:56.769973 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.998362 (+228389us) service_pool.cc:225] Handling call
1027 21:07:59.760288 (+2761926us) ts_tablet_manager.cc:1151] Deleting tablet 0e4d35f229bc46e1859ded7c8196ab9d
1027 21:07:59.760297 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.760369 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2761718}
W20251027 21:07:59.760787 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 0e4d35f229bc46e1859ded7c8196ab9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e4d35f229bc46e1859ded7c8196ab9d already in progress: creating tablet
W20251027 21:07:59.761185 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 8ae222307a374a91916f560a0fbe8d8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ae222307a374a91916f560a0fbe8d8f already in progress: creating tablet
I20251027 21:07:59.761618 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 563) took 2991 ms. Trace:
I20251027 21:07:59.761693 10040 rpcz_store.cc:276] 1027 21:07:56.770082 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.998583 (+228501us) service_pool.cc:225] Handling call
1027 21:07:59.761534 (+2762951us) ts_tablet_manager.cc:1151] Deleting tablet 9351a369dc1e4ac49f6ad419eab3a4c5
1027 21:07:59.761542 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.761604 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2762781}
W20251027 21:07:59.762027 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 9351a369dc1e4ac49f6ad419eab3a4c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9351a369dc1e4ac49f6ad419eab3a4c5 already in progress: creating tablet
W20251027 21:07:59.762596 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet a8328318a4f14b38a4ce36d183e99dfb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8328318a4f14b38a4ce36d183e99dfb already in progress: creating tablet
I20251027 21:07:59.764876 10041 tablet_replica.cc:333] T 343eaeca24dc4816ae032c7e674e9412 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:59.766482 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 564) took 2996 ms. Trace:
I20251027 21:07:59.766566 10040 rpcz_store.cc:276] 1027 21:07:56.770181 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.998768 (+228587us) service_pool.cc:225] Handling call
1027 21:07:59.766383 (+2767615us) ts_tablet_manager.cc:1151] Deleting tablet e477ca4b69c84327b04618d90406e19a
1027 21:07:59.766392 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.766466 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2767435}
I20251027 21:07:59.767177 10041 raft_consensus.cc:2243] T 343eaeca24dc4816ae032c7e674e9412 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.767431 10041 raft_consensus.cc:2272] T 343eaeca24dc4816ae032c7e674e9412 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.767992 10041 ts_tablet_manager.cc:1916] T 343eaeca24dc4816ae032c7e674e9412 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.769004 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 565) took 2998 ms. Trace:
I20251027 21:07:59.769089 10040 rpcz_store.cc:276] 1027 21:07:56.770281 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.998974 (+228693us) service_pool.cc:225] Handling call
1027 21:07:59.768576 (+2769602us) ts_tablet_manager.cc:1151] Deleting tablet bbf92e29993b46389a69efd9e9a2341f
1027 21:07:59.768585 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.768985 (+ 400us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2769454}
I20251027 21:07:59.770059 10040 tablet_replica.cc:333] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:59.770280 10040 raft_consensus.cc:2243] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251027 21:07:59.770372 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet e477ca4b69c84327b04618d90406e19a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e477ca4b69c84327b04618d90406e19a already in progress: creating tablet
I20251027 21:07:59.770462 10040 raft_consensus.cc:2272] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:07:59.770612 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet bbf92e29993b46389a69efd9e9a2341f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbf92e29993b46389a69efd9e9a2341f already in progress: creating tablet
I20251027 21:07:59.770999 10040 ts_tablet_manager.cc:1916] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.772439 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet e841c54a94d649c19b639f5021f150cd with delete_type TABLET_DATA_DELETED (replaced by tablet 666842b8e54c4561989fd9401210464b) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:59.772800 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 49d9e5c5960a4534a6603875006e6266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:59.780467 9924 raft_consensus.cc:1275] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Refusing update from remote peer 51902b6b61ff4d5584ff21896e847e14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 21:07:59.781179 10682 consensus_queue.cc:1048] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:07:59.781332 10043 ts_tablet_manager.cc:1929] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.781509 10043 log.cc:1199] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/6db0f31c12d14e66a4ec7fc722d62c3d
I20251027 21:07:59.781817 10560 raft_consensus.cc:493] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251027 21:07:59.782027 9566 consensus_peers.cc:597] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:07:59.781975 10560 raft_consensus.cc:515] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.782254 10043 ts_tablet_manager.cc:1950] T 6db0f31c12d14e66a4ec7fc722d62c3d P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:07:59.782634 10560 leader_election.cc:290] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:59.782724 10031 tablet_bootstrap.cc:492] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:59.782857 10031 ts_tablet_manager.cc:1403] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.235s user 0.001s sys 0.017s
I20251027 21:07:59.783299 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "132a01405a63447ab27b2137b803224f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:59.783771 9859 leader_election.cc:343] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.783713 10031 raft_consensus.cc:359] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.783919 10031 raft_consensus.cc:385] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.783993 10031 raft_consensus.cc:740] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.784231 10031 consensus_queue.cc:260] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.784665 10043 tablet_replica.cc:333] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:07:59.784718 10031 ts_tablet_manager.cc:1434] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 21:07:59.785086 10043 raft_consensus.cc:2243] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.785564 10043 raft_consensus.cc:2272] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.789541 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "132a01405a63447ab27b2137b803224f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:07:59.790158 9631 raft_consensus.cc:1275] T 995284883d834bbb9ef583b736ca8402 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Refusing update from remote peer 0fbd567f929a4f6d8e4d78feef7ed678: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 21:07:59.790628 10031 tablet_bootstrap.cc:492] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:59.792107 10031 tablet_bootstrap.cc:654] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.795605 10032 tablet_bootstrap.cc:492] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:59.795748 10029 tablet_bootstrap.cc:492] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:07:59.796005 10032 ts_tablet_manager.cc:1403] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.146s user 0.009s sys 0.008s
I20251027 21:07:59.796134 10029 ts_tablet_manager.cc:1403] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.067s user 0.000s sys 0.004s
I20251027 21:07:59.798332 10032 raft_consensus.cc:359] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:59.798574 10032 raft_consensus.cc:385] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.798575 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_92242f48713b48e5969d30e05bed6686" } modify_external_catalogs: true
I20251027 21:07:59.798715 10032 raft_consensus.cc:740] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.798764 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_92242f48713b48e5969d30e05bed6686" } modify_external_catalogs: true
I20251027 21:07:59.798928 10032 consensus_queue.cc:260] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:59.799423 10032 ts_tablet_manager.cc:1434] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20251027 21:07:59.799693 10032 tablet_bootstrap.cc:492] T 94f2dbf47a5c4458bee54b26f272156e P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:59.800618 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1a08b666f7f4426eb19757e15584dd53" } modify_external_catalogs: true
I20251027 21:07:59.800802 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1a08b666f7f4426eb19757e15584dd53" } modify_external_catalogs: true
I20251027 21:07:59.801239 10032 tablet_bootstrap.cc:654] T 94f2dbf47a5c4458bee54b26f272156e P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.803817 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a473b10ab1e94ca885768527b521172b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:59.804983 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_851f66cf4b1047879eed5fb768c07156" } modify_external_catalogs: true
I20251027 21:07:59.805176 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_851f66cf4b1047879eed5fb768c07156" } modify_external_catalogs: true
I20251027 21:07:59.806624 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_385f764be86a472d8360fd95634bdcde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:59.807950 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7fe12d37a07344b4b839c87a4b904c20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:59.809077 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0c098e93345a4423888bb615af233dbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:07:59.812062 10041 ts_tablet_manager.cc:1929] T 343eaeca24dc4816ae032c7e674e9412 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.783519 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 6db0f31c12d14e66a4ec7fc722d62c3d (table default.Table_42431224890f43859abe3dedfae4db51 [id=a6784a116865476b85fe6ff9c32bb9db]) successfully deleted
W20251027 21:07:59.812667 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet a4d26f73b8534f08bcaa386d4de75c01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4d26f73b8534f08bcaa386d4de75c01 already in progress: creating tablet
W20251027 21:07:59.814288 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c098e93345a4423888bb615af233dbe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.815409 10031 tablet_bootstrap.cc:492] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:59.815528 10031 ts_tablet_manager.cc:1403] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.001s
I20251027 21:07:59.815706 10041 log.cc:1199] T 343eaeca24dc4816ae032c7e674e9412 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/343eaeca24dc4816ae032c7e674e9412
I20251027 21:07:59.816087 10041 ts_tablet_manager.cc:1950] T 343eaeca24dc4816ae032c7e674e9412 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:59.816290 10031 raft_consensus.cc:359] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.816805 10031 raft_consensus.cc:385] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.816886 10031 raft_consensus.cc:740] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.817090 10031 consensus_queue.cc:260] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.817543 10031 ts_tablet_manager.cc:1434] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:07:59.817692 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 343eaeca24dc4816ae032c7e674e9412 (table default.Table_546a5293975148e1841c8fff9eeac42a [id=b4569b8a023945989439915acb6097e4]) successfully deleted
I20251027 21:07:59.797569 10029 raft_consensus.cc:359] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.818018 10031 tablet_bootstrap.cc:492] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:59.818027 10029 raft_consensus.cc:385] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.818110 10029 raft_consensus.cc:740] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
W20251027 21:07:59.818692 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fe12d37a07344b4b839c87a4b904c20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.819177 10029 consensus_queue.cc:260] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.817276 10041 tablet_replica.cc:333] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:59.819677 10041 raft_consensus.cc:2243] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.819721 10029 ts_tablet_manager.cc:1434] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.023s user 0.000s sys 0.001s
I20251027 21:07:59.819941 10041 raft_consensus.cc:2272] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:07:59.820127 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_385f764be86a472d8360fd95634bdcde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.820490 10041 ts_tablet_manager.cc:1916] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:07:59.821230 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a473b10ab1e94ca885768527b521172b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.828305 10031 tablet_bootstrap.cc:654] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.810096 10040 ts_tablet_manager.cc:1929] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.829627 10040 log.cc:1199] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/cb1686fc7c3a4996aa4462884d30c41f
I20251027 21:07:59.829974 10040 ts_tablet_manager.cc:1950] T cb1686fc7c3a4996aa4462884d30c41f P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:59.830910 9312 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 7aa8c538b6c849a28a28ffe706306891 (table default.Table_b779f2756a924ab6ba1bc1702a412e06 [id=8ed572891f9b41cba8ed984911fa6b11]). Expected version 1 got 0
I20251027 21:07:59.831346 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d8825b80cec4ba68da077ca3b8d89db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:48 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:59.831645 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet e0968c2f25b64f6685f38a999ee9d60e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:07:59.832655 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 566) took 3060 ms. Trace:
I20251027 21:07:59.832742 10040 rpcz_store.cc:276] 1027 21:07:56.771934 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.999130 (+227196us) service_pool.cc:225] Handling call
1027 21:07:59.770017 (+2770887us) ts_tablet_manager.cc:1151] Deleting tablet cb1686fc7c3a4996aa4462884d30c41f
1027 21:07:59.770026 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.806978 (+ 36952us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.810004 (+ 3026us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.832639 (+ 22635us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2770736}
I20251027 21:07:59.834311 10032 tablet_bootstrap.cc:492] T 94f2dbf47a5c4458bee54b26f272156e P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:59.834492 10032 ts_tablet_manager.cc:1403] T 94f2dbf47a5c4458bee54b26f272156e P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20251027 21:07:59.834470 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet cb1686fc7c3a4996aa4462884d30c41f (table default.Table_4a21d4b2faee414e88ba4533b814cce2 [id=3f2856bb858142989104b3f1d9e7f510]) successfully deleted
I20251027 21:07:59.835289 10032 raft_consensus.cc:359] T 94f2dbf47a5c4458bee54b26f272156e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:59.835492 10032 raft_consensus.cc:385] T 94f2dbf47a5c4458bee54b26f272156e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.835577 10032 raft_consensus.cc:740] T 94f2dbf47a5c4458bee54b26f272156e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.835822 10032 consensus_queue.cc:260] T 94f2dbf47a5c4458bee54b26f272156e P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:59.836299 10032 ts_tablet_manager.cc:1434] T 94f2dbf47a5c4458bee54b26f272156e P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:07:59.836608 10032 tablet_bootstrap.cc:492] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:59.794488 10043 ts_tablet_manager.cc:1916] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.838362 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 567) took 3066 ms. Trace:
I20251027 21:07:59.838449 10040 rpcz_store.cc:276] 1027 21:07:56.772049 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.999288 (+227239us) service_pool.cc:225] Handling call
1027 21:07:59.838237 (+2838949us) ts_tablet_manager.cc:1151] Deleting tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a
1027 21:07:59.838247 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.838343 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2838812}
I20251027 21:07:59.839001 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 568) took 3066 ms. Trace:
I20251027 21:07:59.839092 10040 rpcz_store.cc:276] 1027 21:07:56.772132 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.999439 (+227307us) service_pool.cc:225] Handling call
1027 21:07:59.838895 (+2839456us) ts_tablet_manager.cc:1151] Deleting tablet 78884c2be75f45f881f5424d3b3f4e08
1027 21:07:59.838907 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.838984 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2839204}
I20251027 21:07:59.839555 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 569) took 3067 ms. Trace:
I20251027 21:07:59.839636 10040 rpcz_store.cc:276] 1027 21:07:56.772217 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.999622 (+227405us) service_pool.cc:225] Handling call
1027 21:07:59.839460 (+2839838us) ts_tablet_manager.cc:1151] Deleting tablet 82366f8bf4154ffd87ef45e4a4d5f54b
1027 21:07:59.839469 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.839537 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2839707}
I20251027 21:07:59.840111 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 570) took 3067 ms. Trace:
I20251027 21:07:59.840178 10040 rpcz_store.cc:276] 1027 21:07:56.772295 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.999767 (+227472us) service_pool.cc:225] Handling call
1027 21:07:59.840012 (+2840245us) ts_tablet_manager.cc:1151] Deleting tablet 465be1b24d16457e834023a2b2194049
1027 21:07:59.840022 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.840095 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2840082}
I20251027 21:07:59.840607 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 571) took 3068 ms. Trace:
I20251027 21:07:59.840678 10040 rpcz_store.cc:276] 1027 21:07:56.772370 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:56.999962 (+227592us) service_pool.cc:225] Handling call
1027 21:07:59.840512 (+2840550us) ts_tablet_manager.cc:1151] Deleting tablet a7d8d060916941a5b7e944e8b08a2882
1027 21:07:59.840520 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.840591 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2840423}
I20251027 21:07:59.841077 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 572) took 3068 ms. Trace:
I20251027 21:07:59.841158 10040 rpcz_store.cc:276] 1027 21:07:56.772457 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.000095 (+227638us) service_pool.cc:225] Handling call
1027 21:07:59.840992 (+2840897us) ts_tablet_manager.cc:1151] Deleting tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a
1027 21:07:59.841000 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.841065 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2840746}
I20251027 21:07:59.841534 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 604) took 2422 ms. Trace:
I20251027 21:07:59.841605 10040 rpcz_store.cc:276] 1027 21:07:57.419041 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.419091 (+ 50us) service_pool.cc:225] Handling call
1027 21:07:59.841456 (+2422365us) ts_tablet_manager.cc:1151] Deleting tablet 84b9de82d55447bba2a897edb683e5c5
1027 21:07:59.841463 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.841523 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2410544}
I20251027 21:07:59.841995 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 614) took 2318 ms. Trace:
W20251027 21:07:59.842820 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a already in progress: creating tablet
W20251027 21:07:59.843164 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 78884c2be75f45f881f5424d3b3f4e08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78884c2be75f45f881f5424d3b3f4e08 already in progress: creating tablet
W20251027 21:07:59.843429 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 82366f8bf4154ffd87ef45e4a4d5f54b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82366f8bf4154ffd87ef45e4a4d5f54b already in progress: creating tablet
W20251027 21:07:59.843677 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 465be1b24d16457e834023a2b2194049 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 465be1b24d16457e834023a2b2194049 already in progress: creating tablet
W20251027 21:07:59.843891 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet a7d8d060916941a5b7e944e8b08a2882 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7d8d060916941a5b7e944e8b08a2882 already in progress: creating tablet
W20251027 21:07:59.844125 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12e2ae1cc3e14ecc90a7ae88c8e8051a already in progress: creating tablet
W20251027 21:07:59.844369 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 84b9de82d55447bba2a897edb683e5c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84b9de82d55447bba2a897edb683e5c5 already in progress: creating tablet
I20251027 21:07:59.848011 10029 tablet_bootstrap.cc:492] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:07:59.850896 10032 tablet_bootstrap.cc:654] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.854380 9895 tablet_service.cc:1505] Processing CreateTablet for tablet f33f7c3098674b0987f766f67c28a9c8 (DEFAULT_TABLE table=default.Table_918531cae34b4834ba3017dacf1181d6 [id=08127c09734e4238b86ba38065c4cf0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.854765 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f33f7c3098674b0987f766f67c28a9c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.842059 10040 rpcz_store.cc:276] 1027 21:07:57.523597 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.523656 (+ 59us) service_pool.cc:225] Handling call
1027 21:07:59.841899 (+2318243us) ts_tablet_manager.cc:1151] Deleting tablet 1329ff0f8ebe4da8a6462a574ede98d4
1027 21:07:59.841907 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.841983 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1974027}
I20251027 21:07:59.856092 10040 tablet_replica.cc:333] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:07:59.856266 10040 raft_consensus.cc:2243] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.856426 10040 raft_consensus.cc:2272] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.856959 10031 tablet_bootstrap.cc:492] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:59.857062 10031 ts_tablet_manager.cc:1403] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20251027 21:07:59.857705 10031 raft_consensus.cc:359] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.857923 10031 raft_consensus.cc:385] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.857990 10031 raft_consensus.cc:740] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.858243 10031 consensus_queue.cc:260] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.858470 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 5443164d67cf445ca847b84c756635e6 (DEFAULT_TABLE table=default.Table_2da8f285aede4af7b444347f7183293f [id=62ba4f7526d74721a1fd3676a737fc01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.858875 10031 ts_tablet_manager.cc:1434] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 21:07:59.858883 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5443164d67cf445ca847b84c756635e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.860649 10041 ts_tablet_manager.cc:1929] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.860766 10041 log.cc:1199] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/a287439ea6c9469da5fb56d84ee68048
I20251027 21:07:59.861133 10041 ts_tablet_manager.cc:1950] T a287439ea6c9469da5fb56d84ee68048 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:59.862197 10041 tablet_replica.cc:333] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:59.862385 10041 raft_consensus.cc:2243] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.862517 10041 raft_consensus.cc:2272] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.862557 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet a287439ea6c9469da5fb56d84ee68048 (table default.Table_1ecaac632a024248b04e7c40e2c01d2c [id=4e2210c84e4c464da017d1b36ddb0a37]) successfully deleted
I20251027 21:07:59.862972 10041 ts_tablet_manager.cc:1916] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.863314 10040 ts_tablet_manager.cc:1916] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.863500 9893 tablet_service.cc:1505] Processing CreateTablet for tablet ef6f69db3d2547c88bbc8df2d52e2fb1 (DEFAULT_TABLE table=default.Table_066671f8fa8b403386935bcda7a8d095 [id=24bdea549f024941b08739d57ab2b231]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.863875 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef6f69db3d2547c88bbc8df2d52e2fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.865509 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 8b069d2601394359b2dde518a2df9c67 (DEFAULT_TABLE table=default.Table_7714994006124254a5feb37d68ce5808 [id=729a0016f8a24804841e11703cbdf134]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.865841 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b069d2601394359b2dde518a2df9c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.866382 10031 tablet_bootstrap.cc:492] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:07:59.866406 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 5443164d67cf445ca847b84c756635e6 (DEFAULT_TABLE table=default.Table_2da8f285aede4af7b444347f7183293f [id=62ba4f7526d74721a1fd3676a737fc01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.867314 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5443164d67cf445ca847b84c756635e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.867401 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 2741d50c7654407194488e14b55460ad (DEFAULT_TABLE table=default.Table_e6dc3cc1aeef48a48107dc699cbb2238 [id=337ad864eac647388eda5c3f1acc7dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.867738 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2741d50c7654407194488e14b55460ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.868587 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 49b6d7e547444187b7027b79e4ede9de (DEFAULT_TABLE table=default.Table_6666bff9087744179f5eb067a74af9f9 [id=603277da57d54a7da6e436ef9f3debd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.868924 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49b6d7e547444187b7027b79e4ede9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.869834 10031 tablet_bootstrap.cc:654] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.870540 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 2741d50c7654407194488e14b55460ad (DEFAULT_TABLE table=default.Table_e6dc3cc1aeef48a48107dc699cbb2238 [id=337ad864eac647388eda5c3f1acc7dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.870986 9615 tablet_service.cc:1505] Processing CreateTablet for tablet e73687f0f0e0499780cc40d1f48c91f2 (DEFAULT_TABLE table=default.Table_a5ded5d12b9d45128034b8672f42f808 [id=fb16fc7546ae4d328cedf3b77304560b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.871356 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e73687f0f0e0499780cc40d1f48c91f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.874287 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2741d50c7654407194488e14b55460ad. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:59.877027 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 1329ff0f8ebe4da8a6462a574ede98d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1329ff0f8ebe4da8a6462a574ede98d4 already in progress: creating tablet
I20251027 21:07:59.877524 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 8b069d2601394359b2dde518a2df9c67 (DEFAULT_TABLE table=default.Table_7714994006124254a5feb37d68ce5808 [id=729a0016f8a24804841e11703cbdf134]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.878007 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 5443164d67cf445ca847b84c756635e6 (DEFAULT_TABLE table=default.Table_2da8f285aede4af7b444347f7183293f [id=62ba4f7526d74721a1fd3676a737fc01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.878384 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 2741d50c7654407194488e14b55460ad (DEFAULT_TABLE table=default.Table_e6dc3cc1aeef48a48107dc699cbb2238 [id=337ad864eac647388eda5c3f1acc7dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.878842 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 8b069d2601394359b2dde518a2df9c67 (DEFAULT_TABLE table=default.Table_7714994006124254a5feb37d68ce5808 [id=729a0016f8a24804841e11703cbdf134]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.879128 9755 tablet_service.cc:1505] Processing CreateTablet for tablet e73687f0f0e0499780cc40d1f48c91f2 (DEFAULT_TABLE table=default.Table_a5ded5d12b9d45128034b8672f42f808 [id=fb16fc7546ae4d328cedf3b77304560b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.879208 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b069d2601394359b2dde518a2df9c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.879506 9739 tablet_service.cc:1505] Processing CreateTablet for tablet ef6f69db3d2547c88bbc8df2d52e2fb1 (DEFAULT_TABLE table=default.Table_066671f8fa8b403386935bcda7a8d095 [id=24bdea549f024941b08739d57ab2b231]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.879863 9750 tablet_service.cc:1505] Processing CreateTablet for tablet f33f7c3098674b0987f766f67c28a9c8 (DEFAULT_TABLE table=default.Table_918531cae34b4834ba3017dacf1181d6 [id=08127c09734e4238b86ba38065c4cf0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.880045 9890 tablet_service.cc:1505] Processing CreateTablet for tablet e73687f0f0e0499780cc40d1f48c91f2 (DEFAULT_TABLE table=default.Table_a5ded5d12b9d45128034b8672f42f808 [id=fb16fc7546ae4d328cedf3b77304560b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.880383 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e73687f0f0e0499780cc40d1f48c91f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.884500 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet b87e809ff897417c88e5edc19e2d591e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:07:59.886215 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 49b6d7e547444187b7027b79e4ede9de (DEFAULT_TABLE table=default.Table_6666bff9087744179f5eb067a74af9f9 [id=603277da57d54a7da6e436ef9f3debd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.886633 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49b6d7e547444187b7027b79e4ede9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.890404 9744 tablet_service.cc:1505] Processing CreateTablet for tablet 49b6d7e547444187b7027b79e4ede9de (DEFAULT_TABLE table=default.Table_6666bff9087744179f5eb067a74af9f9 [id=603277da57d54a7da6e436ef9f3debd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.890560 10628 raft_consensus.cc:493] T abd3b8e71d3041b4b9e5f6663841bfcf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.890712 10628 raft_consensus.cc:515] T abd3b8e71d3041b4b9e5f6663841bfcf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:07:59.891312 10628 leader_election.cc:290] T abd3b8e71d3041b4b9e5f6663841bfcf P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:59.891450 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 196a594025ad4564b028b39f1afae48f (DEFAULT_TABLE table=default.Table_3478938a9d2b44e7875de2ae7a4d471b [id=d0c2ddb467d149fe855d2d40575de3cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.891778 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 196a594025ad4564b028b39f1afae48f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.891861 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abd3b8e71d3041b4b9e5f6663841bfcf" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:07:59.892318 9859 leader_election.cc:343] T abd3b8e71d3041b4b9e5f6663841bfcf P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.892999 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 463ac409dcd84dbd989534a3b45c9d4b (DEFAULT_TABLE table=default.Table_131a037349c446e4a99258abc4815b26 [id=19081a807f0c413f99a391c23654c87a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.893464 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 463ac409dcd84dbd989534a3b45c9d4b (DEFAULT_TABLE table=default.Table_131a037349c446e4a99258abc4815b26 [id=19081a807f0c413f99a391c23654c87a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.893828 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 463ac409dcd84dbd989534a3b45c9d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.893883 10032 tablet_bootstrap.cc:492] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:07:59.894049 10032 ts_tablet_manager.cc:1403] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.002s
I20251027 21:07:59.894872 10032 raft_consensus.cc:359] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:59.895047 10032 raft_consensus.cc:385] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.895157 10032 raft_consensus.cc:740] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:07:59.895360 10032 consensus_queue.cc:260] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:59.895936 10032 ts_tablet_manager.cc:1434] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:07:59.896245 10032 tablet_bootstrap.cc:492] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:07:59.897329 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 18a29676db8547929c6cb4084c63f4c8 (DEFAULT_TABLE table=default.Table_fcb69a1b675e4af592b2ce080f423003 [id=44688cf04f994b7ea7ee86fa3e979311]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.897648 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18a29676db8547929c6cb4084c63f4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.898005 10032 tablet_bootstrap.cc:654] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.898468 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 2df0e9fa2c6f448592d348ab19abe377 (DEFAULT_TABLE table=default.Table_3ad4d44124674e9aae353b207951534e [id=a5c2a502cd06450b8942f4a54405dace]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.898784 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2df0e9fa2c6f448592d348ab19abe377. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.900374 10560 raft_consensus.cc:493] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.900547 10560 raft_consensus.cc:515] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:59.901175 10560 leader_election.cc:290] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:59.902407 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b069d2601394359b2dde518a2df9c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.902412 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 2df0e9fa2c6f448592d348ab19abe377 (DEFAULT_TABLE table=default.Table_3ad4d44124674e9aae353b207951534e [id=a5c2a502cd06450b8942f4a54405dace]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.902595 9595 tablet_service.cc:1505] Processing CreateTablet for tablet 35b99f8c57e847b1848cbdea9e50da11 (DEFAULT_TABLE table=default.Table_3de9fdb7f17a46e0a0560f82afaef828 [id=6034a408a8bd40f28ef208755a3bf8ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.902745 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2df0e9fa2c6f448592d348ab19abe377. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.902905 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5443164d67cf445ca847b84c756635e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.902957 9595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35b99f8c57e847b1848cbdea9e50da11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.903302 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2741d50c7654407194488e14b55460ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.903658 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e73687f0f0e0499780cc40d1f48c91f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.904033 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef6f69db3d2547c88bbc8df2d52e2fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.904366 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49b6d7e547444187b7027b79e4ede9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.904696 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f33f7c3098674b0987f766f67c28a9c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.906687 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 18a29676db8547929c6cb4084c63f4c8 (DEFAULT_TABLE table=default.Table_fcb69a1b675e4af592b2ce080f423003 [id=44688cf04f994b7ea7ee86fa3e979311]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.907016 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18a29676db8547929c6cb4084c63f4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.893006 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 196a594025ad4564b028b39f1afae48f (DEFAULT_TABLE table=default.Table_3478938a9d2b44e7875de2ae7a4d471b [id=d0c2ddb467d149fe855d2d40575de3cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.915148 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 463ac409dcd84dbd989534a3b45c9d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.916983 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 35b99f8c57e847b1848cbdea9e50da11 (DEFAULT_TABLE table=default.Table_3de9fdb7f17a46e0a0560f82afaef828 [id=6034a408a8bd40f28ef208755a3bf8ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.917299 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35b99f8c57e847b1848cbdea9e50da11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.920017 9600 tablet_service.cc:1505] Processing CreateTablet for tablet ef6f69db3d2547c88bbc8df2d52e2fb1 (DEFAULT_TABLE table=default.Table_066671f8fa8b403386935bcda7a8d095 [id=24bdea549f024941b08739d57ab2b231]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.920317 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef6f69db3d2547c88bbc8df2d52e2fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.923143 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 463ac409dcd84dbd989534a3b45c9d4b (DEFAULT_TABLE table=default.Table_131a037349c446e4a99258abc4815b26 [id=19081a807f0c413f99a391c23654c87a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.923524 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 463ac409dcd84dbd989534a3b45c9d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.926256 10665 raft_consensus.cc:493] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.926375 10665 raft_consensus.cc:515] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.926599 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "adaa4f66bbab491f8030d439d85cc7d1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:59.926918 10665 leader_election.cc:290] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:07:59.926918 10628 raft_consensus.cc:493] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:07:59.927045 10628 raft_consensus.cc:515] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:07:59.927520 10628 leader_election.cc:290] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:07:59.929620 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 7a28dc9fe75744e5aa3d486afbe29561 (DEFAULT_TABLE table=default.Table_9c8bc163f60741c3bf7fed269c243545 [id=35fe8b06774e41f9bef7e790a46d5cb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.929930 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a28dc9fe75744e5aa3d486afbe29561. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.931551 10041 ts_tablet_manager.cc:1929] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.931633 10041 log.cc:1199] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/132a01405a63447ab27b2137b803224f
I20251027 21:07:59.931875 10041 ts_tablet_manager.cc:1950] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:07:59.938170 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 013dc05778a047f691789cf21d241151 (DEFAULT_TABLE table=default.Table_5291bf5018e443118fabd8171b612b07 [id=db994b7f0dcd4225a39bdacbe1e64feb]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:59.938442 9710 leader_election.cc:343] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: b87e809ff897417c88e5edc19e2d591e
I20251027 21:07:59.938575 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 013dc05778a047f691789cf21d241151. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:59.938814 9859 leader_election.cc:343] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.939085 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8856c5dc8f144358445165324bfb477" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:07:59.939402 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 013dc05778a047f691789cf21d241151 (DEFAULT_TABLE table=default.Table_5291bf5018e443118fabd8171b612b07 [id=db994b7f0dcd4225a39bdacbe1e64feb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.939960 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 013dc05778a047f691789cf21d241151. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.940299 9595 tablet_service.cc:1505] Processing CreateTablet for tablet dac3c4836e91423ba5baa58039dc0c55 (DEFAULT_TABLE table=default.Table_5dee7205c58b4b70beadf92f948aea58 [id=3725abc3a27e4a4e9911eb52b430aae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.940593 9595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dac3c4836e91423ba5baa58039dc0c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.942152 9898 tablet_service.cc:1505] Processing CreateTablet for tablet dac3c4836e91423ba5baa58039dc0c55 (DEFAULT_TABLE table=default.Table_5dee7205c58b4b70beadf92f948aea58 [id=3725abc3a27e4a4e9911eb52b430aae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.942498 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dac3c4836e91423ba5baa58039dc0c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.942826 9893 tablet_service.cc:1505] Processing CreateTablet for tablet b6c4fc648fa04d9aa4a10b5b6b07cd21 (DEFAULT_TABLE table=default.Table_8fee41d5402246f2b96068387e60cc49 [id=97ca5ce4c6724b1ebc2c90f667ffcd71]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:59.943034 9859 leader_election.cc:343] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:07:59.943163 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6c4fc648fa04d9aa4a10b5b6b07cd21. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:07:59.944202 9711 leader_election.cc:343] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: b87e809ff897417c88e5edc19e2d591e
I20251027 21:07:59.944350 9711 leader_election.cc:304] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:07:59.944607 10665 raft_consensus.cc:2749] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:07:59.945559 9608 tablet_service.cc:1505] Processing CreateTablet for tablet 00bfb0bdf2ca4b288eac0c1dbee5765a (DEFAULT_TABLE table=default.Table_0432e7918b494dccb2f6b921657c0d81 [id=f773d65a07a4473ab8b87cd82ac50bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.945837 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00bfb0bdf2ca4b288eac0c1dbee5765a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.946403 9599 tablet_service.cc:1505] Processing CreateTablet for tablet f33f7c3098674b0987f766f67c28a9c8 (DEFAULT_TABLE table=default.Table_918531cae34b4834ba3017dacf1181d6 [id=08127c09734e4238b86ba38065c4cf0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.946696 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f33f7c3098674b0987f766f67c28a9c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.947571 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 7a28dc9fe75744e5aa3d486afbe29561 (DEFAULT_TABLE table=default.Table_9c8bc163f60741c3bf7fed269c243545 [id=35fe8b06774e41f9bef7e790a46d5cb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.947908 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a28dc9fe75744e5aa3d486afbe29561. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.948853 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 00bfb0bdf2ca4b288eac0c1dbee5765a (DEFAULT_TABLE table=default.Table_0432e7918b494dccb2f6b921657c0d81 [id=f773d65a07a4473ab8b87cd82ac50bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.949153 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00bfb0bdf2ca4b288eac0c1dbee5765a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.955513 10029 tablet_bootstrap.cc:700] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14: Time spent opening tablet: real 0.102s user 0.000s sys 0.000s
I20251027 21:07:59.955782 10029 tablet_bootstrap.cc:654] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:07:59.958609 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 31c6aeeac212490ea9b13aa772765277 (DEFAULT_TABLE table=default.Table_56aebdec46ea4837a4a42ff7c45aebba [id=5d21f09d2cea42b0a21318716f2a02f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.958797 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 31c6aeeac212490ea9b13aa772765277 (DEFAULT_TABLE table=default.Table_56aebdec46ea4837a4a42ff7c45aebba [id=5d21f09d2cea42b0a21318716f2a02f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.959080 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31c6aeeac212490ea9b13aa772765277. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.959136 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31c6aeeac212490ea9b13aa772765277. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.960160 10040 ts_tablet_manager.cc:1929] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:07:59.960319 10040 log.cc:1199] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/1776590d8387494da5dfbd08db17f4b1
I20251027 21:07:59.960834 10040 ts_tablet_manager.cc:1950] T 1776590d8387494da5dfbd08db17f4b1 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:07:59.962447 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 132a01405a63447ab27b2137b803224f (table default.Table_673503358b54481c8877034c26349796 [id=c6aed7584a0b49d89bf9a8f8ef0cb429]) successfully deleted
I20251027 21:07:59.962949 10041 tablet_replica.cc:333] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:07:59.962690 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 609) took 2440 ms. Trace:
I20251027 21:07:59.964745 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e7d7ba7c45a54852b436abe87effeaa0" } new_table_name: "default.Table_c09cc86fc2a64450af56fc584302da69" modify_external_catalogs: true
I20251027 21:07:59.964493 10041 raft_consensus.cc:2243] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:07:59.963353 10040 rpcz_store.cc:276] 1027 21:07:57.521813 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.523924 (+ 2111us) service_pool.cc:225] Handling call
1027 21:07:59.856047 (+2332123us) ts_tablet_manager.cc:1151] Deleting tablet 1776590d8387494da5dfbd08db17f4b1
1027 21:07:59.856056 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.905119 (+ 49063us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.960019 (+ 54900us) tablet_metadata.cc:641] Metadata flushed
1027 21:07:59.962510 (+ 2491us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1988070}
I20251027 21:07:59.966185 10041 raft_consensus.cc:2272] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:07:59.967582 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 615) took 2443 ms. Trace:
I20251027 21:07:59.969309 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_352d903f119f41d3b405c925eaf2ebaa" } new_table_name: "default.Table_2d0188001702444a847e0cfbde0ead2f" modify_external_catalogs: true
I20251027 21:07:59.968648 10040 rpcz_store.cc:276] 1027 21:07:57.523886 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.524078 (+ 192us) service_pool.cc:225] Handling call
1027 21:07:59.966355 (+2442277us) ts_tablet_manager.cc:1151] Deleting tablet 69da799c886841a19002d3c7c5084e73
1027 21:07:59.966589 (+ 234us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.967263 (+ 674us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2098314}
I20251027 21:07:59.974375 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0efd5aacf5b54ded90984a7c702d12d6" } modify_external_catalogs: true
I20251027 21:07:59.974571 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0efd5aacf5b54ded90984a7c702d12d6" } modify_external_catalogs: true
I20251027 21:07:59.973893 10041 ts_tablet_manager.cc:1916] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:07:59.974581 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 616) took 2449 ms. Trace:
I20251027 21:07:59.976722 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e13c97413ddf417ea96834ba933b537b" } new_table_name: "default.Table_a05ec8bfd95c4f82bd16fbd66e110afc" modify_external_catalogs: true
I20251027 21:07:59.975319 10040 rpcz_store.cc:276] 1027 21:07:57.524297 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.524340 (+ 43us) service_pool.cc:225] Handling call
1027 21:07:59.972749 (+2448409us) ts_tablet_manager.cc:1151] Deleting tablet 8fc2d7d6601d4cc2849064f4cdddc90d
1027 21:07:59.973175 (+ 426us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.974208 (+ 1033us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2104643}
I20251027 21:07:59.978158 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d120d0c629944b8582b7177c111c78f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.978693 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d120d0c629944b8582b7177c111c78f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.980046 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6f6eba299ba04a7398b597dd4caf6d9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:07:59.980536 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f6eba299ba04a7398b597dd4caf6d9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:07:59.981362 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 605) took 2459 ms. Trace:
I20251027 21:07:59.982347 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 196a594025ad4564b028b39f1afae48f (DEFAULT_TABLE table=default.Table_3478938a9d2b44e7875de2ae7a4d471b [id=d0c2ddb467d149fe855d2d40575de3cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.982702 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 196a594025ad4564b028b39f1afae48f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.982556 10040 rpcz_store.cc:276] 1027 21:07:57.521383 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.543344 (+ 21961us) service_pool.cc:225] Handling call
1027 21:07:59.979642 (+2436298us) ts_tablet_manager.cc:1151] Deleting tablet 07a9c3b7ef1548ec9d5ae9f20545f263
1027 21:07:59.979654 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.980992 (+ 1338us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2108491}
I20251027 21:07:59.987391 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 31c6aeeac212490ea9b13aa772765277 (DEFAULT_TABLE table=default.Table_56aebdec46ea4837a4a42ff7c45aebba [id=5d21f09d2cea42b0a21318716f2a02f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.987756 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 1776590d8387494da5dfbd08db17f4b1 (table default.Table_b779f2756a924ab6ba1bc1702a412e06 [id=8ed572891f9b41cba8ed984911fa6b11]) successfully deleted
I20251027 21:07:59.987759 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 00bfb0bdf2ca4b288eac0c1dbee5765a (DEFAULT_TABLE table=default.Table_0432e7918b494dccb2f6b921657c0d81 [id=f773d65a07a4473ab8b87cd82ac50bcf]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:59.988095 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 69da799c886841a19002d3c7c5084e73 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69da799c886841a19002d3c7c5084e73 already in progress: creating tablet
I20251027 21:07:59.988090 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 013dc05778a047f691789cf21d241151 (DEFAULT_TABLE table=default.Table_5291bf5018e443118fabd8171b612b07 [id=db994b7f0dcd4225a39bdacbe1e64feb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.988405 9744 tablet_service.cc:1505] Processing CreateTablet for tablet 18a29676db8547929c6cb4084c63f4c8 (DEFAULT_TABLE table=default.Table_fcb69a1b675e4af592b2ce080f423003 [id=44688cf04f994b7ea7ee86fa3e979311]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:07:59.988466 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 8fc2d7d6601d4cc2849064f4cdddc90d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fc2d7d6601d4cc2849064f4cdddc90d already in progress: creating tablet
I20251027 21:07:59.989784 9750 tablet_service.cc:1505] Processing CreateTablet for tablet b6c4fc648fa04d9aa4a10b5b6b07cd21 (DEFAULT_TABLE table=default.Table_8fee41d5402246f2b96068387e60cc49 [id=97ca5ce4c6724b1ebc2c90f667ffcd71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.990058 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 7a28dc9fe75744e5aa3d486afbe29561 (DEFAULT_TABLE table=default.Table_9c8bc163f60741c3bf7fed269c243545 [id=35fe8b06774e41f9bef7e790a46d5cb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.990494 9739 tablet_service.cc:1505] Processing CreateTablet for tablet dac3c4836e91423ba5baa58039dc0c55 (DEFAULT_TABLE table=default.Table_5dee7205c58b4b70beadf92f948aea58 [id=3725abc3a27e4a4e9911eb52b430aae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.990832 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 35b99f8c57e847b1848cbdea9e50da11 (DEFAULT_TABLE table=default.Table_3de9fdb7f17a46e0a0560f82afaef828 [id=6034a408a8bd40f28ef208755a3bf8ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.991355 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 2df0e9fa2c6f448592d348ab19abe377 (DEFAULT_TABLE table=default.Table_3ad4d44124674e9aae353b207951534e [id=a5c2a502cd06450b8942f4a54405dace]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:07:59.995200 10031 tablet_bootstrap.cc:492] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:07:59.995332 10031 ts_tablet_manager.cc:1403] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.129s user 0.008s sys 0.001s
I20251027 21:07:59.997756 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 196a594025ad4564b028b39f1afae48f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:07:59.998031 10031 raft_consensus.cc:359] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.998231 10031 raft_consensus.cc:385] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:07:59.998289 10031 raft_consensus.cc:740] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
W20251027 21:07:59.998231 9278 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 45647.9us
I20251027 21:07:59.998486 10031 consensus_queue.cc:260] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:07:59.998942 10031 ts_tablet_manager.cc:1434] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251027 21:07:59.999397 10031 tablet_bootstrap.cc:492] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:00.001215 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18a29676db8547929c6cb4084c63f4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.002229 10029 tablet_bootstrap.cc:492] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:00.002352 10029 ts_tablet_manager.cc:1403] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.154s user 0.001s sys 0.003s
I20251027 21:08:00.002635 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2088d599bbe444a18dee497154bb4379"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:00.003137 10029 raft_consensus.cc:359] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.003304 10029 raft_consensus.cc:385] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.003350 10029 raft_consensus.cc:740] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
W20251027 21:08:00.003587 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2088d599bbe444a18dee497154bb4379 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.003551 10029 consensus_queue.cc:260] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.005234 10029 ts_tablet_manager.cc:1434] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:08:00.005492 10029 tablet_bootstrap.cc:492] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:00.006085 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a28dc9fe75744e5aa3d486afbe29561. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.006455 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00bfb0bdf2ca4b288eac0c1dbee5765a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.005517 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 9b261415f48a43229fbb342ac7f7a677 (DEFAULT_TABLE table=default.Table_c8fded5f6e8b4798a55ec84954d75574 [id=4b8973449dc6418787f2055cb3099fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.006986 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b261415f48a43229fbb342ac7f7a677. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.007292 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 013dc05778a047f691789cf21d241151. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.007493 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31c6aeeac212490ea9b13aa772765277. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.007553 10029 tablet_bootstrap.cc:654] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.007826 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6c4fc648fa04d9aa4a10b5b6b07cd21. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.009474 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dac3c4836e91423ba5baa58039dc0c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.010517 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 9b261415f48a43229fbb342ac7f7a677 (DEFAULT_TABLE table=default.Table_c8fded5f6e8b4798a55ec84954d75574 [id=4b8973449dc6418787f2055cb3099fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.010654 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35b99f8c57e847b1848cbdea9e50da11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.010941 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b261415f48a43229fbb342ac7f7a677. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:00.014285 9860 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.9.14.125:46313, user_credentials={real_user=slave}} blocked reactor thread for 224120us
I20251027 21:08:00.015971 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2df0e9fa2c6f448592d348ab19abe377. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.016415 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abd3b8e71d3041b4b9e5f6663841bfcf" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:00.016774 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "adaa4f66bbab491f8030d439d85cc7d1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:00.017063 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8856c5dc8f144358445165324bfb477" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:00.027952 10043 ts_tablet_manager.cc:1929] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.028071 10043 log.cc:1199] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/ed5619d295a246379b625f35692203e4
I20251027 21:08:00.028383 10043 ts_tablet_manager.cc:1950] T ed5619d295a246379b625f35692203e4 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:00.029239 9602 tablet_service.cc:1505] Processing CreateTablet for tablet b6c4fc648fa04d9aa4a10b5b6b07cd21 (DEFAULT_TABLE table=default.Table_8fee41d5402246f2b96068387e60cc49 [id=97ca5ce4c6724b1ebc2c90f667ffcd71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.029556 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6c4fc648fa04d9aa4a10b5b6b07cd21. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.030118 10665 raft_consensus.cc:493] T 7f7ad06a15c04efbb701634e14904351 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.030261 10665 raft_consensus.cc:515] T 7f7ad06a15c04efbb701634e14904351 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:00.031330 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 628) took 2503 ms. Trace:
I20251027 21:08:00.031409 10040 rpcz_store.cc:276] 1027 21:07:57.527364 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.868046 (+340682us) service_pool.cc:225] Handling call
1027 21:07:59.985873 (+2117827us) ts_tablet_manager.cc:1151] Deleting tablet 9b9ffa18afa54f2c9ffd7b5a03a3a324
1027 21:07:59.985883 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.031313 (+ 45430us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2102389}
W20251027 21:08:00.032203 9297 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 43692.5us
W20251027 21:08:00.032507 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 07a9c3b7ef1548ec9d5ae9f20545f263 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07a9c3b7ef1548ec9d5ae9f20545f263 already in progress: creating tablet
I20251027 21:08:00.038542 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 668) took 1184 ms. Trace:
I20251027 21:08:00.038627 10043 rpcz_store.cc:276] 1027 21:07:58.854129 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.854385 (+ 256us) service_pool.cc:225] Handling call
1027 21:07:59.784533 (+930148us) ts_tablet_manager.cc:1151] Deleting tablet ed5619d295a246379b625f35692203e4
1027 21:07:59.784542 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:07:59.841540 (+ 56998us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.027872 (+186332us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.038523 (+ 10651us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":929972}
I20251027 21:08:00.039371 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 669) took 1172 ms. Trace:
I20251027 21:08:00.039444 10043 rpcz_store.cc:276] 1027 21:07:58.866491 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.866538 (+ 47us) service_pool.cc:225] Handling call
1027 21:08:00.039253 (+1172715us) ts_tablet_manager.cc:1151] Deleting tablet 8b30052ff19744a9bf4bfaaa1bc77b36
1027 21:08:00.039262 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.039352 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1172519}
I20251027 21:08:00.039852 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 670) took 1138 ms. Trace:
I20251027 21:08:00.039928 10043 rpcz_store.cc:276] 1027 21:07:58.900955 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.901154 (+ 199us) service_pool.cc:225] Handling call
1027 21:08:00.039753 (+1138599us) ts_tablet_manager.cc:1151] Deleting tablet 38f2133010b14aad88b550fb12c966a5
1027 21:08:00.039761 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.039839 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1138370}
I20251027 21:08:00.040231 10043 tablet_replica.cc:333] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:00.040385 10043 raft_consensus.cc:2243] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.040499 10043 raft_consensus.cc:2272] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.040953 10043 ts_tablet_manager.cc:1916] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.041868 10041 ts_tablet_manager.cc:1929] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.039111 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet ed5619d295a246379b625f35692203e4 (table default.Table_32f38256207d4fa79ed533151449afa7 [id=e3c04aba1dbf46928a0c36444dd69d02]) successfully deleted
W20251027 21:08:00.042951 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 8b30052ff19744a9bf4bfaaa1bc77b36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b30052ff19744a9bf4bfaaa1bc77b36 already in progress: creating tablet
W20251027 21:08:00.043236 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 38f2133010b14aad88b550fb12c966a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38f2133010b14aad88b550fb12c966a5 already in progress: creating tablet
I20251027 21:08:00.045104 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 123e6e3748574ea581f67fff49037762
I20251027 21:08:00.046869 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e1385beb1dc14ee0b42a2f4e1f66f532
I20251027 21:08:00.047475 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_aca70af1903740b5b35bbc3554beff36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:00.048074 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aca70af1903740b5b35bbc3554beff36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.048614 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4ba3ecc07745498c97bd07deef4525c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:00.049136 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ba3ecc07745498c97bd07deef4525c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.049885 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet deb8b54e5e0f42bca4330d2160e5524a
I20251027 21:08:00.050581 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_63950b7e1b8c4f3a989b8f14a2409f78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:00.051120 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63950b7e1b8c4f3a989b8f14a2409f78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:00.051151 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 9b9ffa18afa54f2c9ffd7b5a03a3a324 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b9ffa18afa54f2c9ffd7b5a03a3a324 already in progress: creating tablet
I20251027 21:08:00.053848 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet e1385beb1dc14ee0b42a2f4e1f66f532 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:00.054056 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet ccaf5c2f8e544842880ba8ba27300ae6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:00.054286 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet deb8b54e5e0f42bca4330d2160e5524a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.056784 10041 log.cc:1199] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/533f34d4872444278477abcce89da9ec
I20251027 21:08:00.057360 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet e1385beb1dc14ee0b42a2f4e1f66f532 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.058012 10694 raft_consensus.cc:493] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.058182 10694 raft_consensus.cc:515] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.058960 10694 leader_election.cc:290] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:00.064272 10043 ts_tablet_manager.cc:1929] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.064433 10043 log.cc:1199] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/8fc2d7d6601d4cc2849064f4cdddc90d
I20251027 21:08:00.064718 10043 ts_tablet_manager.cc:1950] T 8fc2d7d6601d4cc2849064f4cdddc90d P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:00.065743 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 671) took 1156 ms. Trace:
I20251027 21:08:00.065953 10043 rpcz_store.cc:276] 1027 21:07:58.909715 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.909763 (+ 48us) service_pool.cc:225] Handling call
1027 21:08:00.040194 (+1130431us) ts_tablet_manager.cc:1151] Deleting tablet 8fc2d7d6601d4cc2849064f4cdddc90d
1027 21:08:00.040202 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.043694 (+ 3492us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.064188 (+ 20494us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.065726 (+ 1538us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1111821}
I20251027 21:08:00.057355 10041 ts_tablet_manager.cc:1950] T 533f34d4872444278477abcce89da9ec P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:00.066249 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet 123e6e3748574ea581f67fff49037762 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:00.066365 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet deb8b54e5e0f42bca4330d2160e5524a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:00.067639 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 8fc2d7d6601d4cc2849064f4cdddc90d (table default.Table_162a38d7fe14491d8dae6d2e097d3e3a [id=6f39910d23dd455f87a4ac847917b87f]) successfully deleted
I20251027 21:08:00.067222 10029 tablet_bootstrap.cc:492] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:00.068126 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 672) took 1155 ms. Trace:
I20251027 21:08:00.068136 10029 ts_tablet_manager.cc:1403] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.063s user 0.001s sys 0.003s
I20251027 21:08:00.068939 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 533f34d4872444278477abcce89da9ec (table default.Table_61bbf6b5c1e740128b80465ef3b9a50e [id=439f48256cbd43f785f57857e83ae25f]) successfully deleted
I20251027 21:08:00.068243 10043 rpcz_store.cc:276] 1027 21:07:58.913035 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.913084 (+ 49us) service_pool.cc:225] Handling call
1027 21:08:00.067947 (+1154863us) ts_tablet_manager.cc:1151] Deleting tablet 4d88aacbf47144cb837b99d68f6331e2
1027 21:08:00.067960 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.068104 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1139506}
W20251027 21:08:00.069437 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 4d88aacbf47144cb837b99d68f6331e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d88aacbf47144cb837b99d68f6331e2 already in progress: creating tablet
W20251027 21:08:00.069624 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet fc2cf41a7b474f7d86a46af76cef4100 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc2cf41a7b474f7d86a46af76cef4100 already in progress: creating tablet
I20251027 21:08:00.069792 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 676) took 1140 ms. Trace:
W20251027 21:08:00.017517 9860 leader_election.cc:343] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.069868 10043 rpcz_store.cc:276] 1027 21:07:58.929396 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.929447 (+ 51us) service_pool.cc:225] Handling call
1027 21:08:00.069697 (+1140250us) ts_tablet_manager.cc:1151] Deleting tablet a8328318a4f14b38a4ce36d183e99dfb
1027 21:08:00.069705 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.069776 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1140056}
I20251027 21:08:00.070010 9860 leader_election.cc:304] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:00.070215 9860 raft_consensus.cc:2686] T 132a01405a63447ab27b2137b803224f P 0fbd567f929a4f6d8e4d78feef7ed678: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251027 21:08:00.070266 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet a8328318a4f14b38a4ce36d183e99dfb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8328318a4f14b38a4ce36d183e99dfb already in progress: creating tablet
W20251027 21:08:00.070477 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 9239f74d725649cab5ea1b0077bccef3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9239f74d725649cab5ea1b0077bccef3 already in progress: creating tablet
I20251027 21:08:00.070564 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 673) took 1149 ms. Trace:
I20251027 21:08:00.070628 10043 rpcz_store.cc:276] 1027 21:07:58.921580 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.921647 (+ 67us) service_pool.cc:225] Handling call
1027 21:08:00.070465 (+1148818us) ts_tablet_manager.cc:1151] Deleting tablet 6210d3d10fc44b9e9e2b0780883186ce
1027 21:08:00.070474 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.070548 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1140237}
W20251027 21:08:00.070704 9860 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.65:44531, user_credentials={real_user=slave}} blocked reactor thread for 53200.4us
I20251027 21:08:00.070855 10041 tablet_replica.cc:333] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
W20251027 21:08:00.071043 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 6210d3d10fc44b9e9e2b0780883186ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6210d3d10fc44b9e9e2b0780883186ce already in progress: creating tablet
W20251027 21:08:00.071043 9860 leader_election.cc:343] T abd3b8e71d3041b4b9e5f6663841bfcf P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.071074 10041 raft_consensus.cc:2243] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.071161 9860 leader_election.cc:304] T abd3b8e71d3041b4b9e5f6663841bfcf P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:00.071350 10041 raft_consensus.cc:2272] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.071398 10628 consensus_queue.cc:1048] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:00.071906 10628 raft_consensus.cc:2749] T abd3b8e71d3041b4b9e5f6663841bfcf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.071965 10041 ts_tablet_manager.cc:1916] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.072194 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 674) took 1149 ms. Trace:
W20251027 21:08:00.072245 9860 leader_election.cc:343] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.072329 9860 leader_election.cc:304] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:00.072278 10043 rpcz_store.cc:276] 1027 21:07:58.922739 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.922782 (+ 43us) service_pool.cc:225] Handling call
1027 21:08:00.072101 (+1149319us) ts_tablet_manager.cc:1151] Deleting tablet 8ae222307a374a91916f560a0fbe8d8f
1027 21:08:00.072109 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.072177 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1141846}
I20251027 21:08:00.072516 10628 raft_consensus.cc:2749] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:00.072640 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 8ae222307a374a91916f560a0fbe8d8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ae222307a374a91916f560a0fbe8d8f already in progress: creating tablet
W20251027 21:08:00.072880 9860 leader_election.cc:343] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.072971 9860 leader_election.cc:304] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:00.073051 10029 raft_consensus.cc:359] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.073168 10628 raft_consensus.cc:2749] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.074102 10029 raft_consensus.cc:385] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.074601 10029 raft_consensus.cc:740] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.074302 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5f2e60caa824b5db2330b423221873c" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:00.074352 10043 tablet_replica.cc:333] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:00.075387 10043 raft_consensus.cc:2243] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.075525 10043 raft_consensus.cc:2272] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.077668 10029 consensus_queue.cc:260] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.078301 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 123e6e3748574ea581f67fff49037762 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.080129 10029 ts_tablet_manager.cc:1434] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20251027 21:08:00.079002 10043 ts_tablet_manager.cc:1916] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.082481 10041 ts_tablet_manager.cc:1929] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.082773 10041 log.cc:1199] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/21a8fc4a69f3486e92b1a68d10faed38
I20251027 21:08:00.083153 10041 ts_tablet_manager.cc:1950] T 21a8fc4a69f3486e92b1a68d10faed38 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:00.084192 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 21a8fc4a69f3486e92b1a68d10faed38 (table default.Table_ba1d973c05e14c70944d76ff1ecbc1bc [id=490e0e5ec83b4f76bf71eb41c900ac06]) successfully deleted
W20251027 21:08:00.084911 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 57de49dacafa497dad79ab96540b5888 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57de49dacafa497dad79ab96540b5888 already in progress: creating tablet
I20251027 21:08:00.085289 10041 tablet_replica.cc:333] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:00.085543 10041 raft_consensus.cc:2243] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.085736 10041 raft_consensus.cc:2272] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.087190 10665 leader_election.cc:290] T 7f7ad06a15c04efbb701634e14904351 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:00.087548 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 9b261415f48a43229fbb342ac7f7a677 (DEFAULT_TABLE table=default.Table_c8fded5f6e8b4798a55ec84954d75574 [id=4b8973449dc6418787f2055cb3099fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.087850 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 629) took 2560 ms. Trace:
I20251027 21:08:00.088268 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b261415f48a43229fbb342ac7f7a677. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.086660 10041 ts_tablet_manager.cc:1916] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.088788 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f7ad06a15c04efbb701634e14904351" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:00.088469 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f7ad06a15c04efbb701634e14904351" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:00.089834 10703 tablet.cc:1722] T 7aa8c538b6c849a28a28ffe706306891 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:08:00.087922 10040 rpcz_store.cc:276] 1027 21:07:57.527445 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:57.868102 (+340657us) service_pool.cc:225] Handling call
1027 21:08:00.042096 (+2173994us) ts_tablet_manager.cc:1151] Deleting tablet b3abc5f78c4e41e29128e2d790319c75
1027 21:08:00.087729 (+ 45633us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.087831 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2159042}
I20251027 21:08:00.090657 10029 tablet_bootstrap.cc:492] T 21a8fc4a69f3486e92b1a68d10faed38 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:00.090646 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 651) took 1491 ms. Trace:
I20251027 21:08:00.090713 10040 rpcz_store.cc:276] 1027 21:07:58.598972 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.630551 (+ 31579us) service_pool.cc:225] Handling call
1027 21:08:00.090544 (+1459993us) ts_tablet_manager.cc:1151] Deleting tablet d1f24d21bb954ac9b62ce458c6a22462
1027 21:08:00.090554 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.090632 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1459800}
I20251027 21:08:00.091017 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 652) took 1491 ms. Trace:
I20251027 21:08:00.091080 10040 rpcz_store.cc:276] 1027 21:07:58.599122 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.630762 (+ 31640us) service_pool.cc:225] Handling call
1027 21:08:00.090921 (+1460159us) ts_tablet_manager.cc:1151] Deleting tablet de2c033a064643b386ee84085cbdad64
1027 21:08:00.090928 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.091000 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1460012}
I20251027 21:08:00.091347 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 653) took 1490 ms. Trace:
I20251027 21:08:00.091406 10040 rpcz_store.cc:276] 1027 21:07:58.600638 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.630921 (+ 30283us) service_pool.cc:225] Handling call
1027 21:08:00.091264 (+1460343us) ts_tablet_manager.cc:1151] Deleting tablet 2532bf7190064443a898e47e843fd9ac
1027 21:08:00.091271 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.091334 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1460187}
I20251027 21:08:00.091673 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 650) took 1500 ms. Trace:
I20251027 21:08:00.092242 10029 tablet_bootstrap.cc:654] T 21a8fc4a69f3486e92b1a68d10faed38 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.091739 10040 rpcz_store.cc:276] 1027 21:07:58.591311 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.610610 (+ 19299us) service_pool.cc:225] Handling call
1027 21:08:00.091595 (+1480985us) ts_tablet_manager.cc:1151] Deleting tablet 7427bbc47a8643c19fb543626c6aea2a
1027 21:08:00.091602 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.091661 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1459878}
I20251027 21:08:00.094554 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 655) took 1476 ms. Trace:
I20251027 21:08:00.094635 10040 rpcz_store.cc:276] 1027 21:07:58.618067 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.631744 (+ 13677us) service_pool.cc:225] Handling call
1027 21:08:00.094466 (+1462722us) ts_tablet_manager.cc:1151] Deleting tablet fde0980aa64c4496861a662f7ed3ef75
1027 21:08:00.094474 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.094539 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1443884}
I20251027 21:08:00.100278 10041 ts_tablet_manager.cc:1929] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.100450 10041 log.cc:1199] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/f34512059f0c47fc9af7db2645f86b40
I20251027 21:08:00.100703 10665 raft_consensus.cc:493] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.100821 10665 raft_consensus.cc:515] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.101037 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 658) took 1385 ms. Trace:
I20251027 21:08:00.101109 10040 rpcz_store.cc:276] 1027 21:07:58.715462 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.715750 (+ 288us) service_pool.cc:225] Handling call
1027 21:08:00.094839 (+1379089us) ts_tablet_manager.cc:1151] Deleting tablet 2752b86151a6428c88bfaac1ff72cfb2
1027 21:08:00.094848 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.101020 (+ 6172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1378922}
I20251027 21:08:00.101382 10665 leader_election.cc:290] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:00.101735 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5f2e60caa824b5db2330b423221873c" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:00.102236 10041 ts_tablet_manager.cc:1950] T f34512059f0c47fc9af7db2645f86b40 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:00.101388 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 659) took 1310 ms. Trace:
I20251027 21:08:00.102820 10040 rpcz_store.cc:276] 1027 21:07:58.790551 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.792035 (+ 1484us) service_pool.cc:225] Handling call
1027 21:08:00.101299 (+1309264us) ts_tablet_manager.cc:1151] Deleting tablet 133e474f9133487f89c44dbfa6266ca7
1027 21:08:00.101307 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.101375 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1309028}
I20251027 21:08:00.103008 10029 tablet_bootstrap.cc:492] T 21a8fc4a69f3486e92b1a68d10faed38 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:00.103065 10040 tablet_replica.cc:333] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:00.103236 10040 raft_consensus.cc:2243] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.103384 10040 raft_consensus.cc:2272] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.103477 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet f34512059f0c47fc9af7db2645f86b40 (table default.Table_660eb77e69884fe5bd5e65b47ea91dde [id=14daa3e199fe4fa3ba2de06dd6be4330]) successfully deleted
I20251027 21:08:00.103816 10041 tablet_replica.cc:333] T 9882b234e79940088be5f604c41576b7 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:00.103806 10029 ts_tablet_manager.cc:1403] T 21a8fc4a69f3486e92b1a68d10faed38 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.004s
I20251027 21:08:00.103883 10704 tablet.cc:1722] T 7aa8c538b6c849a28a28ffe706306891 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:08:00.104106 10041 raft_consensus.cc:2243] T 9882b234e79940088be5f604c41576b7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.104286 10041 raft_consensus.cc:2272] T 9882b234e79940088be5f604c41576b7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.104537 10029 raft_consensus.cc:359] T 21a8fc4a69f3486e92b1a68d10faed38 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.104694 10029 raft_consensus.cc:385] T 21a8fc4a69f3486e92b1a68d10faed38 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.104746 10029 raft_consensus.cc:740] T 21a8fc4a69f3486e92b1a68d10faed38 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.104950 10041 ts_tablet_manager.cc:1916] T 9882b234e79940088be5f604c41576b7 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.104930 10029 consensus_queue.cc:260] T 21a8fc4a69f3486e92b1a68d10faed38 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.105934 10043 ts_tablet_manager.cc:1929] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.106033 10043 log.cc:1199] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/78884c2be75f45f881f5424d3b3f4e08
I20251027 21:08:00.106344 10043 ts_tablet_manager.cc:1950] T 78884c2be75f45f881f5424d3b3f4e08 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:00.106858 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 123e6e3748574ea581f67fff49037762 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.107237 10029 ts_tablet_manager.cc:1434] T 21a8fc4a69f3486e92b1a68d10faed38 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:08:00.107473 10029 tablet_bootstrap.cc:492] T a287439ea6c9469da5fb56d84ee68048 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:00.108577 10040 ts_tablet_manager.cc:1916] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.109314 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 679) took 1147 ms. Trace:
I20251027 21:08:00.109392 10043 rpcz_store.cc:276] 1027 21:07:58.961863 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.961924 (+ 61us) service_pool.cc:225] Handling call
1027 21:08:00.074314 (+1112390us) ts_tablet_manager.cc:1151] Deleting tablet 78884c2be75f45f881f5424d3b3f4e08
1027 21:08:00.074322 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.083781 (+ 9459us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.105847 (+ 22066us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.109299 (+ 3452us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2225,"tablet-delete.queue_time_us":1112141}
I20251027 21:08:00.109825 10043 tablet_replica.cc:333] T f1aa5f7f76cb4a1ba15f016942f17067 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:00.109951 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 78884c2be75f45f881f5424d3b3f4e08 (table default.Table_1376974f614548189208cb9cf0fd5353 [id=0eb078640c174f8fadb959306a35cada]) successfully deleted
I20251027 21:08:00.109998 10043 raft_consensus.cc:2243] T f1aa5f7f76cb4a1ba15f016942f17067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.110112 10043 raft_consensus.cc:2272] T f1aa5f7f76cb4a1ba15f016942f17067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:00.110262 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet fc2cf41a7b474f7d86a46af76cef4100 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc2cf41a7b474f7d86a46af76cef4100 already in progress: creating tablet
I20251027 21:08:00.110589 10043 ts_tablet_manager.cc:1916] T f1aa5f7f76cb4a1ba15f016942f17067 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.112979 10029 tablet_bootstrap.cc:654] T a287439ea6c9469da5fb56d84ee68048 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:00.114428 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc97e55cfbcb41ec8c6c7410bbf14fb3 consensus state has no leader
I20251027 21:08:00.117089 9304 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 7aa8c538b6c849a28a28ffe706306891 (table default.Table_b779f2756a924ab6ba1bc1702a412e06 [id=8ed572891f9b41cba8ed984911fa6b11]). Expected version 1 got 0
I20251027 21:08:00.117887 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5502390635a249d5a5b5eaf0788236fd" } new_table_name: "default.Table_84434842cc354813bc45be80de0d8322" modify_external_catalogs: true
I20251027 21:08:00.118021 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3c1a4867db0849ceb59d5900eb57eb73" } modify_external_catalogs: true
I20251027 21:08:00.119314 10043 ts_tablet_manager.cc:1929] T f1aa5f7f76cb4a1ba15f016942f17067 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.119403 10043 log.cc:1199] T f1aa5f7f76cb4a1ba15f016942f17067 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/f1aa5f7f76cb4a1ba15f016942f17067
I20251027 21:08:00.119666 10043 ts_tablet_manager.cc:1950] T f1aa5f7f76cb4a1ba15f016942f17067 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
W20251027 21:08:00.119745 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 29693cf06bc24930870dc60457126519 consensus state has no leader
I20251027 21:08:00.120095 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6d50127567904f7cab81c7a90c655c3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:00.120352 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 3103c8e5a7c744be90be097179904ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:08:00.120584 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d50127567904f7cab81c7a90c655c3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:00.121627 9710 leader_election.cc:343] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 49d9e5c5960a4534a6603875006e6266
I20251027 21:08:00.123059 10681 raft_consensus.cc:493] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.123136 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 211ac01a37f8434f95aa6a63970296fa
I20251027 21:08:00.123185 10681 raft_consensus.cc:515] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.123697 10681 leader_election.cc:290] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:00.123809 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_8b1c94c0aec244e3950e08017ed7da7d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:00.124275 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b1c94c0aec244e3950e08017ed7da7d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.125854 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet f1aa5f7f76cb4a1ba15f016942f17067 (table default.Table_3e0f707f8e2844bfaf937150b747a935 [id=d7dcec1e03494fd5a219a97ea8b1ab1d]) successfully deleted
I20251027 21:08:00.126758 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 211ac01a37f8434f95aa6a63970296fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.126874 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 583fba7c750e4ac7b768da447002c211 with delete_type TABLET_DATA_DELETED (replaced by tablet f34512059f0c47fc9af7db2645f86b40) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.126854 9923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc589852710342f7bd017193e4075303" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:00.127048 9923 raft_consensus.cc:2468] T dc589852710342f7bd017193e4075303 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:00.117038 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_aeaed62246864354b04616e1e4461644" } new_table_name: "default.Table_ecf1a7cbc64b46579e08c39312812361" modify_external_catalogs: true
W20251027 21:08:00.128901 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cf66e5a18be84f9d8469c5fd46da6f2f consensus state has no leader
I20251027 21:08:00.129278 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3bdec7059cee4e748b8e00c5ebdb800a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:00.129438 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_70172b5479b846bd884c62506727d518"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:00.129817 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bdec7059cee4e748b8e00c5ebdb800a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:00.129907 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70172b5479b846bd884c62506727d518 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.131547 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet 21a8fc4a69f3486e92b1a68d10faed38 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:00.131948 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet c1f99ca89f924b58b1416d2945a6ba40 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:00.132993 9312 master_service.cc:954] ReplaceTablet: received request to replace tablet 533f231e3ff64531ae2331409a961425 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:00.135260 9568 leader_election.cc:304] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:00.135537 10681 raft_consensus.cc:2804] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:00.135614 10681 raft_consensus.cc:493] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:00.135668 10681 raft_consensus.cc:3060] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:00.137861 9711 leader_election.cc:304] T 7f7ad06a15c04efbb701634e14904351 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:00.138271 9745 tablet_service.cc:1552] Processing DeleteTablet for tablet deb8b54e5e0f42bca4330d2160e5524a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.138278 9307 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 995284883d834bbb9ef583b736ca8402 (table default.Table_140424c161ea4581bed312cf5deb5ab5 [id=186c9cce53d74040b9799ea386fdffd7]). Expected version 4 got 0
I20251027 21:08:00.138270 10681 raft_consensus.cc:515] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.139107 10681 leader_election.cc:290] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:00.139956 10043 tablet_replica.cc:333] T 6bf9c2a78aac42528d67846dc58adc40 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:00.140112 10665 raft_consensus.cc:2749] T 7f7ad06a15c04efbb701634e14904351 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.140594 10043 raft_consensus.cc:2243] T 6bf9c2a78aac42528d67846dc58adc40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.140728 10043 raft_consensus.cc:2272] T 6bf9c2a78aac42528d67846dc58adc40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.141335 9923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc589852710342f7bd017193e4075303" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:00.141492 9923 raft_consensus.cc:3060] T dc589852710342f7bd017193e4075303 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:00.141813 9859 leader_election.cc:343] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.144224 9923 raft_consensus.cc:2468] T dc589852710342f7bd017193e4075303 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
W20251027 21:08:00.144332 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 9239f74d725649cab5ea1b0077bccef3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9239f74d725649cab5ea1b0077bccef3 already in progress: creating tablet
W20251027 21:08:00.144601 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 57de49dacafa497dad79ab96540b5888 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57de49dacafa497dad79ab96540b5888 already in progress: creating tablet
I20251027 21:08:00.145143 10043 ts_tablet_manager.cc:1916] T 6bf9c2a78aac42528d67846dc58adc40 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.145478 9568 leader_election.cc:304] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:00.145745 10681 raft_consensus.cc:2804] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:00.145875 10681 raft_consensus.cc:697] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:00.146092 10681 consensus_queue.cc:237] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.146234 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc589852710342f7bd017193e4075303" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:00.147001 9566 leader_election.cc:343] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.151192 10665 raft_consensus.cc:493] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.151309 10665 raft_consensus.cc:515] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:00.151800 10665 leader_election.cc:290] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:00.154084 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3c1a4867db0849ceb59d5900eb57eb73" } modify_external_catalogs: true
W20251027 21:08:00.155602 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93d2f7df14a74731aea84927f0b7ed45 consensus state has no leader
I20251027 21:08:00.155952 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet e1385beb1dc14ee0b42a2f4e1f66f532 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.146844 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc589852710342f7bd017193e4075303" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:00.157337 10040 ts_tablet_manager.cc:1929] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.157464 10040 log.cc:1199] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/9248f458baca409aa3505cc0db321b34
W20251027 21:08:00.157743 9566 leader_election.cc:343] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:00.158382 9710 leader_election.cc:343] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: a9cc64ad85334daa9a09538e24ebd8a5
I20251027 21:08:00.159806 10031 tablet_bootstrap.cc:654] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:00.160355 9711 leader_election.cc:343] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 49d9e5c5960a4534a6603875006e6266
I20251027 21:08:00.160467 9711 leader_election.cc:304] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
W20251027 21:08:00.160970 9711 leader_election.cc:343] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: a9cc64ad85334daa9a09538e24ebd8a5
I20251027 21:08:00.161085 9711 leader_election.cc:304] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:00.161408 10665 raft_consensus.cc:2749] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.162029 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c1f99ca89f924b58b1416d2945a6ba40
I20251027 21:08:00.162648 9311 catalog_manager.cc:6719] ReplaceTablet: tablet c1f99ca89f924b58b1416d2945a6ba40 deleted and replaced by tablet d25b493ab8ca401bb1cbe01e355d7bcd
I20251027 21:08:00.162694 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet c1f99ca89f924b58b1416d2945a6ba40 with delete_type TABLET_DATA_DELETED (replaced by tablet d25b493ab8ca401bb1cbe01e355d7bcd) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:00.163142 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9ace4188d3494a05a5678381f8d8fd17 consensus state has no leader
I20251027 21:08:00.163127 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet c1f99ca89f924b58b1416d2945a6ba40 with delete_type TABLET_DATA_DELETED (replaced by tablet d25b493ab8ca401bb1cbe01e355d7bcd) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.165000 10665 raft_consensus.cc:2749] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.167572 10040 ts_tablet_manager.cc:1950] T 9248f458baca409aa3505cc0db321b34 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:00.167892 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 661) took 1340 ms. Trace:
I20251027 21:08:00.167975 10040 rpcz_store.cc:276] 1027 21:07:58.827034 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.827449 (+ 415us) service_pool.cc:225] Handling call
1027 21:08:00.103026 (+1275577us) ts_tablet_manager.cc:1151] Deleting tablet 9248f458baca409aa3505cc0db321b34
1027 21:08:00.103033 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.143182 (+ 40149us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.157247 (+ 14065us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.167875 (+ 10628us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1269551}
W20251027 21:08:00.169013 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet b3abc5f78c4e41e29128e2d790319c75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3abc5f78c4e41e29128e2d790319c75 already in progress: creating tablet
I20251027 21:08:00.172173 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet cc4635c00e9b40c49ce04debc76a04bf
I20251027 21:08:00.172984 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet cc4635c00e9b40c49ce04debc76a04bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.173514 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet cc4635c00e9b40c49ce04debc76a04bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:00.174791 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 660) took 1373 ms. Trace:
I20251027 21:08:00.174889 10040 rpcz_store.cc:276] 1027 21:07:58.801746 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.805971 (+ 4225us) service_pool.cc:225] Handling call
1027 21:08:00.168212 (+1362241us) ts_tablet_manager.cc:1151] Deleting tablet 23673513189f47638dbbc737da0991d1
1027 21:08:00.168220 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.174769 (+ 6549us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1333253}
I20251027 21:08:00.175288 10040 tablet_replica.cc:333] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:00.175365 9307 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_b779f2756a924ab6ba1bc1702a412e06 [id=8ed572891f9b41cba8ed984911fa6b11] alter complete (version 1)
I20251027 21:08:00.175511 10040 raft_consensus.cc:2243] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.175679 10040 raft_consensus.cc:2272] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.176486 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet dc589852710342f7bd017193e4075303 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:00.176986 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet 21a8fc4a69f3486e92b1a68d10faed38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:00.177161 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet d1f24d21bb954ac9b62ce458c6a22462 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1f24d21bb954ac9b62ce458c6a22462 already in progress: creating tablet
W20251027 21:08:00.177824 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet de2c033a064643b386ee84085cbdad64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de2c033a064643b386ee84085cbdad64 already in progress: creating tablet
W20251027 21:08:00.178100 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 2532bf7190064443a898e47e843fd9ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2532bf7190064443a898e47e843fd9ac already in progress: creating tablet
W20251027 21:08:00.178364 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 7427bbc47a8643c19fb543626c6aea2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7427bbc47a8643c19fb543626c6aea2a already in progress: creating tablet
W20251027 21:08:00.178602 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet fde0980aa64c4496861a662f7ed3ef75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fde0980aa64c4496861a662f7ed3ef75 already in progress: creating tablet
I20251027 21:08:00.179692 10040 ts_tablet_manager.cc:1916] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:00.180112 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 2752b86151a6428c88bfaac1ff72cfb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2752b86151a6428c88bfaac1ff72cfb2 already in progress: creating tablet
W20251027 21:08:00.184365 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 133e474f9133487f89c44dbfa6266ca7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 133e474f9133487f89c44dbfa6266ca7 already in progress: creating tablet
I20251027 21:08:00.188706 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 94f2dbf47a5c4458bee54b26f272156e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.189472 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet adaa4f66bbab491f8030d439d85cc7d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.190649 10043 ts_tablet_manager.cc:1929] T 6bf9c2a78aac42528d67846dc58adc40 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.190762 10043 log.cc:1199] T 6bf9c2a78aac42528d67846dc58adc40 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/6bf9c2a78aac42528d67846dc58adc40
I20251027 21:08:00.191071 10043 ts_tablet_manager.cc:1950] T 6bf9c2a78aac42528d67846dc58adc40 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:00.191926 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 9248f458baca409aa3505cc0db321b34 (table default.Table_b0a525314a0d4e69aee8e2c4a291d999 [id=64df643709144959945c55df034b77a1]) successfully deleted
I20251027 21:08:00.192126 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 6bf9c2a78aac42528d67846dc58adc40 (table default.Table_e9fd9bee6a2a40d7ad92d2d31b03afce [id=af6cb904f8cd4a929be7a978f9bc2001]) successfully deleted
W20251027 21:08:00.192939 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 0820648719a34f379507efced9cc434b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0820648719a34f379507efced9cc434b already in progress: creating tablet
I20251027 21:08:00.197494 10040 ts_tablet_manager.cc:1929] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.199395 10041 ts_tablet_manager.cc:1929] T 9882b234e79940088be5f604c41576b7 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251027 21:08:00.199826 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 23673513189f47638dbbc737da0991d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23673513189f47638dbbc737da0991d1 already in progress: creating tablet
I20251027 21:08:00.202286 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 211ac01a37f8434f95aa6a63970296fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:00.202550 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 113a5d30d4b3490187681383d4fef325 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 113a5d30d4b3490187681383d4fef325 already in progress: creating tablet
W20251027 21:08:00.203150 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet d2203ee62f7e404c95c5f86b5ae85feb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2203ee62f7e404c95c5f86b5ae85feb already in progress: creating tablet
I20251027 21:08:00.203508 10041 log.cc:1199] T 9882b234e79940088be5f604c41576b7 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/9882b234e79940088be5f604c41576b7
W20251027 21:08:00.203744 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 849b3e9fc34045939a7a678fa8269d7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 849b3e9fc34045939a7a678fa8269d7f already in progress: creating tablet
I20251027 21:08:00.203825 10041 ts_tablet_manager.cc:1950] T 9882b234e79940088be5f604c41576b7 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:00.204386 10032 tablet_bootstrap.cc:492] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:00.204500 10032 ts_tablet_manager.cc:1403] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.308s user 0.013s sys 0.020s
W20251027 21:08:00.204489 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet e2ff7fc3c1fa4dbd9a5c38d3219b03fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2ff7fc3c1fa4dbd9a5c38d3219b03fa already in progress: creating tablet
I20251027 21:08:00.204753 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 9882b234e79940088be5f604c41576b7 (table default.Table_e0dee9209ca14dd98065b0384855d953 [id=92995507b7d2462f91b1a16434d3ac06]) successfully deleted
I20251027 21:08:00.205004 10043 tablet_replica.cc:333] T 9c239216cb8846f0b1aaa360ddb83321 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:00.205233 10032 raft_consensus.cc:359] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.205382 10041 tablet_replica.cc:333] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:00.205657 10041 raft_consensus.cc:2243] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.205821 10041 raft_consensus.cc:2272] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.206364 10041 ts_tablet_manager.cc:1916] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.207298 10681 raft_consensus.cc:493] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.208087 10681 raft_consensus.cc:515] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:08:00.209609 9568 leader_election.cc:343] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: ccaf5c2f8e544842880ba8ba27300ae6
I20251027 21:08:00.209780 10681 leader_election.cc:290] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:00.205425 10032 raft_consensus.cc:385] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.210186 10032 raft_consensus.cc:740] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.210237 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 0bee9d4601c94b138d68f7663769ccbe (DEFAULT_TABLE table=default.Table_a473b10ab1e94ca885768527b521172b [id=7ff01312b293439d8a51b53f5d0d4700]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.210392 10032 consensus_queue.cc:260] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.210582 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bee9d4601c94b138d68f7663769ccbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.210840 10032 ts_tablet_manager.cc:1434] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251027 21:08:00.211102 10032 tablet_bootstrap.cc:492] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:00.211812 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 750b27c74ba3473dad743d60c21912c5 (DEFAULT_TABLE table=default.Table_7fe12d37a07344b4b839c87a4b904c20 [id=71ff12fc440946e0b048072aa0c85c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.212143 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 750b27c74ba3473dad743d60c21912c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.212564 10032 tablet_bootstrap.cc:654] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.216466 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet cc4635c00e9b40c49ce04debc76a04bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.220388 10041 ts_tablet_manager.cc:1929] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.220551 10041 log.cc:1199] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/c5f2e60caa824b5db2330b423221873c
I20251027 21:08:00.220867 10041 ts_tablet_manager.cc:1950] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:00.221601 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet c5f2e60caa824b5db2330b423221873c (table default.Table_a90c80308cb345d7abc6c44d3d4c70ff [id=0e7e4ea125894e78a2162e9e8063ee33]) successfully deleted
I20251027 21:08:00.222050 10041 tablet_replica.cc:333] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:00.222318 10041 raft_consensus.cc:2243] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.222504 10041 raft_consensus.cc:2272] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:00.222856 9297 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet cc97e55cfbcb41ec8c6c7410bbf14fb3 (table default.Table_a05ec8bfd95c4f82bd16fbd66e110afc [id=e08d419154a44c54ad88bddcc5ecdbf7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet cc97e55cfbcb41ec8c6c7410bbf14fb3 consensus state has no leader
I20251027 21:08:00.223076 10041 ts_tablet_manager.cc:1916] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.226223 10032 tablet_bootstrap.cc:492] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:00.226346 10032 ts_tablet_manager.cc:1403] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20251027 21:08:00.226954 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 750b27c74ba3473dad743d60c21912c5 (DEFAULT_TABLE table=default.Table_7fe12d37a07344b4b839c87a4b904c20 [id=71ff12fc440946e0b048072aa0c85c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.227070 10032 raft_consensus.cc:359] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.227207 10032 raft_consensus.cc:385] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.227154 9601 tablet_service.cc:1505] Processing CreateTablet for tablet d715976e84754e31bfaf2cc9246342f4 (DEFAULT_TABLE table=default.Table_0c098e93345a4423888bb615af233dbe [id=be7293ee5c5b42ffbf0d1280782f00d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.227258 10032 raft_consensus.cc:740] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.227300 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 750b27c74ba3473dad743d60c21912c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.227438 10032 consensus_queue.cc:260] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.227553 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d715976e84754e31bfaf2cc9246342f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.227880 10032 ts_tablet_manager.cc:1434] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:00.228135 10032 tablet_bootstrap.cc:492] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:08:00.228305 9860 leader_election.cc:343] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.228410 9860 leader_election.cc:304] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:00.228579 9860 raft_consensus.cc:2686] T c5f2e60caa824b5db2330b423221873c P 0fbd567f929a4f6d8e4d78feef7ed678: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251027 21:08:00.229524 10032 tablet_bootstrap.cc:654] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.232249 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 211ac01a37f8434f95aa6a63970296fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.232501 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccaf5c2f8e544842880ba8ba27300ae6" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:00.233086 9566 leader_election.cc:343] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.233147 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet c1f99ca89f924b58b1416d2945a6ba40 with delete_type TABLET_DATA_DELETED (replaced by tablet d25b493ab8ca401bb1cbe01e355d7bcd) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.233206 9566 leader_election.cc:304] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:00.233531 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 750b27c74ba3473dad743d60c21912c5 (DEFAULT_TABLE table=default.Table_7fe12d37a07344b4b839c87a4b904c20 [id=71ff12fc440946e0b048072aa0c85c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.233840 10681 raft_consensus.cc:2749] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.233965 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 750b27c74ba3473dad743d60c21912c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.239071 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 0bee9d4601c94b138d68f7663769ccbe (DEFAULT_TABLE table=default.Table_a473b10ab1e94ca885768527b521172b [id=7ff01312b293439d8a51b53f5d0d4700]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.239432 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bee9d4601c94b138d68f7663769ccbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.241668 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 0bee9d4601c94b138d68f7663769ccbe (DEFAULT_TABLE table=default.Table_a473b10ab1e94ca885768527b521172b [id=7ff01312b293439d8a51b53f5d0d4700]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.244001 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 83d6f067aef343ddaaa29b1a85e7f328 (DEFAULT_TABLE table=default.Table_b98328c296ab4b6ab486c3477c221544 [id=a14c6e391bd04f2fbb1461b1696302ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.244397 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83d6f067aef343ddaaa29b1a85e7f328. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.244607 10032 tablet_bootstrap.cc:492] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:00.244730 10032 ts_tablet_manager.cc:1403] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.003s
I20251027 21:08:00.244812 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 0166ba1e634747cd866fabc4e9acf1b5 (DEFAULT_TABLE table=default.Table_ef8a1ea1e15249948b532d0bd39d5bc8 [id=12981f0a19d2449f88307ec8dbf86032]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.245165 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0166ba1e634747cd866fabc4e9acf1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.248839 10040 log.cc:1199] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/779b7f63b0fa4f9ab47cf5799b3e085d
I20251027 21:08:00.249709 10040 ts_tablet_manager.cc:1950] T 779b7f63b0fa4f9ab47cf5799b3e085d P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:00.250077 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 662) took 1422 ms. Trace:
I20251027 21:08:00.250336 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bee9d4601c94b138d68f7663769ccbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.250200 10040 rpcz_store.cc:276] 1027 21:07:58.827182 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.827629 (+ 447us) service_pool.cc:225] Handling call
1027 21:08:00.175234 (+1347605us) ts_tablet_manager.cc:1151] Deleting tablet 779b7f63b0fa4f9ab47cf5799b3e085d
1027 21:08:00.175246 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.183347 (+ 8101us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.197414 (+ 14067us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.250016 (+ 52602us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1340058}
I20251027 21:08:00.250624 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 663) took 1396 ms. Trace:
I20251027 21:08:00.250733 10040 rpcz_store.cc:276] 1027 21:07:58.854132 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.855347 (+ 1215us) service_pool.cc:225] Handling call
1027 21:08:00.250472 (+1395125us) ts_tablet_manager.cc:1151] Deleting tablet a4d26f73b8534f08bcaa386d4de75c01
1027 21:08:00.250479 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.250574 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1392233}
I20251027 21:08:00.251075 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 664) took 1384 ms. Trace:
I20251027 21:08:00.251157 10040 rpcz_store.cc:276] 1027 21:07:58.866489 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.866539 (+ 50us) service_pool.cc:225] Handling call
1027 21:08:00.250944 (+1384405us) ts_tablet_manager.cc:1151] Deleting tablet 8b30052ff19744a9bf4bfaaa1bc77b36
1027 21:08:00.250952 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.251031 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1384220}
I20251027 21:08:00.251418 10040 tablet_replica.cc:333] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:00.251600 10040 raft_consensus.cc:2243] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.251773 10040 raft_consensus.cc:2272] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.252121 9905 tablet_service.cc:1505] Processing CreateTablet for tablet d715976e84754e31bfaf2cc9246342f4 (DEFAULT_TABLE table=default.Table_0c098e93345a4423888bb615af233dbe [id=be7293ee5c5b42ffbf0d1280782f00d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.252331 10040 ts_tablet_manager.cc:1916] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.253926 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d715976e84754e31bfaf2cc9246342f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.254566 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 779b7f63b0fa4f9ab47cf5799b3e085d (table default.Table_d98a9c0b61d649f2945cb6c199f70a64 [id=6c62d93feb804d6aa1d442f8458b7ef8]) successfully deleted
W20251027 21:08:00.254938 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet a4d26f73b8534f08bcaa386d4de75c01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4d26f73b8534f08bcaa386d4de75c01 already in progress: creating tablet
I20251027 21:08:00.255017 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 83d6f067aef343ddaaa29b1a85e7f328 (DEFAULT_TABLE table=default.Table_b98328c296ab4b6ab486c3477c221544 [id=a14c6e391bd04f2fbb1461b1696302ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.255416 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83d6f067aef343ddaaa29b1a85e7f328. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:00.255550 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 8b30052ff19744a9bf4bfaaa1bc77b36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b30052ff19744a9bf4bfaaa1bc77b36 already in progress: creating tablet
I20251027 21:08:00.256496 10043 raft_consensus.cc:2243] T 9c239216cb8846f0b1aaa360ddb83321 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.256512 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 0166ba1e634747cd866fabc4e9acf1b5 (DEFAULT_TABLE table=default.Table_ef8a1ea1e15249948b532d0bd39d5bc8 [id=12981f0a19d2449f88307ec8dbf86032]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.256858 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0166ba1e634747cd866fabc4e9acf1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.257184 10043 raft_consensus.cc:2272] T 9c239216cb8846f0b1aaa360ddb83321 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.258757 10043 ts_tablet_manager.cc:1916] T 9c239216cb8846f0b1aaa360ddb83321 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.258757 9750 tablet_service.cc:1505] Processing CreateTablet for tablet d715976e84754e31bfaf2cc9246342f4 (DEFAULT_TABLE table=default.Table_0c098e93345a4423888bb615af233dbe [id=be7293ee5c5b42ffbf0d1280782f00d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.258816 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7a9e545bc9374c8b986687268e421690" } modify_external_catalogs: true
I20251027 21:08:00.259002 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7a9e545bc9374c8b986687268e421690" } modify_external_catalogs: true
I20251027 21:08:00.259073 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d715976e84754e31bfaf2cc9246342f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.260215 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e763bc3cda90472897f0f798424199b4" } modify_external_catalogs: true
I20251027 21:08:00.260370 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e763bc3cda90472897f0f798424199b4" } modify_external_catalogs: true
I20251027 21:08:00.260466 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 8649a3b611cc48ff8832c0fdeb128efe (DEFAULT_TABLE table=default.Table_190963122a7245dfa310ab8c69edf80f [id=b25a76e43b6c4acb89764bab21dec297]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.260743 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8649a3b611cc48ff8832c0fdeb128efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.263015 10031 tablet_bootstrap.cc:492] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:00.263132 10031 ts_tablet_manager.cc:1403] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.264s user 0.006s sys 0.004s
I20251027 21:08:00.263890 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 8649a3b611cc48ff8832c0fdeb128efe (DEFAULT_TABLE table=default.Table_190963122a7245dfa310ab8c69edf80f [id=b25a76e43b6c4acb89764bab21dec297]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.263898 10031 raft_consensus.cc:359] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:00.264053 10031 raft_consensus.cc:385] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.264129 10031 raft_consensus.cc:740] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.264217 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8649a3b611cc48ff8832c0fdeb128efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.264318 10031 consensus_queue.cc:260] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:00.265045 10031 ts_tablet_manager.cc:1434] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:00.265307 10031 tablet_bootstrap.cc:492] T 995284883d834bbb9ef583b736ca8402 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:00.266878 10031 tablet_bootstrap.cc:654] T 995284883d834bbb9ef583b736ca8402 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.270947 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 933b33aac5354778a5866d340fd34db4
I20251027 21:08:00.272046 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet 933b33aac5354778a5866d340fd34db4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.273204 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 4ba13a89a33549af921d3b55023faf15
I20251027 21:08:00.273707 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 0166ba1e634747cd866fabc4e9acf1b5 (DEFAULT_TABLE table=default.Table_ef8a1ea1e15249948b532d0bd39d5bc8 [id=12981f0a19d2449f88307ec8dbf86032]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.274058 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0166ba1e634747cd866fabc4e9acf1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.274017 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ba13a89a33549af921d3b55023faf15 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.274499 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ba13a89a33549af921d3b55023faf15 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:00.275076 9611 tablet_service.cc:1552] Processing DeleteTablet for tablet 933b33aac5354778a5866d340fd34db4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:00.276449 10681 raft_consensus.cc:493] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.276561 10681 raft_consensus.cc:515] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.276659 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 4a621d5246f14997a2f2952ab640efc8 (DEFAULT_TABLE table=default.Table_ab6ea9b86ffd4160a20af2687c8ef0ae [id=c605be7b6a4e46ab82c0acca63fef426]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.276973 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a621d5246f14997a2f2952ab640efc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.277128 10681 leader_election.cc:290] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:00.277648 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0968c2f25b64f6685f38a999ee9d60e" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:00.278057 9566 leader_election.cc:343] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:00.278404 9568 leader_election.cc:343] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: e0968c2f25b64f6685f38a999ee9d60e
I20251027 21:08:00.278489 9568 leader_election.cc:304] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:00.278818 10681 raft_consensus.cc:2749] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.279743 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9b0fd27b43c842008090999fe7da9233" } modify_external_catalogs: true
I20251027 21:08:00.280012 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9b0fd27b43c842008090999fe7da9233" } modify_external_catalogs: true
I20251027 21:08:00.280711 10718 raft_consensus.cc:493] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.280854 10718 raft_consensus.cc:515] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:00.281657 10718 leader_election.cc:290] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
W20251027 21:08:00.282209 9710 leader_election.cc:343] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 94e94cd50c91429d8da9804149501c86
I20251027 21:08:00.282359 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 83d6f067aef343ddaaa29b1a85e7f328 (DEFAULT_TABLE table=default.Table_b98328c296ab4b6ab486c3477c221544 [id=a14c6e391bd04f2fbb1461b1696302ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.282673 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83d6f067aef343ddaaa29b1a85e7f328. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.282744 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 8649a3b611cc48ff8832c0fdeb128efe (DEFAULT_TABLE table=default.Table_190963122a7245dfa310ab8c69edf80f [id=b25a76e43b6c4acb89764bab21dec297]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.283061 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8649a3b611cc48ff8832c0fdeb128efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.285718 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet a8c46c2a01f946aea8b85875115f7f37
I20251027 21:08:00.286808 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet a8c46c2a01f946aea8b85875115f7f37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.286970 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet a8c46c2a01f946aea8b85875115f7f37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:00.290445 9860 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.65:44531, user_credentials={real_user=slave}} blocked reactor thread for 61276.6us
I20251027 21:08:00.290522 10032 raft_consensus.cc:359] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.290681 10032 raft_consensus.cc:385] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.290743 10032 raft_consensus.cc:740] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.290931 10032 consensus_queue.cc:260] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.291661 10032 ts_tablet_manager.cc:1434] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.047s user 0.001s sys 0.001s
I20251027 21:08:00.292292 10032 tablet_bootstrap.cc:492] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:00.294875 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 4a621d5246f14997a2f2952ab640efc8 (DEFAULT_TABLE table=default.Table_ab6ea9b86ffd4160a20af2687c8ef0ae [id=c605be7b6a4e46ab82c0acca63fef426]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.295372 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a621d5246f14997a2f2952ab640efc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.301295 10032 tablet_bootstrap.cc:654] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.301409 9754 tablet_service.cc:1552] Processing DeleteTablet for tablet 6db0f31c12d14e66a4ec7fc722d62c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.302471 9754 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ba13a89a33549af921d3b55023faf15 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.303202 9307 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 45568792d7954b498581ce69fb611948 (table default.Table_3e3e0c17a3bf452fbb217c6e3ee8bdcb [id=2eb093cd58e643a58b14e564d5914fbc]). Expected version 1 got 0
I20251027 21:08:00.303324 9754 tablet_service.cc:1552] Processing DeleteTablet for tablet a8c46c2a01f946aea8b85875115f7f37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.306244 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 4a621d5246f14997a2f2952ab640efc8 (DEFAULT_TABLE table=default.Table_ab6ea9b86ffd4160a20af2687c8ef0ae [id=c605be7b6a4e46ab82c0acca63fef426]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.306592 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a621d5246f14997a2f2952ab640efc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.307265 9310 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 45568792d7954b498581ce69fb611948 (table default.Table_3e3e0c17a3bf452fbb217c6e3ee8bdcb [id=2eb093cd58e643a58b14e564d5914fbc]). Expected version 1 got 0
I20251027 21:08:00.311583 10723 tablet.cc:1722] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251027 21:08:00.314282 9896 tablet_service.cc:1552] Processing DeleteTablet for tablet a9bf3a093be8498c9dd5511b53436b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.317154 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 9e23fc39bdff46889d74daf52aaf4096 (DEFAULT_TABLE table=default.Table_011a37833a9142ad95940903f3185dfb [id=955e94b7e61e48828a95b639b998a445]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.319311 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 9e23fc39bdff46889d74daf52aaf4096 (DEFAULT_TABLE table=default.Table_011a37833a9142ad95940903f3185dfb [id=955e94b7e61e48828a95b639b998a445]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.319698 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e23fc39bdff46889d74daf52aaf4096. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.322405 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_ee755fc680144410828dd30c2c19b0bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:00.322628 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9eb69e3772794bcf8f0662a43bed4bd1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:00.322878 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1e71059726554fec9b06fa78154e9ad5" } new_table_name: "default.Table_2c2ec3f9d23142c9b8c2d66d1c979525" modify_external_catalogs: true
W20251027 21:08:00.322952 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee755fc680144410828dd30c2c19b0bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:00.323062 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9eb69e3772794bcf8f0662a43bed4bd1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.323398 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f0e8c68b9f1243bfb3584cca5b5f7002" } new_table_name: "default.Table_423b90a6cbf94715b7d8344c550bb10f" modify_external_catalogs: true
W20251027 21:08:00.328385 9711 leader_election.cc:343] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 94e94cd50c91429d8da9804149501c86
I20251027 21:08:00.328536 9711 leader_election.cc:304] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:00.328805 10718 raft_consensus.cc:2749] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.332433 10031 tablet_bootstrap.cc:492] T 995284883d834bbb9ef583b736ca8402 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:00.332357 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 9e23fc39bdff46889d74daf52aaf4096 (DEFAULT_TABLE table=default.Table_011a37833a9142ad95940903f3185dfb [id=955e94b7e61e48828a95b639b998a445]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.332891 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e23fc39bdff46889d74daf52aaf4096. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.338171 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3d031fe204804a95b5bc07e8b54573d8" } modify_external_catalogs: true
I20251027 21:08:00.338315 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9406473db81e445ab8be74949a2545c3" } new_table_name: "default.Table_47e6de266b544e31a4d781f8c9377717" modify_external_catalogs: true
I20251027 21:08:00.338485 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3d031fe204804a95b5bc07e8b54573d8" } modify_external_catalogs: true
I20251027 21:08:00.339033 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a47950555ce34e4aabcf33ae10344421"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:00.339610 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a47950555ce34e4aabcf33ae10344421 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.339648 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c9eb5134bf16410ca17a6600553250d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:00.340142 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9eb5134bf16410ca17a6600553250d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.341179 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_8b0c128ff09445d3b6b5ece094733305"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:00.341636 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b0c128ff09445d3b6b5ece094733305 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:00.342335 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f901626832f4adb948ccb65f8573e86 consensus state has no leader
I20251027 21:08:00.342881 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_29369a432b8c4b108720c969d77f25b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:00.343446 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29369a432b8c4b108720c969d77f25b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.345330 9594 tablet_service.cc:1505] Processing CreateTablet for tablet bdd382c4d376484cb7767d30681f966f (DEFAULT_TABLE table=default.Table_385f764be86a472d8360fd95634bdcde [id=dd7a81c891d1449f8a3443184fc1aa2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.347095 9899 tablet_service.cc:1505] Processing CreateTablet for tablet bdd382c4d376484cb7767d30681f966f (DEFAULT_TABLE table=default.Table_385f764be86a472d8360fd95634bdcde [id=dd7a81c891d1449f8a3443184fc1aa2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.347481 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdd382c4d376484cb7767d30681f966f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.353152 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 4da0bb391aba49cbaeb72983553362b2 (DEFAULT_TABLE table=default.Table_c51243cadc5c403d84acc583c03edaba [id=310d5d0fd4294104b481db4e2a7f67f9]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:00.353430 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a6edd6a5c4434f0497667b1a2771f885 consensus state has no leader
I20251027 21:08:00.353463 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4da0bb391aba49cbaeb72983553362b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.354028 10029 tablet_bootstrap.cc:492] T a287439ea6c9469da5fb56d84ee68048 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:00.354175 10029 ts_tablet_manager.cc:1403] T a287439ea6c9469da5fb56d84ee68048 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.247s user 0.008s sys 0.012s
I20251027 21:08:00.354912 10029 raft_consensus.cc:359] T a287439ea6c9469da5fb56d84ee68048 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.355060 10029 raft_consensus.cc:385] T a287439ea6c9469da5fb56d84ee68048 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.355105 10029 raft_consensus.cc:740] T a287439ea6c9469da5fb56d84ee68048 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.355289 10029 consensus_queue.cc:260] T a287439ea6c9469da5fb56d84ee68048 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.355721 10029 ts_tablet_manager.cc:1434] T a287439ea6c9469da5fb56d84ee68048 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:00.356899 10029 tablet_bootstrap.cc:492] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:00.358749 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ab6ea9b86ffd4160a20af2687c8ef0ae" } new_table_name: "default.Table_6bc36020327e4d289973d93a0c1def37" modify_external_catalogs: true
I20251027 21:08:00.359133 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_97cd866055604dbca95060d43b29dc75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:00.360518 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97cd866055604dbca95060d43b29dc75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.361932 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e23fc39bdff46889d74daf52aaf4096. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.362426 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdd382c4d376484cb7767d30681f966f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.363888 10703 tablet.cc:1722] T 995284883d834bbb9ef583b736ca8402 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251027 21:08:00.364611 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b221c4f03f1e49d0aab44626822ea9ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:00.365047 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b221c4f03f1e49d0aab44626822ea9ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:00.371888 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3b2c479d11794a6a99b5e05d3954ad54 consensus state has no leader
I20251027 21:08:00.372262 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e18ff5a3d21f467db8e09554b370ef9d" } new_table_name: "default.Table_fd331f3cfc07425082d58ab5af3c1094" modify_external_catalogs: true
I20251027 21:08:00.373404 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_ec8494b7496d4d08828dce88b99c18f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:00.373922 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec8494b7496d4d08828dce88b99c18f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.375479 10029 tablet_bootstrap.cc:654] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.376317 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e9b8f76c33444bcb9b488af65e4d3e6b
I20251027 21:08:00.376641 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_573e41ad7d2845c8a4af93a93ed20525" } modify_external_catalogs: true
I20251027 21:08:00.377377 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_573e41ad7d2845c8a4af93a93ed20525" } modify_external_catalogs: true
I20251027 21:08:00.378644 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_feecf5aa70d7443293e8b3af119d90e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:00.379438 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_feecf5aa70d7443293e8b3af119d90e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.380272 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet e9b8f76c33444bcb9b488af65e4d3e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.380559 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet e9b8f76c33444bcb9b488af65e4d3e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:00.381845 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4a621d5246f14997a2f2952ab640efc8 consensus state has no leader
I20251027 21:08:00.383760 10032 tablet_bootstrap.cc:492] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:00.383889 10032 ts_tablet_manager.cc:1403] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.092s user 0.010s sys 0.014s
I20251027 21:08:00.384756 10032 raft_consensus.cc:359] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.384932 10032 raft_consensus.cc:385] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.384985 10032 raft_consensus.cc:740] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.386821 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 4da0bb391aba49cbaeb72983553362b2 (DEFAULT_TABLE table=default.Table_c51243cadc5c403d84acc583c03edaba [id=310d5d0fd4294104b481db4e2a7f67f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.387207 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4da0bb391aba49cbaeb72983553362b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.388016 10718 raft_consensus.cc:493] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.388175 10718 raft_consensus.cc:515] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.388787 10718 leader_election.cc:290] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:00.392704 10681 raft_consensus.cc:493] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.392840 10681 raft_consensus.cc:515] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.393136 10041 ts_tablet_manager.cc:1929] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.393242 10041 log.cc:1199] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/d52d71f4976143e4a514350621ebe309
I20251027 21:08:00.393359 10681 leader_election.cc:290] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:00.393529 10041 ts_tablet_manager.cc:1950] T d52d71f4976143e4a514350621ebe309 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:00.393934 10032 consensus_queue.cc:260] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:00.394435 9568 leader_election.cc:343] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: e841c54a94d649c19b639f5021f150cd
I20251027 21:08:00.394577 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 704) took 1086 ms. Trace:
W20251027 21:08:00.394676 9296 consensus_peers.cc:597] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 -> Peer 6e86ca5e83dd4df5b72f045e7b787e81 (127.9.14.124:45071): Couldn't send request to peer 6e86ca5e83dd4df5b72f045e7b787e81. Status: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251027 21:08:00.394723 10041 rpcz_store.cc:276] 1027 21:07:59.307745 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.307805 (+ 60us) service_pool.cc:225] Handling call
1027 21:08:00.221888 (+914083us) ts_tablet_manager.cc:1151] Deleting tablet d52d71f4976143e4a514350621ebe309
1027 21:08:00.222010 (+ 122us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.241809 (+ 19799us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.393036 (+151227us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.394560 (+ 1524us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":913910}
I20251027 21:08:00.394641 10032 ts_tablet_manager.cc:1434] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251027 21:08:00.395416 10032 tablet_bootstrap.cc:492] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:00.395473 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet d52d71f4976143e4a514350621ebe309 (table default.Table_c305a5155cce4754bcbc792f3900b690 [id=82b0b6e6d2e145a7870a13ed5b407b47]) successfully deleted
I20251027 21:08:00.396045 10041 tablet_replica.cc:333] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:00.396307 10041 raft_consensus.cc:2243] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.396512 10041 raft_consensus.cc:2272] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.397011 10032 tablet_bootstrap.cc:654] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.399468 9304 master_service.cc:954] ReplaceTablet: received request to replace tablet 7c6c2292ad0e4066958897fff01370af from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:00.399752 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 933b33aac5354778a5866d340fd34db4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.403008 9749 tablet_service.cc:1505] Processing CreateTablet for tablet bdd382c4d376484cb7767d30681f966f (DEFAULT_TABLE table=default.Table_385f764be86a472d8360fd95634bdcde [id=dd7a81c891d1449f8a3443184fc1aa2f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:00.403442 9710 leader_election.cc:343] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 77797eea3e9c4f8099dc9d58f116200b
I20251027 21:08:00.404469 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e841c54a94d649c19b639f5021f150cd" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:00.404515 9307 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_140424c161ea4581bed312cf5deb5ab5 [id=186c9cce53d74040b9799ea386fdffd7] alter complete (version 4)
W20251027 21:08:00.405057 9566 leader_election.cc:343] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.405170 9566 leader_election.cc:304] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:00.405689 10718 raft_consensus.cc:493] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.405861 10031 ts_tablet_manager.cc:1403] T 995284883d834bbb9ef583b736ca8402 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.000s
I20251027 21:08:00.406472 10718 raft_consensus.cc:515] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:00.406572 10681 raft_consensus.cc:2749] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.407071 10718 leader_election.cc:290] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
W20251027 21:08:00.407902 9710 leader_election.cc:343] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 583fba7c750e4ac7b768da447002c211
I20251027 21:08:00.412429 10031 raft_consensus.cc:359] T 995284883d834bbb9ef583b736ca8402 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:00.415045 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet e9b8f76c33444bcb9b488af65e4d3e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:08:00.416023 9711 leader_election.cc:343] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 77797eea3e9c4f8099dc9d58f116200b
I20251027 21:08:00.416172 9711 leader_election.cc:304] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:00.416429 10718 raft_consensus.cc:2749] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.416589 10043 ts_tablet_manager.cc:1929] T 9c239216cb8846f0b1aaa360ddb83321 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251027 21:08:00.416914 9711 leader_election.cc:343] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 583fba7c750e4ac7b768da447002c211
I20251027 21:08:00.417032 9711 leader_election.cc:304] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:00.417098 10043 log.cc:1199] T 9c239216cb8846f0b1aaa360ddb83321 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/9c239216cb8846f0b1aaa360ddb83321
I20251027 21:08:00.417277 10718 raft_consensus.cc:2749] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.417625 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 4da0bb391aba49cbaeb72983553362b2 (DEFAULT_TABLE table=default.Table_c51243cadc5c403d84acc583c03edaba [id=310d5d0fd4294104b481db4e2a7f67f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.418170 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdd382c4d376484cb7767d30681f966f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.418651 10043 ts_tablet_manager.cc:1950] T 9c239216cb8846f0b1aaa360ddb83321 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:00.418972 9309 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 995284883d834bbb9ef583b736ca8402 (table default.Table_140424c161ea4581bed312cf5deb5ab5 [id=186c9cce53d74040b9799ea386fdffd7]). Expected version 4 got 0
I20251027 21:08:00.424714 10031 raft_consensus.cc:385] T 995284883d834bbb9ef583b736ca8402 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.424839 10031 raft_consensus.cc:740] T 995284883d834bbb9ef583b736ca8402 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.425168 10031 consensus_queue.cc:260] T 995284883d834bbb9ef583b736ca8402 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:00.425552 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 648f506ec87c4fb8a3f6d4591873dbb8 with delete_type TABLET_DATA_DELETED (replaced by tablet 96a680e8f16343b1908829a67e960b60) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.426380 10031 ts_tablet_manager.cc:1434] T 995284883d834bbb9ef583b736ca8402 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.020s user 0.002s sys 0.001s
I20251027 21:08:00.426635 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4da0bb391aba49cbaeb72983553362b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.426877 10043 tablet_replica.cc:333] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:00.427050 10043 raft_consensus.cc:2243] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.427177 10043 raft_consensus.cc:2272] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.427627 10043 ts_tablet_manager.cc:1916] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.397030 10041 ts_tablet_manager.cc:1916] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.430567 10031 tablet_bootstrap.cc:492] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:00.432192 10031 tablet_bootstrap.cc:654] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:00.433714 9278 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 129638us
I20251027 21:08:00.433871 10043 ts_tablet_manager.cc:1929] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.433972 10043 log.cc:1199] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/e841c54a94d649c19b639f5021f150cd
I20251027 21:08:00.435026 9303 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 995284883d834bbb9ef583b736ca8402 (table default.Table_140424c161ea4581bed312cf5deb5ab5 [id=186c9cce53d74040b9799ea386fdffd7]). Expected version 4 got 0
I20251027 21:08:00.440122 9303 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 995284883d834bbb9ef583b736ca8402 (table default.Table_140424c161ea4581bed312cf5deb5ab5 [id=186c9cce53d74040b9799ea386fdffd7]). Expected version 4 got 0
I20251027 21:08:00.440886 10040 ts_tablet_manager.cc:1929] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.441063 10040 log.cc:1199] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/533f231e3ff64531ae2331409a961425
I20251027 21:08:00.441151 10043 ts_tablet_manager.cc:1950] T e841c54a94d649c19b639f5021f150cd P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:00.441473 10040 ts_tablet_manager.cc:1950] T 533f231e3ff64531ae2331409a961425 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:00.441773 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 665) took 1570 ms. Trace:
I20251027 21:08:00.441885 10040 rpcz_store.cc:276] 1027 21:07:58.871036 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.871191 (+ 155us) service_pool.cc:225] Handling call
1027 21:08:00.251349 (+1380158us) ts_tablet_manager.cc:1151] Deleting tablet 533f231e3ff64531ae2331409a961425
1027 21:08:00.251356 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.385377 (+134021us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.410651 (+ 25274us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.441735 (+ 31084us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1379956}
I20251027 21:08:00.442315 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 533f231e3ff64531ae2331409a961425 (table default.Table_913666abca6044f9aacb48e76118aac5 [id=b45a015e747e4f0db54899a28ab97a7c]) successfully deleted
I20251027 21:08:00.442356 10043 tablet_replica.cc:333] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:00.442512 10043 raft_consensus.cc:2243] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.442705 10043 raft_consensus.cc:2272] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.442795 10029 tablet_bootstrap.cc:492] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:00.442934 10029 ts_tablet_manager.cc:1403] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.000s
I20251027 21:08:00.443208 10043 ts_tablet_manager.cc:1916] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.443641 10029 raft_consensus.cc:359] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.443861 10029 raft_consensus.cc:385] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.443953 10029 raft_consensus.cc:740] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.444232 10029 consensus_queue.cc:260] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.444711 10029 ts_tablet_manager.cc:1434] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:00.456724 10029 tablet_bootstrap.cc:492] T f22f013b25074d4db9c7f13caf5b9b42 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:00.458827 10029 tablet_bootstrap.cc:654] T f22f013b25074d4db9c7f13caf5b9b42 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.459188 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 666) took 1557 ms. Trace:
I20251027 21:08:00.459404 10040 rpcz_store.cc:276] 1027 21:07:58.901390 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.901441 (+ 51us) service_pool.cc:225] Handling call
1027 21:08:00.458973 (+1557532us) ts_tablet_manager.cc:1151] Deleting tablet 38f2133010b14aad88b550fb12c966a5
1027 21:08:00.458989 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.459143 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1540504}
W20251027 21:08:00.459918 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 38f2133010b14aad88b550fb12c966a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38f2133010b14aad88b550fb12c966a5 already in progress: creating tablet
I20251027 21:08:00.460134 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 667) took 1547 ms. Trace:
I20251027 21:08:00.460341 10040 rpcz_store.cc:276] 1027 21:07:58.912402 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.912684 (+ 282us) service_pool.cc:225] Handling call
1027 21:08:00.459905 (+1547221us) ts_tablet_manager.cc:1151] Deleting tablet 4d88aacbf47144cb837b99d68f6331e2
1027 21:08:00.459916 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.460093 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1546999}
W20251027 21:08:00.460794 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 4d88aacbf47144cb837b99d68f6331e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d88aacbf47144cb837b99d68f6331e2 already in progress: creating tablet
W20251027 21:08:00.463109 9298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.125:46313, user_credentials={real_user=slave}} blocked reactor thread for 54689us
W20251027 21:08:00.463238 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet abd3b8e71d3041b4b9e5f6663841bfcf consensus state has no leader
I20251027 21:08:00.463584 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 9c239216cb8846f0b1aaa360ddb83321 (table default.Table_2d758ba690f8478ba7f983e8fd770219 [id=849a0760791548a38aceb18d0533f5f4]) successfully deleted
W20251027 21:08:00.463771 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet c4996d8fae094db8a96599b9057f1649 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4996d8fae094db8a96599b9057f1649 already in progress: creating tablet
W20251027 21:08:00.463961 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 83a040332f994055a9ecf4c66720b7e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83a040332f994055a9ecf4c66720b7e1 already in progress: creating tablet
I20251027 21:08:00.464114 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet e841c54a94d649c19b639f5021f150cd (table default.Table_5cdd921f5bdf47bab7ec19ac00c6292f [id=d392262a11e24cf0991bc200475b0ed6]) successfully deleted
I20251027 21:08:00.466187 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c25be263facd42c5bb6fa421160fbef0
I20251027 21:08:00.467219 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet c25be263facd42c5bb6fa421160fbef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.467799 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 7c6c2292ad0e4066958897fff01370af
I20251027 21:08:00.468284 9304 catalog_manager.cc:6719] ReplaceTablet: tablet 7c6c2292ad0e4066958897fff01370af deleted and replaced by tablet aeae07c3906145149cea31e5a7c466fe
I20251027 21:08:00.468665 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c6c2292ad0e4066958897fff01370af with delete_type TABLET_DATA_DELETED (replaced by tablet aeae07c3906145149cea31e5a7c466fe) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.470675 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c6c2292ad0e4066958897fff01370af with delete_type TABLET_DATA_DELETED (replaced by tablet aeae07c3906145149cea31e5a7c466fe) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.472208 10718 raft_consensus.cc:493] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.472357 10718 raft_consensus.cc:515] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:00.473012 10718 leader_election.cc:290] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:00.473801 9923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d94c695320f24a7f9776c493e462ed24" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:00.475838 10031 tablet_bootstrap.cc:492] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:00.476017 10031 ts_tablet_manager.cc:1403] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.046s user 0.001s sys 0.001s
I20251027 21:08:00.478076 10031 raft_consensus.cc:359] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.478331 10031 raft_consensus.cc:385] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.478605 10031 raft_consensus.cc:740] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.478912 10031 consensus_queue.cc:260] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.480406 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_63950b7e1b8c4f3a989b8f14a2409f78" } modify_external_catalogs: true
I20251027 21:08:00.480624 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9c8bc163f60741c3bf7fed269c243545" } modify_external_catalogs: true
I20251027 21:08:00.480792 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9c8bc163f60741c3bf7fed269c243545" } modify_external_catalogs: true
I20251027 21:08:00.480631 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_63950b7e1b8c4f3a989b8f14a2409f78" } modify_external_catalogs: true
I20251027 21:08:00.481745 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet c25be263facd42c5bb6fa421160fbef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.483572 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c239216cb8846f0b1aaa360ddb83321 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.484349 10031 ts_tablet_manager.cc:1434] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20251027 21:08:00.487537 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2088d599bbe444a18dee497154bb4379" } modify_external_catalogs: true
I20251027 21:08:00.487710 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2088d599bbe444a18dee497154bb4379" } modify_external_catalogs: true
I20251027 21:08:00.488476 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 668) took 1567 ms. Trace:
I20251027 21:08:00.488797 10040 rpcz_store.cc:276] 1027 21:07:58.920943 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.920990 (+ 47us) service_pool.cc:225] Handling call
1027 21:08:00.460795 (+1539805us) ts_tablet_manager.cc:1151] Deleting tablet 6210d3d10fc44b9e9e2b0780883186ce
1027 21:08:00.460807 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.488431 (+ 27624us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1539600}
W20251027 21:08:00.489225 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 6210d3d10fc44b9e9e2b0780883186ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6210d3d10fc44b9e9e2b0780883186ce already in progress: creating tablet
I20251027 21:08:00.498715 9769 raft_consensus.cc:3060] T 7f7ad06a15c04efbb701634e14904351 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:00.499816 10041 ts_tablet_manager.cc:1929] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.499903 10041 log.cc:1199] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/abc7458fcfb94985acf76d77c0292b08
I20251027 21:08:00.500175 10041 ts_tablet_manager.cc:1950] T abc7458fcfb94985acf76d77c0292b08 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:00.500841 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 705) took 1151 ms. Trace:
I20251027 21:08:00.500912 10041 rpcz_store.cc:276] 1027 21:07:59.349653 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.349831 (+ 178us) service_pool.cc:225] Handling call
1027 21:08:00.395994 (+1046163us) ts_tablet_manager.cc:1151] Deleting tablet abc7458fcfb94985acf76d77c0292b08
1027 21:08:00.396004 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.477908 (+ 81904us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.499721 (+ 21813us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.500824 (+ 1103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1045960}
I20251027 21:08:00.501332 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet abc7458fcfb94985acf76d77c0292b08 (table default.Table_9c027ff744824da9b9cac5574017e7fd [id=40a6848701284864b34f727da289347b]) successfully deleted
I20251027 21:08:00.506130 10031 tablet_bootstrap.cc:492] T 78884c2be75f45f881f5424d3b3f4e08 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:00.467306 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet a287439ea6c9469da5fb56d84ee68048 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:00.467429 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f1129eecfb54448965222269d9c32b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:00.473517 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet c25be263facd42c5bb6fa421160fbef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:00.473604 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d94c695320f24a7f9776c493e462ed24" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:00.473664 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c6c2292ad0e4066958897fff01370af with delete_type TABLET_DATA_DELETED (replaced by tablet aeae07c3906145149cea31e5a7c466fe) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:00.485296 10043 ts_tablet_manager.cc:1929] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.507349 10043 log.cc:1199] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/e0968c2f25b64f6685f38a999ee9d60e
I20251027 21:08:00.507697 10043 ts_tablet_manager.cc:1950] T e0968c2f25b64f6685f38a999ee9d60e P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:00.508729 10043 tablet_replica.cc:333] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:00.508893 10043 raft_consensus.cc:2243] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.509009 10043 raft_consensus.cc:2272] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.509361 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 708) took 1019 ms. Trace:
I20251027 21:08:00.509478 10043 ts_tablet_manager.cc:1916] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.509431 10041 rpcz_store.cc:276] 1027 21:07:59.489737 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.489983 (+ 246us) service_pool.cc:225] Handling call
1027 21:08:00.509243 (+1019260us) ts_tablet_manager.cc:1151] Deleting tablet 0820648719a34f379507efced9cc434b
1027 21:08:00.509253 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.509343 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1018986}
W20251027 21:08:00.509799 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 0820648719a34f379507efced9cc434b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0820648719a34f379507efced9cc434b already in progress: creating tablet
I20251027 21:08:00.510476 10041 tablet_replica.cc:333] T dc589852710342f7bd017193e4075303 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:00.510634 10041 raft_consensus.cc:2243] T dc589852710342f7bd017193e4075303 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.510784 10041 raft_consensus.cc:2272] T dc589852710342f7bd017193e4075303 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.506623 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 669) took 1584 ms. Trace:
I20251027 21:08:00.507618 9710 leader_election.cc:304] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:00.511227 10041 ts_tablet_manager.cc:1916] T dc589852710342f7bd017193e4075303 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.511435 10718 raft_consensus.cc:2749] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.512555 9771 raft_consensus.cc:3060] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:00.514554 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet e0968c2f25b64f6685f38a999ee9d60e (table default.Table_68f478946b1543a68842ffad9e14864f [id=830444c79e80486aa3e7668a0ec91023]) successfully deleted
W20251027 21:08:00.514792 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet e1385beb1dc14ee0b42a2f4e1f66f532 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1385beb1dc14ee0b42a2f4e1f66f532 already in progress: creating tablet
I20251027 21:08:00.518863 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 7a28dc9fe75744e5aa3d486afbe29561
I20251027 21:08:00.520282 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a28dc9fe75744e5aa3d486afbe29561 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.521284 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a28dc9fe75744e5aa3d486afbe29561 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:00.522428 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a28dc9fe75744e5aa3d486afbe29561 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.523365 10040 rpcz_store.cc:276] 1027 21:07:58.922260 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.922309 (+ 49us) service_pool.cc:225] Handling call
1027 21:08:00.506444 (+1584135us) ts_tablet_manager.cc:1151] Deleting tablet 8ae222307a374a91916f560a0fbe8d8f
1027 21:08:00.506457 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.506575 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1583945}
W20251027 21:08:00.524163 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 8ae222307a374a91916f560a0fbe8d8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ae222307a374a91916f560a0fbe8d8f already in progress: creating tablet
I20251027 21:08:00.524575 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 670) took 1594 ms. Trace:
I20251027 21:08:00.525489 10043 ts_tablet_manager.cc:1929] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.525585 10043 log.cc:1199] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/ccaf5c2f8e544842880ba8ba27300ae6
I20251027 21:08:00.525836 10043 ts_tablet_manager.cc:1950] T ccaf5c2f8e544842880ba8ba27300ae6 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:00.526770 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet ccaf5c2f8e544842880ba8ba27300ae6 (table default.Table_9c6b1e43fc5f4f4389620f1ee244db12 [id=2b0d4e085a8e42879c67963231a24b03]) successfully deleted
I20251027 21:08:00.530074 10040 rpcz_store.cc:276] 1027 21:07:58.930390 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:58.935041 (+ 4651us) service_pool.cc:225] Handling call
1027 21:08:00.524414 (+1589373us) ts_tablet_manager.cc:1151] Deleting tablet a8328318a4f14b38a4ce36d183e99dfb
1027 21:08:00.524424 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.524545 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1589161}
I20251027 21:08:00.530812 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 24e6f57c27c743a2a260b52d264d92ea (DEFAULT_TABLE table=default.Table_b221c4f03f1e49d0aab44626822ea9ae [id=d6aaf80441554598a6ed6e4ce0dc8e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.531239 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24e6f57c27c743a2a260b52d264d92ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.531244 10032 tablet_bootstrap.cc:492] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:00.531365 10032 ts_tablet_manager.cc:1403] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.136s user 0.005s sys 0.013s
I20251027 21:08:00.532241 10032 raft_consensus.cc:359] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.532392 10032 raft_consensus.cc:385] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.532436 10032 raft_consensus.cc:740] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.532617 10032 consensus_queue.cc:260] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.532996 10032 ts_tablet_manager.cc:1434] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20251027 21:08:00.534556 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet a8328318a4f14b38a4ce36d183e99dfb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8328318a4f14b38a4ce36d183e99dfb already in progress: creating tablet
I20251027 21:08:00.534786 9304 master_service.cc:954] ReplaceTablet: received request to replace tablet 95f9595784df434dbb0dd29d9a05511e from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:00.535818 10032 tablet_bootstrap.cc:492] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:00.537549 10032 tablet_bootstrap.cc:654] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.538805 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 22d861810ed846c3b77a452d05e93d72 with delete_type TABLET_DATA_DELETED (replaced by tablet c0e2772d66404d74891e0893c49a66e0) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.539531 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3f1fa06443254429a4f8cd54e07636cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:00.540056 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f1fa06443254429a4f8cd54e07636cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.543632 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet d1d16726537a473ebd731e7a1f448400
I20251027 21:08:00.544545 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet d1d16726537a473ebd731e7a1f448400 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.545480 10694 raft_consensus.cc:493] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.545598 10694 raft_consensus.cc:515] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.546254 10694 leader_election.cc:290] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:00.546288 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet d1d16726537a473ebd731e7a1f448400 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:00.547070 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a58cd07e1bb4ffdbf39e7e16a46022a" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:00.547907 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 123e6e3748574ea581f67fff49037762 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 123e6e3748574ea581f67fff49037762 already in progress: creating tablet
W20251027 21:08:00.548447 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet deb8b54e5e0f42bca4330d2160e5524a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet deb8b54e5e0f42bca4330d2160e5524a already in progress: creating tablet
W20251027 21:08:00.548530 9860 leader_election.cc:343] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.549165 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6b471834498c4afeb57279278d45d9a9" } new_table_name: "default.Table_c4824ada6d0f433dbb4770562a66e1ad" modify_external_catalogs: true
W20251027 21:08:00.549482 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet c1f99ca89f924b58b1416d2945a6ba40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1f99ca89f924b58b1416d2945a6ba40 already in progress: creating tablet
I20251027 21:08:00.552762 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet fb385f17054e430bbd69a2c9d22944ca
I20251027 21:08:00.554083 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet fb385f17054e430bbd69a2c9d22944ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.554245 10694 raft_consensus.cc:493] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.554351 10694 raft_consensus.cc:515] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.554839 10694 leader_election.cc:290] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:00.554942 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet fb385f17054e430bbd69a2c9d22944ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:00.555996 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103c8e5a7c744be90be097179904ba3" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:00.556516 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_169ebf3b9cd4423aad9f0c459be30992" } new_table_name: "default.Table_4ba4c823d97045959f6926a92a489794" modify_external_catalogs: true
I20251027 21:08:00.556571 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_94b3855384ef4019bf69b5ff2ccc6dd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:00.557076 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94b3855384ef4019bf69b5ff2ccc6dd6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:00.558383 9860 leader_election.cc:343] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:00.559429 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet cc4635c00e9b40c49ce04debc76a04bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc4635c00e9b40c49ce04debc76a04bf already in progress: creating tablet
I20251027 21:08:00.559686 10043 tablet_replica.cc:333] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:00.559907 10043 raft_consensus.cc:2243] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:00.560635 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 24e6f57c27c743a2a260b52d264d92ea (DEFAULT_TABLE table=default.Table_b221c4f03f1e49d0aab44626822ea9ae [id=d6aaf80441554598a6ed6e4ce0dc8e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.560973 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24e6f57c27c743a2a260b52d264d92ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.562491 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a58cd07e1bb4ffdbf39e7e16a46022a" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:00.562754 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet fb385f17054e430bbd69a2c9d22944ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.562835 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103c8e5a7c744be90be097179904ba3" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:00.563050 9859 leader_election.cc:343] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.563138 9859 leader_election.cc:304] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:00.563447 10694 raft_consensus.cc:2749] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:00.563782 9859 leader_election.cc:343] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.563869 9859 leader_election.cc:304] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:00.564184 10694 raft_consensus.cc:2749] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.566434 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 24e6f57c27c743a2a260b52d264d92ea (DEFAULT_TABLE table=default.Table_b221c4f03f1e49d0aab44626822ea9ae [id=d6aaf80441554598a6ed6e4ce0dc8e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.566896 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24e6f57c27c743a2a260b52d264d92ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.569010 10043 pending_rounds.cc:70] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14: Trying to abort 1 pending ops.
I20251027 21:08:00.570268 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet d1d16726537a473ebd731e7a1f448400 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.572271 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 678) took 1376 ms. Trace:
I20251027 21:08:00.572348 10040 rpcz_store.cc:276] 1027 21:07:59.195307 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.195446 (+ 139us) service_pool.cc:225] Handling call
1027 21:08:00.572141 (+1376695us) ts_tablet_manager.cc:1151] Deleting tablet 9239f74d725649cab5ea1b0077bccef3
1027 21:08:00.572150 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.572239 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1376247}
W20251027 21:08:00.572723 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 9239f74d725649cab5ea1b0077bccef3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9239f74d725649cab5ea1b0077bccef3 already in progress: creating tablet
I20251027 21:08:00.572875 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 677) took 1377 ms. Trace:
I20251027 21:08:00.572947 10040 rpcz_store.cc:276] 1027 21:07:59.195153 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.195363 (+ 210us) service_pool.cc:225] Handling call
1027 21:08:00.572763 (+1377400us) ts_tablet_manager.cc:1151] Deleting tablet fc2cf41a7b474f7d86a46af76cef4100
1027 21:08:00.572771 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.572859 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1371158}
W20251027 21:08:00.573529 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet fc2cf41a7b474f7d86a46af76cef4100 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc2cf41a7b474f7d86a46af76cef4100 already in progress: creating tablet
I20251027 21:08:00.575552 10032 tablet_bootstrap.cc:492] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:00.575686 10032 ts_tablet_manager.cc:1403] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.001s
I20251027 21:08:00.576363 10032 raft_consensus.cc:359] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.576515 10032 raft_consensus.cc:385] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.568727 9615 tablet_service.cc:1505] Processing CreateTablet for tablet c4f03fd46465446393f2b1fbaac167dc (DEFAULT_TABLE table=default.Table_97cd866055604dbca95060d43b29dc75 [id=e24ed5be8dbf44629f7d85d2b3da8871]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.576573 10032 raft_consensus.cc:740] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.576761 10032 consensus_queue.cc:260] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.577149 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4f03fd46465446393f2b1fbaac167dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.577307 10032 ts_tablet_manager.cc:1434] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:00.577654 10032 tablet_bootstrap.cc:492] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:00.578995 10041 ts_tablet_manager.cc:1929] T dc589852710342f7bd017193e4075303 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.582234 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d120d0c629944b8582b7177c111c78f0" } new_table_name: "default.Table_9f43636ecbc942769ef30bffe0139f58" modify_external_catalogs: true
I20251027 21:08:00.582052 10043 pending_rounds.cc:77] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510651480752128 op_type: NO_OP noop_request { }
I20251027 21:08:00.582682 10043 raft_consensus.cc:2889] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:00.582780 10043 raft_consensus.cc:2272] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.583647 10043 ts_tablet_manager.cc:1916] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.583628 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4aebdc4cec714b2cb02e19c014b27779"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:00.584224 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4aebdc4cec714b2cb02e19c014b27779 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.585407 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e9721dd5a67f48559827c11b5944ba0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:00.586524 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_84434842cc354813bc45be80de0d8322" } new_table_name: "default.Table_df7c27fe46e74ecda6bd9e58bc24bf72" modify_external_catalogs: true
I20251027 21:08:00.586711 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_090167e6602f4bff86b275f39d9a58cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:00.587172 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_090167e6602f4bff86b275f39d9a58cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.586570 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8b0df27ef0ab4ed6afe3765a736f8f20" } modify_external_catalogs: true
I20251027 21:08:00.587767 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8b0df27ef0ab4ed6afe3765a736f8f20" } modify_external_catalogs: true
I20251027 21:08:00.588199 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_398fb57a71854dccb9a50f46010e7d5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:00.588644 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_398fb57a71854dccb9a50f46010e7d5c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:00.589355 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9721dd5a67f48559827c11b5944ba0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.591176 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_46c804a5818a4a4ca815f2449ccbe842"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:00.579521 10032 tablet_bootstrap.cc:654] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.584787 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 630ae4d66f99474382814cc32cc60ea0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2f641bc4dbed4c0abd0a55b253352507) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.585810 9899 tablet_service.cc:1505] Processing CreateTablet for tablet c4f03fd46465446393f2b1fbaac167dc (DEFAULT_TABLE table=default.Table_97cd866055604dbca95060d43b29dc75 [id=e24ed5be8dbf44629f7d85d2b3da8871]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.591418 10041 log.cc:1199] T dc589852710342f7bd017193e4075303 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/dc589852710342f7bd017193e4075303
I20251027 21:08:00.591696 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4f03fd46465446393f2b1fbaac167dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.591632 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 96ebda06261b47ac956a16132c9a8879 (DEFAULT_TABLE table=default.Table_8b0c128ff09445d3b6b5ece094733305 [id=64c35a3c47344c6da31e2a482ee4d194]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.591753 10041 ts_tablet_manager.cc:1950] T dc589852710342f7bd017193e4075303 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:00.591975 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96ebda06261b47ac956a16132c9a8879. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.592537 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 709) took 1101 ms. Trace:
I20251027 21:08:00.592631 10041 rpcz_store.cc:276] 1027 21:07:59.490951 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.491133 (+ 182us) service_pool.cc:225] Handling call
1027 21:08:00.510436 (+1019303us) ts_tablet_manager.cc:1151] Deleting tablet dc589852710342f7bd017193e4075303
1027 21:08:00.510444 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.546583 (+ 36139us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.578898 (+ 32315us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.592514 (+ 13616us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1018987}
I20251027 21:08:00.596928 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 711) took 1084 ms. Trace:
I20251027 21:08:00.593163 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet dc589852710342f7bd017193e4075303 (table default.Table_d2ff301b3e2b4602a2483859973a02d6 [id=ccc88d98bcd74bde873cca8c9ce3e134]) successfully deleted
I20251027 21:08:00.605034 10694 raft_consensus.cc:493] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.604890 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_ec0b0d1ae0774aa598e7311546fcedd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:00.610284 10694 raft_consensus.cc:515] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:00.610641 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec0b0d1ae0774aa598e7311546fcedd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.611102 10694 leader_election.cc:290] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:00.610994 10041 rpcz_store.cc:276] 1027 21:07:59.512011 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.512060 (+ 49us) service_pool.cc:225] Handling call
1027 21:08:00.596764 (+1084704us) ts_tablet_manager.cc:1151] Deleting tablet 113a5d30d4b3490187681383d4fef325
1027 21:08:00.596777 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.596904 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1084442}
I20251027 21:08:00.611582 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 713) took 1091 ms. Trace:
I20251027 21:08:00.611665 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4996d8fae094db8a96599b9057f1649" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:00.611646 10041 rpcz_store.cc:276] 1027 21:07:59.520002 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.520193 (+ 191us) service_pool.cc:225] Handling call
1027 21:08:00.611439 (+1091246us) ts_tablet_manager.cc:1151] Deleting tablet 849b3e9fc34045939a7a678fa8269d7f
1027 21:08:00.611453 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.611568 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1091041}
I20251027 21:08:00.611976 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 714) took 1091 ms. Trace:
I20251027 21:08:00.612061 10041 rpcz_store.cc:276] 1027 21:07:59.520136 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.520426 (+ 290us) service_pool.cc:225] Handling call
1027 21:08:00.611864 (+1091438us) ts_tablet_manager.cc:1151] Deleting tablet e2ff7fc3c1fa4dbd9a5c38d3219b03fa
1027 21:08:00.611872 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.611957 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1091277}
W20251027 21:08:00.612176 9860 leader_election.cc:343] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.612414 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 712) took 1092 ms. Trace:
I20251027 21:08:00.612485 10041 rpcz_store.cc:276] 1027 21:07:59.519896 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.520614 (+ 718us) service_pool.cc:225] Handling call
1027 21:08:00.612297 (+1091683us) ts_tablet_manager.cc:1151] Deleting tablet d2203ee62f7e404c95c5f86b5ae85feb
1027 21:08:00.612308 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.612398 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1091539}
W20251027 21:08:00.612689 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46c804a5818a4a4ca815f2449ccbe842 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.612774 10041 tablet_replica.cc:333] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:00.613008 10041 raft_consensus.cc:2243] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.613168 10041 raft_consensus.cc:2272] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.613746 10041 ts_tablet_manager.cc:1916] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.613796 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4996d8fae094db8a96599b9057f1649" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:00.614055 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 96ebda06261b47ac956a16132c9a8879 (DEFAULT_TABLE table=default.Table_8b0c128ff09445d3b6b5ece094733305 [id=64c35a3c47344c6da31e2a482ee4d194]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.614461 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96ebda06261b47ac956a16132c9a8879. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:00.618912 9859 leader_election.cc:343] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.619020 9859 leader_election.cc:304] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:00.619199 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 113a5d30d4b3490187681383d4fef325 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 113a5d30d4b3490187681383d4fef325 already in progress: creating tablet
W20251027 21:08:00.619550 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 849b3e9fc34045939a7a678fa8269d7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 849b3e9fc34045939a7a678fa8269d7f already in progress: creating tablet
I20251027 21:08:00.619629 10694 raft_consensus.cc:2749] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:00.619767 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet e2ff7fc3c1fa4dbd9a5c38d3219b03fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2ff7fc3c1fa4dbd9a5c38d3219b03fa already in progress: creating tablet
W20251027 21:08:00.620028 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet d2203ee62f7e404c95c5f86b5ae85feb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2203ee62f7e404c95c5f86b5ae85feb already in progress: creating tablet
I20251027 21:08:00.620254 9754 tablet_service.cc:1505] Processing CreateTablet for tablet e53b2c28c75b43bdafa0d20f4ad61fcc (DEFAULT_TABLE table=default.Table_a47950555ce34e4aabcf33ae10344421 [id=789cc809a05b4238af0991373502f55d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.620623 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e53b2c28c75b43bdafa0d20f4ad61fcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.620743 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 96ebda06261b47ac956a16132c9a8879 (DEFAULT_TABLE table=default.Table_8b0c128ff09445d3b6b5ece094733305 [id=64c35a3c47344c6da31e2a482ee4d194]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.621234 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96ebda06261b47ac956a16132c9a8879. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.621996 10043 ts_tablet_manager.cc:1929] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:00.624301 9749 tablet_service.cc:1505] Processing CreateTablet for tablet c4f03fd46465446393f2b1fbaac167dc (DEFAULT_TABLE table=default.Table_97cd866055604dbca95060d43b29dc75 [id=e24ed5be8dbf44629f7d85d2b3da8871]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.624612 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4f03fd46465446393f2b1fbaac167dc. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:00.626022 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fa39697360c46138ef4cd21e8c7a950 consensus state has no leader
I20251027 21:08:00.631134 10718 raft_consensus.cc:493] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.631167 9895 tablet_service.cc:1505] Processing CreateTablet for tablet e53b2c28c75b43bdafa0d20f4ad61fcc (DEFAULT_TABLE table=default.Table_a47950555ce34e4aabcf33ae10344421 [id=789cc809a05b4238af0991373502f55d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.631266 10718 raft_consensus.cc:515] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.631664 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e53b2c28c75b43bdafa0d20f4ad61fcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.631862 10718 leader_election.cc:290] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:00.632721 9710 leader_election.cc:343] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 9c239216cb8846f0b1aaa360ddb83321
W20251027 21:08:00.633625 9711 leader_election.cc:343] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 9c239216cb8846f0b1aaa360ddb83321
I20251027 21:08:00.633725 9711 leader_election.cc:304] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:00.633955 10718 raft_consensus.cc:2749] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.532341 10031 tablet_bootstrap.cc:654] T 78884c2be75f45f881f5424d3b3f4e08 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.634888 10041 ts_tablet_manager.cc:1929] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.637128 10041 log.cc:1199] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/0a58cd07e1bb4ffdbf39e7e16a46022a
I20251027 21:08:00.637539 10041 ts_tablet_manager.cc:1950] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:00.638600 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 719) took 1026 ms. Trace:
I20251027 21:08:00.639701 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet be33f1825a4d433ea9ed859f37fe208e from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:00.639786 10041 rpcz_store.cc:276] 1027 21:07:59.611807 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.646512 (+ 34705us) service_pool.cc:225] Handling call
1027 21:08:00.612729 (+966217us) ts_tablet_manager.cc:1151] Deleting tablet 0a58cd07e1bb4ffdbf39e7e16a46022a
1027 21:08:00.612738 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.617991 (+ 5253us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.634770 (+ 16779us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.638578 (+ 3808us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":966028}
I20251027 21:08:00.640347 10041 tablet_replica.cc:333] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:00.640575 10041 raft_consensus.cc:2243] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.640738 10041 raft_consensus.cc:2272] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.641310 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 0a58cd07e1bb4ffdbf39e7e16a46022a (table default.Table_bc5ec745ebe54017b6bc9635581c813d [id=31172964ef544c5f99cade30a42b340d]) successfully deleted
W20251027 21:08:00.646116 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9e4469ce1625455f8ac66cd3094cc421 consensus state has no leader
I20251027 21:08:00.646898 10041 ts_tablet_manager.cc:1916] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.652832 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 676) took 1457 ms. Trace:
I20251027 21:08:00.652984 10040 rpcz_store.cc:276] 1027 21:07:59.195007 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.195624 (+ 617us) service_pool.cc:225] Handling call
1027 21:08:00.635927 (+1440303us) ts_tablet_manager.cc:1151] Deleting tablet 57de49dacafa497dad79ab96540b5888
1027 21:08:00.635936 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.652803 (+ 16867us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1425381}
I20251027 21:08:00.653409 10040 tablet_replica.cc:333] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:00.653636 10040 raft_consensus.cc:2243] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.653822 10040 raft_consensus.cc:2272] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:00.654196 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 57de49dacafa497dad79ab96540b5888 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57de49dacafa497dad79ab96540b5888 already in progress: creating tablet
I20251027 21:08:00.656972 10040 ts_tablet_manager.cc:1916] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.657792 9772 raft_consensus.cc:3060] T 995284883d834bbb9ef583b736ca8402 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:00.660138 10041 ts_tablet_manager.cc:1929] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.660970 9742 tablet_service.cc:1505] Processing CreateTablet for tablet d1d16726537a473ebd731e7a1f448400 (DEFAULT_TABLE table=default.Table_63950b7e1b8c4f3a989b8f14a2409f78 [id=ccc1386dac0446339f471d81a53f5855]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.661381 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1d16726537a473ebd731e7a1f448400. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.661828 9899 tablet_service.cc:1505] Processing CreateTablet for tablet d1d16726537a473ebd731e7a1f448400 (DEFAULT_TABLE table=default.Table_63950b7e1b8c4f3a989b8f14a2409f78 [id=ccc1386dac0446339f471d81a53f5855]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.663676 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1d16726537a473ebd731e7a1f448400. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.664438 10723 tablet.cc:1722] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:00.668627 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5cb117352e754df2a40d8e5cfed465b5" } new_table_name: "default.Table_cfef185a988e4d61ae6575c228abc88d" modify_external_catalogs: true
I20251027 21:08:00.674482 10041 log.cc:1199] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/c4996d8fae094db8a96599b9057f1649
I20251027 21:08:00.674582 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_fb6d994c828141ddb43073cb17d9fad6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:00.675014 10041 ts_tablet_manager.cc:1950] T c4996d8fae094db8a96599b9057f1649 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
W20251027 21:08:00.675166 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb6d994c828141ddb43073cb17d9fad6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.676636 10041 tablet_replica.cc:333] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:00.676653 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet c4996d8fae094db8a96599b9057f1649 (table default.Table_b462e4bf95dc4fc4ad3fd23b236fffa7 [id=68da6285e1d247f1950a0b4929c885a4]) successfully deleted
I20251027 21:08:00.676847 10041 raft_consensus.cc:2243] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251027 21:08:00.676929 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 83a040332f994055a9ecf4c66720b7e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83a040332f994055a9ecf4c66720b7e1 already in progress: creating tablet
I20251027 21:08:00.677003 10041 raft_consensus.cc:2272] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:00.677179 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet deb8b54e5e0f42bca4330d2160e5524a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet deb8b54e5e0f42bca4330d2160e5524a already in progress: creating tablet
W20251027 21:08:00.677433 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet e1385beb1dc14ee0b42a2f4e1f66f532 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1385beb1dc14ee0b42a2f4e1f66f532 already in progress: creating tablet
W20251027 21:08:00.678390 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 123e6e3748574ea581f67fff49037762 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 123e6e3748574ea581f67fff49037762 already in progress: creating tablet
I20251027 21:08:00.682046 10031 tablet_bootstrap.cc:492] T 78884c2be75f45f881f5424d3b3f4e08 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:00.682241 10031 ts_tablet_manager.cc:1403] T 78884c2be75f45f881f5424d3b3f4e08 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.176s user 0.004s sys 0.000s
I20251027 21:08:00.683121 10031 raft_consensus.cc:359] T 78884c2be75f45f881f5424d3b3f4e08 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.683338 10031 raft_consensus.cc:385] T 78884c2be75f45f881f5424d3b3f4e08 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.683403 10031 raft_consensus.cc:740] T 78884c2be75f45f881f5424d3b3f4e08 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.683938 10041 ts_tablet_manager.cc:1916] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.689443 10031 consensus_queue.cc:260] T 78884c2be75f45f881f5424d3b3f4e08 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.690788 10031 ts_tablet_manager.cc:1434] T 78884c2be75f45f881f5424d3b3f4e08 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251027 21:08:00.691040 10031 tablet_bootstrap.cc:492] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:00.693240 10031 tablet_bootstrap.cc:654] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.698602 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 9e4469ce1625455f8ac66cd3094cc421 (DEFAULT_TABLE table=default.Table_9f43636ecbc942769ef30bffe0139f58 [id=fd5ae7255ebc4966b5e1f37564d79b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.698975 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e4469ce1625455f8ac66cd3094cc421. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.699277 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 9e4469ce1625455f8ac66cd3094cc421 (DEFAULT_TABLE table=default.Table_9f43636ecbc942769ef30bffe0139f58 [id=fd5ae7255ebc4966b5e1f37564d79b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.699666 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e4469ce1625455f8ac66cd3094cc421. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:00.705468 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93d2f7df14a74731aea84927f0b7ed45 consensus state has no leader
I20251027 21:08:00.706585 10041 ts_tablet_manager.cc:1929] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.706683 10041 log.cc:1199] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/3103c8e5a7c744be90be097179904ba3
I20251027 21:08:00.707062 10041 ts_tablet_manager.cc:1950] T 3103c8e5a7c744be90be097179904ba3 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:00.708069 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 3103c8e5a7c744be90be097179904ba3 (table default.Table_c04ff0f25ba04d25958346fc631b2802 [id=a6a32b47c2354a2d9535ff2b48d506de]) successfully deleted
I20251027 21:08:00.710079 10742 tablet.cc:1722] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251027 21:08:00.710273 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 211ac01a37f8434f95aa6a63970296fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 211ac01a37f8434f95aa6a63970296fa already in progress: creating tablet
I20251027 21:08:00.710942 10041 tablet_replica.cc:333] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:00.711452 10041 raft_consensus.cc:2243] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.711915 10041 raft_consensus.cc:2272] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.712623 10041 ts_tablet_manager.cc:1916] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.716584 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_773dc0a1742a4f9194e9b26ea3d07123" } modify_external_catalogs: true
I20251027 21:08:00.716802 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_773dc0a1742a4f9194e9b26ea3d07123" } modify_external_catalogs: true
I20251027 21:08:00.718604 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 9aa1838741874e3f8e37617c771852ad (DEFAULT_TABLE table=default.Table_6f6eba299ba04a7398b597dd4caf6d9c [id=6e3e3733563c444583da9ec34a676772]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.719007 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9aa1838741874e3f8e37617c771852ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.719085 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 9aa1838741874e3f8e37617c771852ad (DEFAULT_TABLE table=default.Table_6f6eba299ba04a7398b597dd4caf6d9c [id=6e3e3733563c444583da9ec34a676772]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.719398 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9aa1838741874e3f8e37617c771852ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.719657 10031 tablet_bootstrap.cc:492] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:00.719763 10031 ts_tablet_manager.cc:1403] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.000s
I20251027 21:08:00.720357 10694 raft_consensus.cc:493] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.720408 10031 raft_consensus.cc:359] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.720468 10694 raft_consensus.cc:515] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.720551 10031 raft_consensus.cc:385] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.720602 10031 raft_consensus.cc:740] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.720810 10031 consensus_queue.cc:260] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.721020 10694 leader_election.cc:290] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:00.721378 10031 ts_tablet_manager.cc:1434] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:00.721612 10031 tablet_bootstrap.cc:492] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:00.724812 10740 tablet.cc:1722] T 995284883d834bbb9ef583b736ca8402 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251027 21:08:00.727173 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9bf3a093be8498c9dd5511b53436b16" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:00.727658 9859 leader_election.cc:343] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.725033 10031 tablet_bootstrap.cc:654] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.731016 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 622fa1b3ff134f2d8a087ba13a85e597
I20251027 21:08:00.731763 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_8c2f4fc52c37435fb5a25817b1545503"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:00.732301 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c2f4fc52c37435fb5a25817b1545503 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.733178 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 622fa1b3ff134f2d8a087ba13a85e597 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.630630 9615 tablet_service.cc:1505] Processing CreateTablet for tablet e53b2c28c75b43bdafa0d20f4ad61fcc (DEFAULT_TABLE table=default.Table_a47950555ce34e4aabcf33ae10344421 [id=789cc809a05b4238af0991373502f55d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.642196 9602 tablet_service.cc:1505] Processing CreateTablet for tablet d1d16726537a473ebd731e7a1f448400 (DEFAULT_TABLE table=default.Table_63950b7e1b8c4f3a989b8f14a2409f78 [id=ccc1386dac0446339f471d81a53f5855]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.734609 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1d16726537a473ebd731e7a1f448400. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.735780 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 8db666d9b7eb458b843ab5910cd7dff4 (DEFAULT_TABLE table=default.Table_8b1c94c0aec244e3950e08017ed7da7d [id=c025dc84a195497497000214bbd3f6e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.736100 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8db666d9b7eb458b843ab5910cd7dff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.738484 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e53b2c28c75b43bdafa0d20f4ad61fcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.745396 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 8db666d9b7eb458b843ab5910cd7dff4 (DEFAULT_TABLE table=default.Table_8b1c94c0aec244e3950e08017ed7da7d [id=c025dc84a195497497000214bbd3f6e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.745813 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8db666d9b7eb458b843ab5910cd7dff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.746336 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5315576a7f87464dbbfded0081ddef2d" } modify_external_catalogs: true
I20251027 21:08:00.746549 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5315576a7f87464dbbfded0081ddef2d" } modify_external_catalogs: true
I20251027 21:08:00.642645 10029 tablet_bootstrap.cc:492] T f22f013b25074d4db9c7f13caf5b9b42 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:00.666373 10725 tablet.cc:1722] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:00.679826 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 9e4469ce1625455f8ac66cd3094cc421 (DEFAULT_TABLE table=default.Table_9f43636ecbc942769ef30bffe0139f58 [id=fd5ae7255ebc4966b5e1f37564d79b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.750362 10029 ts_tablet_manager.cc:1403] T f22f013b25074d4db9c7f13caf5b9b42 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.298s user 0.005s sys 0.000s
I20251027 21:08:00.712230 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 9aa1838741874e3f8e37617c771852ad (DEFAULT_TABLE table=default.Table_6f6eba299ba04a7398b597dd4caf6d9c [id=6e3e3733563c444583da9ec34a676772]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.721652 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9bf3a093be8498c9dd5511b53436b16" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:00.733018 9611 tablet_service.cc:1552] Processing DeleteTablet for tablet 622fa1b3ff134f2d8a087ba13a85e597 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:00.750815 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9aa1838741874e3f8e37617c771852ad. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:00.751050 9860 leader_election.cc:343] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.751183 9860 leader_election.cc:304] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:00.751570 10694 raft_consensus.cc:2749] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.752373 10029 raft_consensus.cc:359] T f22f013b25074d4db9c7f13caf5b9b42 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.752619 10029 raft_consensus.cc:385] T f22f013b25074d4db9c7f13caf5b9b42 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.752722 10031 tablet_bootstrap.cc:492] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:00.752758 10029 raft_consensus.cc:740] T f22f013b25074d4db9c7f13caf5b9b42 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.752904 10031 ts_tablet_manager.cc:1403] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.002s
I20251027 21:08:00.753123 10029 consensus_queue.cc:260] T f22f013b25074d4db9c7f13caf5b9b42 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.753661 10029 ts_tablet_manager.cc:1434] T f22f013b25074d4db9c7f13caf5b9b42 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:08:00.754024 10029 tablet_bootstrap.cc:492] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:00.733918 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 8db666d9b7eb458b843ab5910cd7dff4 (DEFAULT_TABLE table=default.Table_8b1c94c0aec244e3950e08017ed7da7d [id=c025dc84a195497497000214bbd3f6e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.755872 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e4469ce1625455f8ac66cd3094cc421. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.756775 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8db666d9b7eb458b843ab5910cd7dff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.757246 10029 tablet_bootstrap.cc:654] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.757498 10031 raft_consensus.cc:359] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.757663 10031 raft_consensus.cc:385] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.757709 10031 raft_consensus.cc:740] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.757915 10031 consensus_queue.cc:260] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.751058 10041 ts_tablet_manager.cc:1929] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.758862 10041 log.cc:1199] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/c1f99ca89f924b58b1416d2945a6ba40
I20251027 21:08:00.759143 10041 ts_tablet_manager.cc:1950] T c1f99ca89f924b58b1416d2945a6ba40 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:00.760334 10031 ts_tablet_manager.cc:1434] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20251027 21:08:00.760371 10041 tablet_replica.cc:333] T 94f2dbf47a5c4458bee54b26f272156e P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:00.760541 10041 raft_consensus.cc:2243] T 94f2dbf47a5c4458bee54b26f272156e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.760680 10041 raft_consensus.cc:2272] T 94f2dbf47a5c4458bee54b26f272156e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.760948 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet c1f99ca89f924b58b1416d2945a6ba40 (table default.Table_90157e3841104d92aafc5d82ffea13a9 [id=bc676ad23e1f4c4bae88752f31108ee2]) successfully deleted
I20251027 21:08:00.761173 10041 ts_tablet_manager.cc:1916] T 94f2dbf47a5c4458bee54b26f272156e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:00.761250 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet cc4635c00e9b40c49ce04debc76a04bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc4635c00e9b40c49ce04debc76a04bf already in progress: creating tablet
I20251027 21:08:00.761324 10031 tablet_bootstrap.cc:492] T c400101d2a39490e83722500839198cd P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:00.763371 10740 tablet.cc:1722] T 7f7ad06a15c04efbb701634e14904351 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:08:00.766646 10718 raft_consensus.cc:493] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.766762 10718 raft_consensus.cc:515] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.767002 10043 log.cc:1199] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/dc589852710342f7bd017193e4075303
I20251027 21:08:00.767361 10718 leader_election.cc:290] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:00.767688 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 622fa1b3ff134f2d8a087ba13a85e597 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.769740 10031 tablet_bootstrap.cc:654] T c400101d2a39490e83722500839198cd P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.771741 9611 tablet_service.cc:1505] Processing CreateTablet for tablet f9534e65d8bb4b3a88df934871792119 (DEFAULT_TABLE table=default.Table_aca70af1903740b5b35bbc3554beff36 [id=f4cfb9996f6c41558497feb9e4c68ce2]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:00.772274 9710 leader_election.cc:343] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 7d8825b80cec4ba68da077ca3b8d89db
I20251027 21:08:00.773466 9905 tablet_service.cc:1505] Processing CreateTablet for tablet f9534e65d8bb4b3a88df934871792119 (DEFAULT_TABLE table=default.Table_aca70af1903740b5b35bbc3554beff36 [id=f4cfb9996f6c41558497feb9e4c68ce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.773792 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9534e65d8bb4b3a88df934871792119. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.774315 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9534e65d8bb4b3a88df934871792119. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:00.776396 9711 leader_election.cc:343] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 7d8825b80cec4ba68da077ca3b8d89db
I20251027 21:08:00.776495 9711 leader_election.cc:304] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:00.776690 10718 raft_consensus.cc:2749] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.781878 9742 tablet_service.cc:1505] Processing CreateTablet for tablet f9534e65d8bb4b3a88df934871792119 (DEFAULT_TABLE table=default.Table_aca70af1903740b5b35bbc3554beff36 [id=f4cfb9996f6c41558497feb9e4c68ce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.782239 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9534e65d8bb4b3a88df934871792119. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.793479 9303 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 7f7ad06a15c04efbb701634e14904351 (table default.Table_318d41ccfb2c4d2c99bafcea875143a1 [id=2bc39b5bc02645ef830eee14e92707db]). Expected version 1 got 0
I20251027 21:08:00.794966 10043 ts_tablet_manager.cc:1950] T dc589852710342f7bd017193e4075303 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:00.800966 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet f22f013b25074d4db9c7f13caf5b9b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:00.801610 10031 tablet_bootstrap.cc:492] T c400101d2a39490e83722500839198cd P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:00.801728 10031 ts_tablet_manager.cc:1403] T c400101d2a39490e83722500839198cd P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.040s user 0.000s sys 0.006s
I20251027 21:08:00.802448 10031 raft_consensus.cc:359] T c400101d2a39490e83722500839198cd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.802630 10031 raft_consensus.cc:385] T c400101d2a39490e83722500839198cd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.802690 10031 raft_consensus.cc:740] T c400101d2a39490e83722500839198cd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.802903 10031 consensus_queue.cc:260] T c400101d2a39490e83722500839198cd P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.804757 9609 tablet_service.cc:1505] Processing CreateTablet for tablet d25b493ab8ca401bb1cbe01e355d7bcd (DEFAULT_TABLE table=default.Table_90157e3841104d92aafc5d82ffea13a9 [id=bc676ad23e1f4c4bae88752f31108ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.804983 10031 ts_tablet_manager.cc:1434] T c400101d2a39490e83722500839198cd P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:08:00.805411 10031 tablet_bootstrap.cc:492] T d8e008c868d04281beba0e5b204eb69a P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:00.806406 9755 tablet_service.cc:1505] Processing CreateTablet for tablet d25b493ab8ca401bb1cbe01e355d7bcd (DEFAULT_TABLE table=default.Table_90157e3841104d92aafc5d82ffea13a9 [id=bc676ad23e1f4c4bae88752f31108ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.806737 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d25b493ab8ca401bb1cbe01e355d7bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.807011 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d25b493ab8ca401bb1cbe01e355d7bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.807075 10031 tablet_bootstrap.cc:654] T d8e008c868d04281beba0e5b204eb69a P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.815335 9905 tablet_service.cc:1505] Processing CreateTablet for tablet d25b493ab8ca401bb1cbe01e355d7bcd (DEFAULT_TABLE table=default.Table_90157e3841104d92aafc5d82ffea13a9 [id=bc676ad23e1f4c4bae88752f31108ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.815698 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d25b493ab8ca401bb1cbe01e355d7bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.824975 10694 raft_consensus.cc:493] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.825140 10694 raft_consensus.cc:515] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.825693 10694 leader_election.cc:290] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:00.826594 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22d861810ed846c3b77a452d05e93d72" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:00.827121 9859 leader_election.cc:343] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.827315 10694 raft_consensus.cc:493] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.827419 10694 raft_consensus.cc:515] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.827862 10694 leader_election.cc:290] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:00.829280 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "648f506ec87c4fb8a3f6d4591873dbb8" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:00.829680 9859 leader_election.cc:343] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.830713 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "648f506ec87c4fb8a3f6d4591873dbb8" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:00.831490 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet dc589852710342f7bd017193e4075303 (table default.Table_d2ff301b3e2b4602a2483859973a02d6 [id=ccc88d98bcd74bde873cca8c9ce3e134]) successfully deleted
I20251027 21:08:00.833185 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_385f764be86a472d8360fd95634bdcde" } modify_external_catalogs: true
I20251027 21:08:00.833379 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_385f764be86a472d8360fd95634bdcde" } modify_external_catalogs: true
I20251027 21:08:00.833576 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22d861810ed846c3b77a452d05e93d72" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:00.834270 9860 leader_election.cc:343] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.834360 9860 leader_election.cc:304] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:00.834061 9600 tablet_service.cc:1505] Processing CreateTablet for tablet d8224d24b1144ff0b8bc593fc4e244de (DEFAULT_TABLE table=default.Table_4ba3ecc07745498c97bd07deef4525c6 [id=43b59bc0435749a395eb73cd02e4c20d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:00.834690 9860 leader_election.cc:343] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.834771 9860 leader_election.cc:304] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:00.835214 10694 raft_consensus.cc:2749] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.835371 10694 raft_consensus.cc:2749] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.836709 9742 tablet_service.cc:1505] Processing CreateTablet for tablet d8224d24b1144ff0b8bc593fc4e244de (DEFAULT_TABLE table=default.Table_4ba3ecc07745498c97bd07deef4525c6 [id=43b59bc0435749a395eb73cd02e4c20d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.836885 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 229ea4caa6af493cb5b3d593f22aa113 (DEFAULT_TABLE table=default.Table_6d50127567904f7cab81c7a90c655c3d [id=3acc47b267bc47178429d564f16f3c93]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.837301 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 229ea4caa6af493cb5b3d593f22aa113. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.839908 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 229ea4caa6af493cb5b3d593f22aa113 (DEFAULT_TABLE table=default.Table_6d50127567904f7cab81c7a90c655c3d [id=3acc47b267bc47178429d564f16f3c93]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.845355 9899 tablet_service.cc:1505] Processing CreateTablet for tablet d8224d24b1144ff0b8bc593fc4e244de (DEFAULT_TABLE table=default.Table_4ba3ecc07745498c97bd07deef4525c6 [id=43b59bc0435749a395eb73cd02e4c20d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.845705 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8224d24b1144ff0b8bc593fc4e244de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.851585 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet be33f1825a4d433ea9ed859f37fe208e
I20251027 21:08:00.851986 9311 catalog_manager.cc:6719] ReplaceTablet: tablet be33f1825a4d433ea9ed859f37fe208e deleted and replaced by tablet 915f250f9fd046cfa9c6590bd8e4f33c
I20251027 21:08:00.853417 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet be33f1825a4d433ea9ed859f37fe208e with delete_type TABLET_DATA_DELETED (replaced by tablet 915f250f9fd046cfa9c6590bd8e4f33c) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.856148 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet be33f1825a4d433ea9ed859f37fe208e with delete_type TABLET_DATA_DELETED (replaced by tablet 915f250f9fd046cfa9c6590bd8e4f33c) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.856488 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8224d24b1144ff0b8bc593fc4e244de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.857599 10529 raft_consensus.cc:493] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.859192 10043 tablet_replica.cc:333] T 21a8fc4a69f3486e92b1a68d10faed38 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:00.859767 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet be33f1825a4d433ea9ed859f37fe208e with delete_type TABLET_DATA_DELETED (replaced by tablet 915f250f9fd046cfa9c6590bd8e4f33c) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:00.860656 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 229ea4caa6af493cb5b3d593f22aa113 (DEFAULT_TABLE table=default.Table_6d50127567904f7cab81c7a90c655c3d [id=3acc47b267bc47178429d564f16f3c93]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.861238 10529 raft_consensus.cc:515] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.861939 10529 leader_election.cc:290] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:00.863284 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e2cb137bea0d419b9791305f97bc0efd" } modify_external_catalogs: true
I20251027 21:08:00.862512 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f1129eecfb54448965222269d9c32b7" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:00.863492 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e2cb137bea0d419b9791305f97bc0efd" } modify_external_catalogs: true
W20251027 21:08:00.865016 9566 leader_election.cc:343] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:00.865466 9568 leader_election.cc:343] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 6f1129eecfb54448965222269d9c32b7
I20251027 21:08:00.875566 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 229ea4caa6af493cb5b3d593f22aa113. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.875696 10041 ts_tablet_manager.cc:1929] T 94f2dbf47a5c4458bee54b26f272156e P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.875823 10041 log.cc:1199] T 94f2dbf47a5c4458bee54b26f272156e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/94f2dbf47a5c4458bee54b26f272156e
I20251027 21:08:00.875876 9568 leader_election.cc:304] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:00.876128 10041 ts_tablet_manager.cc:1950] T 94f2dbf47a5c4458bee54b26f272156e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:00.876617 10529 raft_consensus.cc:2749] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.877028 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 94f2dbf47a5c4458bee54b26f272156e (table default.Table_7954e74d079b416cac73f9a77979c8b3 [id=0103a75b0a714d3c9a533d94b6bda41f]) successfully deleted
W20251027 21:08:00.878878 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f641bc4dbed4c0abd0a55b253352507 consensus state has no leader
I20251027 21:08:00.882062 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e832c43c91c74891aaed15942e4b96dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:00.882182 10043 raft_consensus.cc:2243] T 21a8fc4a69f3486e92b1a68d10faed38 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.882405 10043 raft_consensus.cc:2272] T 21a8fc4a69f3486e92b1a68d10faed38 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.882550 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 229ea4caa6af493cb5b3d593f22aa113. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:00.882695 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e832c43c91c74891aaed15942e4b96dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.882736 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8224d24b1144ff0b8bc593fc4e244de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.882989 10043 ts_tablet_manager.cc:1916] T 21a8fc4a69f3486e92b1a68d10faed38 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.884819 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 54ad199e11dd46f4ac334cac76d070b4 (DEFAULT_TABLE table=default.Table_70172b5479b846bd884c62506727d518 [id=8ead93f650de4e3a83c770f860cb097d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.887356 9609 tablet_service.cc:1505] Processing CreateTablet for tablet b98a836363414da489fb4090abe5d088 (DEFAULT_TABLE table=default.Table_29369a432b8c4b108720c969d77f25b9 [id=7f026c55741d4bceb66eb5c99edf7519]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.887375 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54ad199e11dd46f4ac334cac76d070b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.888458 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b98a836363414da489fb4090abe5d088. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.889089 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 54ad199e11dd46f4ac334cac76d070b4 (DEFAULT_TABLE table=default.Table_70172b5479b846bd884c62506727d518 [id=8ead93f650de4e3a83c770f860cb097d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.889396 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54ad199e11dd46f4ac334cac76d070b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.894469 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 54ad199e11dd46f4ac334cac76d070b4 (DEFAULT_TABLE table=default.Table_70172b5479b846bd884c62506727d518 [id=8ead93f650de4e3a83c770f860cb097d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.894887 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54ad199e11dd46f4ac334cac76d070b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.895388 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6b913111f04c4f1abe4b7244040b07d3" } new_table_name: "default.Table_2592614ec6a74410b7050171ee4d6d1b" modify_external_catalogs: true
I20251027 21:08:00.898116 9742 tablet_service.cc:1505] Processing CreateTablet for tablet b98a836363414da489fb4090abe5d088 (DEFAULT_TABLE table=default.Table_29369a432b8c4b108720c969d77f25b9 [id=7f026c55741d4bceb66eb5c99edf7519]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.898483 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b98a836363414da489fb4090abe5d088. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.904696 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 78884c2be75f45f881f5424d3b3f4e08 with delete_type TABLET_DATA_DELETED (replaced by tablet 4996016747cd484cbda5f11e40e269b8) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.904965 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet f22f013b25074d4db9c7f13caf5b9b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.905110 9754 tablet_service.cc:1552] Processing DeleteTablet for tablet ccaf5c2f8e544842880ba8ba27300ae6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.905249 10747 raft_consensus.cc:493] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.905367 10747 raft_consensus.cc:515] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.905942 10718 raft_consensus.cc:493] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.906082 10718 raft_consensus.cc:515] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.906324 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45568792d7954b498581ce69fb611948" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:00.906390 10040 ts_tablet_manager.cc:1929] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.906490 10040 log.cc:1199] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/ae124514964440c5b88b16ac9f8a2567
I20251027 21:08:00.901898 9305 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_c4824ada6d0f433dbb4770562a66e1ad [id=f2edcf33b0b74923b4ddbbc2419e14ca] alter complete (version 2)
I20251027 21:08:00.906668 10718 leader_election.cc:290] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:00.906760 9305 master_service.cc:954] ReplaceTablet: received request to replace tablet 55a545185c0140e1aa1ddf3fb6ceba9a from {username='slave'} at 127.0.0.1:41710
W20251027 21:08:00.906818 9860 leader_election.cc:343] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.907153 10040 ts_tablet_manager.cc:1950] T ae124514964440c5b88b16ac9f8a2567 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:00.907449 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 679) took 1637 ms. Trace:
W20251027 21:08:00.907728 9710 leader_election.cc:343] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: ccaf5c2f8e544842880ba8ba27300ae6
W20251027 21:08:00.908141 9711 leader_election.cc:343] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: ccaf5c2f8e544842880ba8ba27300ae6
I20251027 21:08:00.908224 9711 leader_election.cc:304] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:00.908340 10032 tablet_bootstrap.cc:492] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:00.908406 10718 raft_consensus.cc:2749] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.908456 10032 ts_tablet_manager.cc:1403] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.331s user 0.069s sys 0.049s
I20251027 21:08:00.908066 10040 rpcz_store.cc:276] 1027 21:07:59.269847 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.269898 (+ 51us) service_pool.cc:225] Handling call
1027 21:08:00.653343 (+1383445us) ts_tablet_manager.cc:1151] Deleting tablet ae124514964440c5b88b16ac9f8a2567
1027 21:08:00.653358 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.660463 (+ 7105us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.906290 (+245827us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.907434 (+ 1144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1383237}
I20251027 21:08:00.909008 10040 tablet_replica.cc:333] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:00.909179 10040 raft_consensus.cc:2243] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.909307 10040 raft_consensus.cc:2272] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.909222 10032 raft_consensus.cc:359] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.909282 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet ae124514964440c5b88b16ac9f8a2567 (table default.Table_11fbadd9eed14cdba7664715e389f7e5 [id=83ec44dba5c444aba48846811a5f5fc9]) successfully deleted
I20251027 21:08:00.909360 10032 raft_consensus.cc:385] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.909405 10032 raft_consensus.cc:740] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.909611 10032 consensus_queue.cc:260] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.909798 10040 ts_tablet_manager.cc:1916] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.910090 10032 ts_tablet_manager.cc:1434] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:00.911860 10032 tablet_bootstrap.cc:492] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:00.913354 10747 leader_election.cc:290] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:00.913586 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45568792d7954b498581ce69fb611948" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:00.913998 9859 leader_election.cc:343] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.914106 9859 leader_election.cc:304] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:00.915561 9895 tablet_service.cc:1505] Processing CreateTablet for tablet b98a836363414da489fb4090abe5d088 (DEFAULT_TABLE table=default.Table_29369a432b8c4b108720c969d77f25b9 [id=7f026c55741d4bceb66eb5c99edf7519]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.917336 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b98a836363414da489fb4090abe5d088. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.918216 10747 raft_consensus.cc:2749] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.918330 10032 tablet_bootstrap.cc:654] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.921620 10747 raft_consensus.cc:493] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:00.921795 10747 raft_consensus.cc:515] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.922391 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b0bf4b9e330b4ec48becdc293dcef72a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:00.922582 10747 leader_election.cc:290] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:00.922926 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0bf4b9e330b4ec48becdc293dcef72a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:00.922917 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5af4607f04f92abcc7292b4176cbd" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:00.923326 9859 leader_election.cc:343] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.924072 10040 ts_tablet_manager.cc:1929] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.924175 10040 log.cc:1199] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/a9cc64ad85334daa9a09538e24ebd8a5
I20251027 21:08:00.924459 10040 ts_tablet_manager.cc:1950] T a9cc64ad85334daa9a09538e24ebd8a5 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:00.924759 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 680) took 1633 ms. Trace:
I20251027 21:08:00.924850 10040 rpcz_store.cc:276] 1027 21:07:59.290902 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.290952 (+ 50us) service_pool.cc:225] Handling call
1027 21:08:00.908968 (+1618016us) ts_tablet_manager.cc:1151] Deleting tablet a9cc64ad85334daa9a09538e24ebd8a5
1027 21:08:00.908977 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.912535 (+ 3558us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.923981 (+ 11446us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.924744 (+ 763us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1617196}
I20251027 21:08:00.925855 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet a9cc64ad85334daa9a09538e24ebd8a5 (table default.Table_03046ffdd6ca47e8aa4cde13574dc119 [id=8a893ef2396748f08f737fddfb76fd65]) successfully deleted
I20251027 21:08:00.925897 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet c400101d2a39490e83722500839198cd with delete_type TABLET_DATA_DELETED (replaced by tablet c9efffa9766e46aca8eb286fc2ad48d2) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.927240 9303 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 87d5af4607f04f92abcc7292b4176cbd (table default.Table_c78ef9ebe86b4da08443e37c83d4e80d [id=cdb023218cda4e9ea04a226b37a220d9]). Expected version 2 got 0
I20251027 21:08:00.927536 10029 tablet_bootstrap.cc:492] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:00.927717 10029 ts_tablet_manager.cc:1403] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.174s user 0.001s sys 0.004s
I20251027 21:08:00.928522 10040 tablet_replica.cc:333] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:00.928694 10040 raft_consensus.cc:2243] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.928632 10029 raft_consensus.cc:359] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.928826 10040 raft_consensus.cc:2272] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.928828 10029 raft_consensus.cc:385] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.928913 10029 raft_consensus.cc:740] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.929179 10029 consensus_queue.cc:260] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.929272 10040 ts_tablet_manager.cc:1916] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.929618 10043 ts_tablet_manager.cc:1929] T 21a8fc4a69f3486e92b1a68d10faed38 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.929661 10029 ts_tablet_manager.cc:1434] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251027 21:08:00.932169 10029 tablet_bootstrap.cc:492] T d8e008c868d04281beba0e5b204eb69a P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:00.933738 10029 tablet_bootstrap.cc:654] T d8e008c868d04281beba0e5b204eb69a P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.940750 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5af4607f04f92abcc7292b4176cbd" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:00.944245 9860 leader_election.cc:343] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:00.944384 9860 leader_election.cc:304] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:00.946972 10040 ts_tablet_manager.cc:1929] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.947086 10040 log.cc:1199] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/94e94cd50c91429d8da9804149501c86
I20251027 21:08:00.949121 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c37baeeee577492c99b99ee09a3781db" } new_table_name: "default.Table_4567c3392eb24748840827eb96835642" modify_external_catalogs: true
I20251027 21:08:00.949302 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 6a5071cb8c5a4ef59286b5970c68845b (DEFAULT_TABLE table=default.Table_3bdec7059cee4e748b8e00c5ebdb800a [id=0994fe17ddf249058b68dc2aa110d49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.949753 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a5071cb8c5a4ef59286b5970c68845b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.950238 10747 raft_consensus.cc:2749] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:00.950549 10041 tablet_replica.cc:333] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:00.950752 10041 raft_consensus.cc:2243] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.950953 10041 raft_consensus.cc:2272] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.951510 10041 ts_tablet_manager.cc:1916] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.952445 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet c400101d2a39490e83722500839198cd with delete_type TABLET_DATA_DELETED (replaced by tablet c9efffa9766e46aca8eb286fc2ad48d2) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:00.952653 10040 ts_tablet_manager.cc:1950] T 94e94cd50c91429d8da9804149501c86 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:00.953330 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 681) took 1592 ms. Trace:
I20251027 21:08:00.953411 10040 rpcz_store.cc:276] 1027 21:07:59.360822 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.360880 (+ 58us) service_pool.cc:225] Handling call
1027 21:08:00.928474 (+1567594us) ts_tablet_manager.cc:1151] Deleting tablet 94e94cd50c91429d8da9804149501c86
1027 21:08:00.928484 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.936639 (+ 8155us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.946883 (+ 10244us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.953314 (+ 6431us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1567368}
I20251027 21:08:00.953850 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 94e94cd50c91429d8da9804149501c86 (table default.Table_b23313ec4d7b4e32a4283dd404578616 [id=6a545a959eb74d0eb589f1be92a0bc0a]) successfully deleted
I20251027 21:08:00.954447 10040 tablet_replica.cc:333] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:00.954614 10040 raft_consensus.cc:2243] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.954742 10040 raft_consensus.cc:2272] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.955211 10040 ts_tablet_manager.cc:1916] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.958489 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 6a5071cb8c5a4ef59286b5970c68845b (DEFAULT_TABLE table=default.Table_3bdec7059cee4e748b8e00c5ebdb800a [id=0994fe17ddf249058b68dc2aa110d49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.958899 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a5071cb8c5a4ef59286b5970c68845b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.962929 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 842c1a1298234d9ea2719bab29c54ae3
I20251027 21:08:00.963557 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 842c1a1298234d9ea2719bab29c54ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:00.964562 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 842c1a1298234d9ea2719bab29c54ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:00.929761 10043 log.cc:1199] T 21a8fc4a69f3486e92b1a68d10faed38 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/21a8fc4a69f3486e92b1a68d10faed38
I20251027 21:08:00.965180 10043 ts_tablet_manager.cc:1950] T 21a8fc4a69f3486e92b1a68d10faed38 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:00.966181 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 21a8fc4a69f3486e92b1a68d10faed38 (table default.Table_ba1d973c05e14c70944d76ff1ecbc1bc [id=490e0e5ec83b4f76bf71eb41c900ac06]) successfully deleted
W20251027 21:08:00.966845 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 211ac01a37f8434f95aa6a63970296fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 211ac01a37f8434f95aa6a63970296fa already in progress: creating tablet
W20251027 21:08:00.967458 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 4ba13a89a33549af921d3b55023faf15 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ba13a89a33549af921d3b55023faf15 already in progress: creating tablet
W20251027 21:08:00.968093 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 933b33aac5354778a5866d340fd34db4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 933b33aac5354778a5866d340fd34db4 already in progress: creating tablet
I20251027 21:08:00.968396 9309 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 87d5af4607f04f92abcc7292b4176cbd (table default.Table_c78ef9ebe86b4da08443e37c83d4e80d [id=cdb023218cda4e9ea04a226b37a220d9]). Expected version 2 got 0
I20251027 21:08:00.968650 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 842c1a1298234d9ea2719bab29c54ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:00.969533 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3ac9f7acd5ac43f1a585c29ecae682eb" } new_table_name: "default.Table_06c15ff076cd4c24bc49b16b01bff965" modify_external_catalogs: true
I20251027 21:08:00.971978 10032 tablet_bootstrap.cc:492] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:00.972139 10032 ts_tablet_manager.cc:1403] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.000s
I20251027 21:08:00.974987 10031 tablet_bootstrap.cc:492] T d8e008c868d04281beba0e5b204eb69a P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:00.975153 10031 ts_tablet_manager.cc:1403] T d8e008c868d04281beba0e5b204eb69a P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.170s user 0.016s sys 0.093s
I20251027 21:08:00.975898 10031 raft_consensus.cc:359] T d8e008c868d04281beba0e5b204eb69a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:00.976104 10031 raft_consensus.cc:385] T d8e008c868d04281beba0e5b204eb69a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.976184 10031 raft_consensus.cc:740] T d8e008c868d04281beba0e5b204eb69a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.976459 10031 consensus_queue.cc:260] T d8e008c868d04281beba0e5b204eb69a P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:08:00.977897 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet a8c46c2a01f946aea8b85875115f7f37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8c46c2a01f946aea8b85875115f7f37 already in progress: creating tablet
I20251027 21:08:00.979008 10031 ts_tablet_manager.cc:1434] T d8e008c868d04281beba0e5b204eb69a P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251027 21:08:00.979513 10031 tablet_bootstrap.cc:492] T 21a8fc4a69f3486e92b1a68d10faed38 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:08:00.979624 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet e9b8f76c33444bcb9b488af65e4d3e6b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9b8f76c33444bcb9b488af65e4d3e6b already in progress: creating tablet
I20251027 21:08:00.979997 10043 tablet_replica.cc:333] T a287439ea6c9469da5fb56d84ee68048 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:00.980233 10043 raft_consensus.cc:2243] T a287439ea6c9469da5fb56d84ee68048 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:00.980449 10043 raft_consensus.cc:2272] T a287439ea6c9469da5fb56d84ee68048 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:00.983740 10032 raft_consensus.cc:359] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.983916 10032 raft_consensus.cc:385] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:00.984016 10032 raft_consensus.cc:740] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.984308 10031 tablet_bootstrap.cc:654] T 21a8fc4a69f3486e92b1a68d10faed38 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.984256 10032 consensus_queue.cc:260] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:00.984757 10032 ts_tablet_manager.cc:1434] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251027 21:08:00.985456 10032 tablet_bootstrap.cc:492] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:00.986733 10043 ts_tablet_manager.cc:1916] T a287439ea6c9469da5fb56d84ee68048 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.987214 10032 tablet_bootstrap.cc:654] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:00.990496 10040 ts_tablet_manager.cc:1929] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.991580 10040 log.cc:1199] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/77797eea3e9c4f8099dc9d58f116200b
I20251027 21:08:00.991962 10040 ts_tablet_manager.cc:1950] T 77797eea3e9c4f8099dc9d58f116200b P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:00.992307 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 682) took 1598 ms. Trace:
I20251027 21:08:00.992697 10040 rpcz_store.cc:276] 1027 21:07:59.393473 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.393520 (+ 47us) service_pool.cc:225] Handling call
1027 21:08:00.954402 (+1560882us) ts_tablet_manager.cc:1151] Deleting tablet 77797eea3e9c4f8099dc9d58f116200b
1027 21:08:00.954411 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.987385 (+ 32974us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.990412 (+ 3027us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:00.992293 (+ 1881us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1560662}
I20251027 21:08:00.993397 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 77797eea3e9c4f8099dc9d58f116200b (table default.Table_ed23972576b74b5fbc7167cf7b541bfc [id=e7b7bdb2ea7f468c98bfe88f900a5be8]) successfully deleted
I20251027 21:08:00.993880 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 686) took 1501 ms. Trace:
I20251027 21:08:00.994298 10040 rpcz_store.cc:276] 1027 21:07:59.492821 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.492884 (+ 63us) service_pool.cc:225] Handling call
1027 21:08:00.993720 (+1500836us) ts_tablet_manager.cc:1151] Deleting tablet 0820648719a34f379507efced9cc434b
1027 21:08:00.993731 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.993862 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1500648}
W20251027 21:08:00.995039 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 0820648719a34f379507efced9cc434b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0820648719a34f379507efced9cc434b already in progress: creating tablet
I20251027 21:08:00.995532 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 688) took 1485 ms. Trace:
I20251027 21:08:00.995687 10041 ts_tablet_manager.cc:1929] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:00.995941 10040 rpcz_store.cc:276] 1027 21:07:59.510268 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.510322 (+ 54us) service_pool.cc:225] Handling call
1027 21:08:00.995409 (+1485087us) ts_tablet_manager.cc:1151] Deleting tablet 113a5d30d4b3490187681383d4fef325
1027 21:08:00.995420 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:00.995516 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1484820}
W20251027 21:08:00.996356 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 113a5d30d4b3490187681383d4fef325 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 113a5d30d4b3490187681383d4fef325 already in progress: creating tablet
I20251027 21:08:00.996868 9600 tablet_service.cc:1505] Processing CreateTablet for tablet d042a15408484fbc9ee9666476b103d5 (DEFAULT_TABLE table=default.Table_9eb69e3772794bcf8f0662a43bed4bd1 [id=1a02c0c2354d43b9b41d2f007dbff3c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.997138 9749 tablet_service.cc:1505] Processing CreateTablet for tablet d042a15408484fbc9ee9666476b103d5 (DEFAULT_TABLE table=default.Table_9eb69e3772794bcf8f0662a43bed4bd1 [id=1a02c0c2354d43b9b41d2f007dbff3c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.997274 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d042a15408484fbc9ee9666476b103d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:00.997895 9755 tablet_service.cc:1505] Processing CreateTablet for tablet eb35559220434ef48dd2a8e07d2ef04a (DEFAULT_TABLE table=default.Table_c9eb5134bf16410ca17a6600553250d4 [id=ad43237dd8b946f5ba4759bb77d6b8c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.998602 9750 tablet_service.cc:1505] Processing CreateTablet for tablet fb385f17054e430bbd69a2c9d22944ca (DEFAULT_TABLE table=default.Table_2088d599bbe444a18dee497154bb4379 [id=35b20c53f9ce4ae1847268757f60c488]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:00.998920 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb385f17054e430bbd69a2c9d22944ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.003186 9615 tablet_service.cc:1505] Processing CreateTablet for tablet fb385f17054e430bbd69a2c9d22944ca (DEFAULT_TABLE table=default.Table_2088d599bbe444a18dee497154bb4379 [id=35b20c53f9ce4ae1847268757f60c488]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.003535 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb385f17054e430bbd69a2c9d22944ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.005316 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb35559220434ef48dd2a8e07d2ef04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.003199 9609 tablet_service.cc:1505] Processing CreateTablet for tablet eb35559220434ef48dd2a8e07d2ef04a (DEFAULT_TABLE table=default.Table_c9eb5134bf16410ca17a6600553250d4 [id=ad43237dd8b946f5ba4759bb77d6b8c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.006556 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb35559220434ef48dd2a8e07d2ef04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.007158 10718 raft_consensus.cc:493] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:01.007263 10718 raft_consensus.cc:515] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.007794 10718 leader_election.cc:290] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:01.008170 10529 raft_consensus.cc:493] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:01.008278 10529 raft_consensus.cc:515] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:08:01.008687 9711 leader_election.cc:343] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: f22f013b25074d4db9c7f13caf5b9b42
I20251027 21:08:01.008816 10529 leader_election.cc:290] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:01.009474 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c400101d2a39490e83722500839198cd" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:01.009639 9769 raft_consensus.cc:2468] T c400101d2a39490e83722500839198cd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:01.010457 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f22f013b25074d4db9c7f13caf5b9b42" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:01.010515 9566 leader_election.cc:304] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:01.010608 9632 raft_consensus.cc:2468] T f22f013b25074d4db9c7f13caf5b9b42 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
W20251027 21:08:01.011015 9568 leader_election.cc:343] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: c400101d2a39490e83722500839198cd
I20251027 21:08:01.011405 10529 raft_consensus.cc:2804] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:01.011482 10529 raft_consensus.cc:493] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:01.011554 10529 raft_consensus.cc:3060] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:01.011513 9710 leader_election.cc:304] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:01.012167 10718 raft_consensus.cc:2804] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:01.012236 10718 raft_consensus.cc:493] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:01.012288 10718 raft_consensus.cc:3060] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:01.013197 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d042a15408484fbc9ee9666476b103d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.014834 10529 raft_consensus.cc:515] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.014839 10718 raft_consensus.cc:515] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.015357 10718 leader_election.cc:290] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:01.015995 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c400101d2a39490e83722500839198cd" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:01.016011 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f22f013b25074d4db9c7f13caf5b9b42" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
W20251027 21:08:01.016047 9711 leader_election.cc:343] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: f22f013b25074d4db9c7f13caf5b9b42
I20251027 21:08:01.016153 9632 raft_consensus.cc:3060] T f22f013b25074d4db9c7f13caf5b9b42 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:01.016151 9769 raft_consensus.cc:3060] T c400101d2a39490e83722500839198cd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:01.018651 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 689) took 1500 ms. Trace:
I20251027 21:08:01.018774 9769 raft_consensus.cc:2468] T c400101d2a39490e83722500839198cd P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:01.018724 10040 rpcz_store.cc:276] 1027 21:07:59.517722 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.517773 (+ 51us) service_pool.cc:225] Handling call
1027 21:08:01.018537 (+1500764us) ts_tablet_manager.cc:1151] Deleting tablet d2203ee62f7e404c95c5f86b5ae85feb
1027 21:08:01.018546 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:01.018634 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1500535}
I20251027 21:08:01.019260 9632 raft_consensus.cc:2468] T f22f013b25074d4db9c7f13caf5b9b42 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
W20251027 21:08:01.019364 9568 leader_election.cc:343] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: c400101d2a39490e83722500839198cd
W20251027 21:08:01.019538 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet d2203ee62f7e404c95c5f86b5ae85feb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2203ee62f7e404c95c5f86b5ae85feb already in progress: creating tablet
I20251027 21:08:01.019711 9710 leader_election.cc:304] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:01.019858 9566 leader_election.cc:304] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:01.020105 10681 raft_consensus.cc:2804] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:01.020115 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 691) took 1498 ms. Trace:
I20251027 21:08:01.020177 10040 rpcz_store.cc:276] 1027 21:07:59.521517 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.521568 (+ 51us) service_pool.cc:225] Handling call
1027 21:08:01.020008 (+1498440us) ts_tablet_manager.cc:1151] Deleting tablet e2ff7fc3c1fa4dbd9a5c38d3219b03fa
1027 21:08:01.020016 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:01.020101 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1498260}
I20251027 21:08:01.020387 10529 leader_election.cc:290] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:01.020675 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet e2ff7fc3c1fa4dbd9a5c38d3219b03fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2ff7fc3c1fa4dbd9a5c38d3219b03fa already in progress: creating tablet
I20251027 21:08:01.021299 9755 tablet_service.cc:1505] Processing CreateTablet for tablet f99aaa7daa4348fba71c392eafa4a188 (DEFAULT_TABLE table=default.Table_ee755fc680144410828dd30c2c19b0bc [id=7e75ac95183349c6b5ca64abcba001f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.024537 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 690) took 1503 ms. Trace:
I20251027 21:08:01.025033 10040 rpcz_store.cc:276] 1027 21:07:59.521357 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.522037 (+ 680us) service_pool.cc:225] Handling call
1027 21:08:01.024406 (+1502369us) ts_tablet_manager.cc:1151] Deleting tablet 849b3e9fc34045939a7a678fa8269d7f
1027 21:08:01.024418 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:01.024516 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1502143}
W20251027 21:08:01.026083 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 849b3e9fc34045939a7a678fa8269d7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 849b3e9fc34045939a7a678fa8269d7f already in progress: creating tablet
I20251027 21:08:01.026558 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 702) took 1317 ms. Trace:
I20251027 21:08:01.024637 10718 raft_consensus.cc:2804] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:01.026973 10040 rpcz_store.cc:276] 1027 21:07:59.709277 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.709726 (+ 449us) service_pool.cc:225] Handling call
1027 21:08:01.026454 (+1316728us) ts_tablet_manager.cc:1151] Deleting tablet c4996d8fae094db8a96599b9057f1649
1027 21:08:01.026465 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:01.026541 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1316507}
W20251027 21:08:01.027861 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c4996d8fae094db8a96599b9057f1649 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4996d8fae094db8a96599b9057f1649 already in progress: creating tablet
I20251027 21:08:01.028384 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 703) took 1275 ms. Trace:
I20251027 21:08:01.028893 10040 rpcz_store.cc:276] 1027 21:07:59.753035 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.753089 (+ 54us) service_pool.cc:225] Handling call
1027 21:08:01.028266 (+1275177us) ts_tablet_manager.cc:1151] Deleting tablet 83a040332f994055a9ecf4c66720b7e1
1027 21:08:01.028278 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:01.028366 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1274918}
W20251027 21:08:01.029845 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 83a040332f994055a9ecf4c66720b7e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83a040332f994055a9ecf4c66720b7e1 already in progress: creating tablet
I20251027 21:08:01.030323 10040 tablet_replica.cc:333] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:01.031385 10040 raft_consensus.cc:2243] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.032146 10040 raft_consensus.cc:2272] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.032480 9615 tablet_service.cc:1505] Processing CreateTablet for tablet f99aaa7daa4348fba71c392eafa4a188 (DEFAULT_TABLE table=default.Table_ee755fc680144410828dd30c2c19b0bc [id=7e75ac95183349c6b5ca64abcba001f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.033485 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f99aaa7daa4348fba71c392eafa4a188. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.034548 10040 ts_tablet_manager.cc:1916] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:00.996363 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 6a5071cb8c5a4ef59286b5970c68845b (DEFAULT_TABLE table=default.Table_3bdec7059cee4e748b8e00c5ebdb800a [id=0994fe17ddf249058b68dc2aa110d49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.038625 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a5071cb8c5a4ef59286b5970c68845b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.041571 10029 tablet_bootstrap.cc:492] T d8e008c868d04281beba0e5b204eb69a P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:01.041683 10029 ts_tablet_manager.cc:1403] T d8e008c868d04281beba0e5b204eb69a P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.110s user 0.002s sys 0.003s
I20251027 21:08:00.997341 9896 tablet_service.cc:1505] Processing CreateTablet for tablet d042a15408484fbc9ee9666476b103d5 (DEFAULT_TABLE table=default.Table_9eb69e3772794bcf8f0662a43bed4bd1 [id=1a02c0c2354d43b9b41d2f007dbff3c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.042450 10029 raft_consensus.cc:359] T d8e008c868d04281beba0e5b204eb69a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.042596 10029 raft_consensus.cc:385] T d8e008c868d04281beba0e5b204eb69a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.042645 10029 raft_consensus.cc:740] T d8e008c868d04281beba0e5b204eb69a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.042807 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d042a15408484fbc9ee9666476b103d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.043408 10029 consensus_queue.cc:260] T d8e008c868d04281beba0e5b204eb69a P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.044268 10029 ts_tablet_manager.cc:1434] T d8e008c868d04281beba0e5b204eb69a P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251027 21:08:01.044514 10029 tablet_bootstrap.cc:492] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:01.044636 10032 tablet_bootstrap.cc:492] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:01.044770 10032 ts_tablet_manager.cc:1403] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.000s
I20251027 21:08:01.045600 10032 raft_consensus.cc:359] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.045780 10032 raft_consensus.cc:385] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.045806 10029 tablet_bootstrap.cc:654] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.045842 10032 raft_consensus.cc:740] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:00.998051 9898 tablet_service.cc:1505] Processing CreateTablet for tablet eb35559220434ef48dd2a8e07d2ef04a (DEFAULT_TABLE table=default.Table_c9eb5134bf16410ca17a6600553250d4 [id=ad43237dd8b946f5ba4759bb77d6b8c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.046716 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb35559220434ef48dd2a8e07d2ef04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.046069 10032 consensus_queue.cc:260] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.049017 10032 ts_tablet_manager.cc:1434] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251027 21:08:01.049378 10032 tablet_bootstrap.cc:492] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:01.052690 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet bdd382c4d376484cb7767d30681f966f
I20251027 21:08:01.052785 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e04f4c5cf724419ab0be0e84d30a9a57
I20251027 21:08:01.054306 9888 tablet_service.cc:1552] Processing DeleteTablet for tablet bdd382c4d376484cb7767d30681f966f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.054605 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet bdd382c4d376484cb7767d30681f966f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.054878 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet e04f4c5cf724419ab0be0e84d30a9a57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.057359 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet e04f4c5cf724419ab0be0e84d30a9a57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.057891 10718 raft_consensus.cc:697] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:01.058229 10718 consensus_queue.cc:237] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.058894 10681 raft_consensus.cc:697] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:01.059329 10681 consensus_queue.cc:237] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.061057 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet ae0ee7cd2c1b406f84285f25420588d0
I20251027 21:08:01.061817 9888 tablet_service.cc:1552] Processing DeleteTablet for tablet ae0ee7cd2c1b406f84285f25420588d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.063813 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet ae0ee7cd2c1b406f84285f25420588d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.066706 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet ae0ee7cd2c1b406f84285f25420588d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.067662 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet bdd382c4d376484cb7767d30681f966f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.072332 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet e04f4c5cf724419ab0be0e84d30a9a57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.072649 10041 log.cc:1199] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/adaa4f66bbab491f8030d439d85cc7d1
I20251027 21:08:01.073164 10041 ts_tablet_manager.cc:1950] T adaa4f66bbab491f8030d439d85cc7d1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:01.073601 10032 tablet_bootstrap.cc:654] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.074508 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet adaa4f66bbab491f8030d439d85cc7d1 (table default.Table_3f5e27698f4c4da88d9bacc9dd719304 [id=d174e821ea8641fc8b6658d5c61c7e8a]) successfully deleted
W20251027 21:08:01.075353 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 933b33aac5354778a5866d340fd34db4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 933b33aac5354778a5866d340fd34db4 already in progress: creating tablet
W20251027 21:08:01.076121 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 4ba13a89a33549af921d3b55023faf15 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ba13a89a33549af921d3b55023faf15 already in progress: creating tablet
W20251027 21:08:01.076817 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet a8c46c2a01f946aea8b85875115f7f37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8c46c2a01f946aea8b85875115f7f37 already in progress: creating tablet
I20251027 21:08:01.077145 10041 tablet_replica.cc:333] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:01.077400 10041 raft_consensus.cc:2243] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.077656 10041 raft_consensus.cc:2272] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.078487 10041 ts_tablet_manager.cc:1916] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.025326 9899 tablet_service.cc:1505] Processing CreateTablet for tablet f99aaa7daa4348fba71c392eafa4a188 (DEFAULT_TABLE table=default.Table_ee755fc680144410828dd30c2c19b0bc [id=7e75ac95183349c6b5ca64abcba001f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.082572 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f99aaa7daa4348fba71c392eafa4a188. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.021307 9895 tablet_service.cc:1505] Processing CreateTablet for tablet fb385f17054e430bbd69a2c9d22944ca (DEFAULT_TABLE table=default.Table_2088d599bbe444a18dee497154bb4379 [id=35b20c53f9ce4ae1847268757f60c488]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.090466 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb385f17054e430bbd69a2c9d22944ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.093262 10031 tablet_bootstrap.cc:492] T 21a8fc4a69f3486e92b1a68d10faed38 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:01.093483 10031 ts_tablet_manager.cc:1403] T 21a8fc4a69f3486e92b1a68d10faed38 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.114s user 0.015s sys 0.034s
I20251027 21:08:01.094652 10031 raft_consensus.cc:359] T 21a8fc4a69f3486e92b1a68d10faed38 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.094945 10031 raft_consensus.cc:385] T 21a8fc4a69f3486e92b1a68d10faed38 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.095080 10031 raft_consensus.cc:740] T 21a8fc4a69f3486e92b1a68d10faed38 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.095388 10031 consensus_queue.cc:260] T 21a8fc4a69f3486e92b1a68d10faed38 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.096508 10031 ts_tablet_manager.cc:1434] T 21a8fc4a69f3486e92b1a68d10faed38 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:08:01.097081 10031 tablet_bootstrap.cc:492] T 132a01405a63447ab27b2137b803224f P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:01.097925 10032 tablet_bootstrap.cc:492] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:01.098055 10032 ts_tablet_manager.cc:1403] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.049s user 0.001s sys 0.002s
I20251027 21:08:01.098845 10032 raft_consensus.cc:359] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.099035 10032 raft_consensus.cc:385] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.099109 10032 raft_consensus.cc:740] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.099612 10032 consensus_queue.cc:260] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.100519 10032 ts_tablet_manager.cc:1434] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:01.100821 10032 tablet_bootstrap.cc:492] T dd7029986a5246ee823e5fdb50683118 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:01.100809 10031 tablet_bootstrap.cc:654] T 132a01405a63447ab27b2137b803224f P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.106717 10029 tablet_bootstrap.cc:492] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:01.106884 10029 ts_tablet_manager.cc:1403] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.062s user 0.000s sys 0.004s
I20251027 21:08:01.107820 10029 raft_consensus.cc:359] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.108021 10029 raft_consensus.cc:385] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.108084 10029 raft_consensus.cc:740] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.108522 10029 consensus_queue.cc:260] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.108996 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f99aaa7daa4348fba71c392eafa4a188. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.112741 10043 ts_tablet_manager.cc:1929] T a287439ea6c9469da5fb56d84ee68048 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.114387 10032 tablet_bootstrap.cc:654] T dd7029986a5246ee823e5fdb50683118 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.122452 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 55a545185c0140e1aa1ddf3fb6ceba9a
I20251027 21:08:01.123008 9305 catalog_manager.cc:6719] ReplaceTablet: tablet 55a545185c0140e1aa1ddf3fb6ceba9a deleted and replaced by tablet 8885465e81444b468b96955f0e44e9d2
I20251027 21:08:01.123229 9308 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 345d3795a37b4a40b40967cfacf0f893 (table default.Table_ef79e4d09fb741429811f81e1efd90cb [id=93d6307da3a34f87a67018a06825cd15]). Expected version 2 got 0
I20251027 21:08:01.124280 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_ba2ceb40c6f745c099ab46df479c912c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.124922 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba2ceb40c6f745c099ab46df479c912c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.125059 9305 master_service.cc:954] ReplaceTablet: received request to replace tablet e5d8837ff3c44a66a1f933d80540ba0c from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:01.126281 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_048e52e3f32d46a1a3d1619888efc5ae" } new_table_name: "default.Table_fbc3168ad4454ab3a6a83c1226e329f5" modify_external_catalogs: true
I20251027 21:08:01.126240 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_456ae12ae5d04554b252e71a72c3d9c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.126791 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_456ae12ae5d04554b252e71a72c3d9c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.128171 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 55a545185c0140e1aa1ddf3fb6ceba9a with delete_type TABLET_DATA_DELETED (replaced by tablet 8885465e81444b468b96955f0e44e9d2) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.131258 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_48f4510204084408aef690097faeaca1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.131686 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48f4510204084408aef690097faeaca1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.132633 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c5e35fa342f64432b4654f177b365624"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.133044 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5e35fa342f64432b4654f177b365624 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.133931 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3e9f6e0f115040639f405946e55a2637" } modify_external_catalogs: true
I20251027 21:08:01.134078 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 55a545185c0140e1aa1ddf3fb6ceba9a with delete_type TABLET_DATA_DELETED (replaced by tablet 8885465e81444b468b96955f0e44e9d2) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.134095 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3e9f6e0f115040639f405946e55a2637" } modify_external_catalogs: true
I20251027 21:08:01.137228 10032 tablet_bootstrap.cc:492] T dd7029986a5246ee823e5fdb50683118 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:01.137362 10032 ts_tablet_manager.cc:1403] T dd7029986a5246ee823e5fdb50683118 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20251027 21:08:01.138065 10032 raft_consensus.cc:359] T dd7029986a5246ee823e5fdb50683118 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.138257 10032 raft_consensus.cc:385] T dd7029986a5246ee823e5fdb50683118 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.138320 10032 raft_consensus.cc:740] T dd7029986a5246ee823e5fdb50683118 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.138530 10032 consensus_queue.cc:260] T dd7029986a5246ee823e5fdb50683118 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.139041 10032 ts_tablet_manager.cc:1434] T dd7029986a5246ee823e5fdb50683118 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:01.139446 10032 tablet_bootstrap.cc:492] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:01.140931 10032 tablet_bootstrap.cc:654] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.143888 10747 raft_consensus.cc:493] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:01.144011 10747 raft_consensus.cc:515] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.144547 10747 leader_election.cc:290] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:01.152561 10040 ts_tablet_manager.cc:1929] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.152683 10040 log.cc:1199] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/49d9e5c5960a4534a6603875006e6266
I20251027 21:08:01.153007 10040 ts_tablet_manager.cc:1950] T 49d9e5c5960a4534a6603875006e6266 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:01.153522 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d872cf3ef39042ff9a615b55e7195afa" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:01.154268 9859 leader_election.cc:343] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:01.157828 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe7b00e9cf984d12817c9ea860391199 consensus state has no leader
W20251027 21:08:01.157922 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 334c580862bd4bf09be3f1710aa6a092 consensus state has no leader
I20251027 21:08:01.158273 10043 log.cc:1199] T a287439ea6c9469da5fb56d84ee68048 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/a287439ea6c9469da5fb56d84ee68048
I20251027 21:08:01.123492 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet 55a545185c0140e1aa1ddf3fb6ceba9a with delete_type TABLET_DATA_DELETED (replaced by tablet 8885465e81444b468b96955f0e44e9d2) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.158573 10043 ts_tablet_manager.cc:1950] T a287439ea6c9469da5fb56d84ee68048 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:01.145262 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d872cf3ef39042ff9a615b55e7195afa" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:01.159189 9860 leader_election.cc:343] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:01.159322 9860 leader_election.cc:304] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:01.160445 10747 raft_consensus.cc:2749] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:01.169061 10041 ts_tablet_manager.cc:1929] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.173259 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet d8e008c868d04281beba0e5b204eb69a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.173496 10041 log.cc:1199] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/a9bf3a093be8498c9dd5511b53436b16
I20251027 21:08:01.173902 10041 ts_tablet_manager.cc:1950] T a9bf3a093be8498c9dd5511b53436b16 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:01.174898 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet a9bf3a093be8498c9dd5511b53436b16 (table default.Table_627b1e3af2d34bc9b4508237abf020f5 [id=5c9fd870185a4211a5a956171f491cc0]) successfully deleted
W20251027 21:08:01.175889 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d2d977e360b4ed29c9ed63a309783d6 consensus state has no leader
W20251027 21:08:01.175604 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet e9b8f76c33444bcb9b488af65e4d3e6b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9b8f76c33444bcb9b488af65e4d3e6b already in progress: creating tablet
I20251027 21:08:01.176520 9309 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 345d3795a37b4a40b40967cfacf0f893 (table default.Table_ef79e4d09fb741429811f81e1efd90cb [id=93d6307da3a34f87a67018a06825cd15]). Expected version 2 got 0
I20251027 21:08:01.176590 10041 tablet_replica.cc:333] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:01.176784 10041 raft_consensus.cc:2243] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.176934 10041 raft_consensus.cc:2272] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.177152 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_56d94ef7571b42b4ba4e66f6ce476ead"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:01.177400 10041 ts_tablet_manager.cc:1916] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:01.177630 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56d94ef7571b42b4ba4e66f6ce476ead in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.178402 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet d872cf3ef39042ff9a615b55e7195afa with delete_type TABLET_DATA_DELETED (replaced by tablet 6f901626832f4adb948ccb65f8573e86) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.178383 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_cff1f8d7488047e3b7f4608d376e6f5b" } new_table_name: "default.Table_42f66392d92c497db0755186643e6486" modify_external_catalogs: true
I20251027 21:08:01.179579 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet a287439ea6c9469da5fb56d84ee68048 (table default.Table_1ecaac632a024248b04e7c40e2c01d2c [id=4e2210c84e4c464da017d1b36ddb0a37]) successfully deleted
I20251027 21:08:01.179653 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b94af6879cd940829a7f531277d5544c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.180217 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b94af6879cd940829a7f531277d5544c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:01.180305 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet c25be263facd42c5bb6fa421160fbef0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c25be263facd42c5bb6fa421160fbef0 already in progress: creating tablet
I20251027 21:08:01.180603 10043 tablet_replica.cc:333] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:01.180840 10043 raft_consensus.cc:2243] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.181016 10043 raft_consensus.cc:2272] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.181483 10725 tablet.cc:1722] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:08:01.181573 10043 ts_tablet_manager.cc:1916] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.181854 10747 raft_consensus.cc:493] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:01.181964 10747 raft_consensus.cc:515] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.182484 10747 leader_election.cc:290] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:01.182849 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet 84b9de82d55447bba2a897edb683e5c5 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:01.183490 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b85311fce5904b9fa4e6bc7c318bbd47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:01.183787 10740 tablet.cc:1722] T e5d8837ff3c44a66a1f933d80540ba0c P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251027 21:08:01.183863 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b85311fce5904b9fa4e6bc7c318bbd47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.186394 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet d8e008c868d04281beba0e5b204eb69a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.188679 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "630ae4d66f99474382814cc32cc60ea0" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:01.189877 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet dd7029986a5246ee823e5fdb50683118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:08:01.190164 9860 leader_election.cc:343] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:01.191262 9309 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet a9339c6b1154400eace53d765528fdf9 (table default.Table_5a6211c8465040dbad1cc86377a370ea [id=91fb9f95fe9343fca1d8cb0d0f781afb]). Expected version 2 got 0
I20251027 21:08:01.191365 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "630ae4d66f99474382814cc32cc60ea0" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:01.191793 9859 leader_election.cc:343] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:01.191902 9859 leader_election.cc:304] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:01.193061 10747 raft_consensus.cc:2749] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:01.193625 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_691240a058cc4f879e5737f73a4c12e7" } modify_external_catalogs: true
I20251027 21:08:01.193826 10043 ts_tablet_manager.cc:1929] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.193953 10043 log.cc:1199] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/6f1129eecfb54448965222269d9c32b7
I20251027 21:08:01.193837 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_691240a058cc4f879e5737f73a4c12e7" } modify_external_catalogs: true
I20251027 21:08:01.194278 10043 ts_tablet_manager.cc:1950] T 6f1129eecfb54448965222269d9c32b7 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:01.194370 10031 tablet_bootstrap.cc:492] T 132a01405a63447ab27b2137b803224f P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:01.194512 10031 ts_tablet_manager.cc:1403] T 132a01405a63447ab27b2137b803224f P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.098s user 0.003s sys 0.039s
I20251027 21:08:01.195442 10031 raft_consensus.cc:359] T 132a01405a63447ab27b2137b803224f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.195632 10031 raft_consensus.cc:385] T 132a01405a63447ab27b2137b803224f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.195694 10031 raft_consensus.cc:740] T 132a01405a63447ab27b2137b803224f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.195969 10031 consensus_queue.cc:260] T 132a01405a63447ab27b2137b803224f P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.196710 10031 ts_tablet_manager.cc:1434] T 132a01405a63447ab27b2137b803224f P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:01.197228 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 6f1129eecfb54448965222269d9c32b7 (table default.Table_f16bc07fcb4c48b7969d13b8341aefdc [id=b5f4e2be1a5e42f58d98b64b8700ef1a]) successfully deleted
W20251027 21:08:01.197721 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 7c6c2292ad0e4066958897fff01370af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c6c2292ad0e4066958897fff01370af already in progress: creating tablet
W20251027 21:08:01.198444 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 7a28dc9fe75744e5aa3d486afbe29561 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a28dc9fe75744e5aa3d486afbe29561 already in progress: creating tablet
I20251027 21:08:01.200881 10029 ts_tablet_manager.cc:1434] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.094s user 0.000s sys 0.002s
W20251027 21:08:01.201709 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet d1d16726537a473ebd731e7a1f448400 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1d16726537a473ebd731e7a1f448400 already in progress: creating tablet
W20251027 21:08:01.202472 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet fb385f17054e430bbd69a2c9d22944ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb385f17054e430bbd69a2c9d22944ca already in progress: creating tablet
W20251027 21:08:01.205130 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 622fa1b3ff134f2d8a087ba13a85e597 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 622fa1b3ff134f2d8a087ba13a85e597 already in progress: creating tablet
I20251027 21:08:01.207996 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 695) took 1604 ms. Trace:
I20251027 21:08:01.208083 10040 rpcz_store.cc:276] 1027 21:07:59.603993 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.772723 (+168730us) service_pool.cc:225] Handling call
1027 21:08:01.030270 (+1257547us) ts_tablet_manager.cc:1151] Deleting tablet 49d9e5c5960a4534a6603875006e6266
1027 21:08:01.030282 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:01.113124 (+ 82842us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:01.152434 (+ 39310us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:01.207977 (+ 55543us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1257293}
I20251027 21:08:01.208396 10040 tablet_replica.cc:333] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:01.208564 10040 raft_consensus.cc:2243] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.208715 10040 raft_consensus.cc:2272] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.209201 10040 ts_tablet_manager.cc:1916] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.210242 10029 tablet_bootstrap.cc:492] T fe24af9838954a9bb680474800642e62 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:01.205928 10043 tablet_replica.cc:333] T f22f013b25074d4db9c7f13caf5b9b42 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:01.211223 10043 raft_consensus.cc:2243] T f22f013b25074d4db9c7f13caf5b9b42 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.211412 10043 raft_consensus.cc:2272] T f22f013b25074d4db9c7f13caf5b9b42 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.211675 10029 tablet_bootstrap.cc:654] T fe24af9838954a9bb680474800642e62 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.212033 10043 ts_tablet_manager.cc:1916] T f22f013b25074d4db9c7f13caf5b9b42 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.214421 10031 tablet_bootstrap.cc:492] T 7aa8c538b6c849a28a28ffe706306891 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:01.214623 10041 ts_tablet_manager.cc:1929] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.215140 10041 log.cc:1199] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/648f506ec87c4fb8a3f6d4591873dbb8
I20251027 21:08:01.216060 10031 tablet_bootstrap.cc:654] T 7aa8c538b6c849a28a28ffe706306891 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.216446 10041 ts_tablet_manager.cc:1950] T 648f506ec87c4fb8a3f6d4591873dbb8 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:01.216980 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 49d9e5c5960a4534a6603875006e6266 (table default.Table_52c53bafe66b4c66b99e5b26e4df7ef5 [id=27dc835f316f41e091be9b2b1f560bbf]) successfully deleted
I20251027 21:08:01.218125 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 648f506ec87c4fb8a3f6d4591873dbb8 (table default.Table_a46825cc888a4a0898d6db70b02e5ab6 [id=8579d3ff46d64c75b1da213fdee9cc1b]) successfully deleted
I20251027 21:08:01.220012 10040 ts_tablet_manager.cc:1929] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.220121 10040 log.cc:1199] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/7d8825b80cec4ba68da077ca3b8d89db
I20251027 21:08:01.220394 10040 ts_tablet_manager.cc:1950] T 7d8825b80cec4ba68da077ca3b8d89db P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:01.220690 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 704) took 1396 ms. Trace:
I20251027 21:08:01.220768 10040 rpcz_store.cc:276] 1027 21:07:59.824700 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.831295 (+ 6595us) service_pool.cc:225] Handling call
1027 21:08:01.208348 (+1377053us) ts_tablet_manager.cc:1151] Deleting tablet 7d8825b80cec4ba68da077ca3b8d89db
1027 21:08:01.208356 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:01.213245 (+ 4889us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:01.219930 (+ 6685us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:01.220675 (+ 745us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1376880}
I20251027 21:08:01.221239 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 7d8825b80cec4ba68da077ca3b8d89db (table default.Table_943417713b3b456ab4ca691d2fb168fd [id=1cb32eea25114c6dbcb3b28044036c08]) successfully deleted
W20251027 21:08:01.221329 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 7c6c2292ad0e4066958897fff01370af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c6c2292ad0e4066958897fff01370af already in progress: creating tablet
I20251027 21:08:01.224149 10718 raft_consensus.cc:493] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:01.224285 10718 raft_consensus.cc:515] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
W20251027 21:08:01.224763 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet c25be263facd42c5bb6fa421160fbef0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c25be263facd42c5bb6fa421160fbef0 already in progress: creating tablet
I20251027 21:08:01.224792 10718 leader_election.cc:290] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
W20251027 21:08:01.225410 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 7a28dc9fe75744e5aa3d486afbe29561 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a28dc9fe75744e5aa3d486afbe29561 already in progress: creating tablet
W20251027 21:08:01.225551 9711 leader_election.cc:343] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 6db0f31c12d14e66a4ec7fc722d62c3d
I20251027 21:08:01.225760 10041 tablet_replica.cc:333] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:01.226344 10040 tablet_replica.cc:333] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:01.226591 10041 raft_consensus.cc:2243] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251027 21:08:01.226651 9710 leader_election.cc:343] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 6db0f31c12d14e66a4ec7fc722d62c3d
I20251027 21:08:01.226758 9710 leader_election.cc:304] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:01.226819 10041 raft_consensus.cc:2272] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.227092 10718 raft_consensus.cc:2749] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:01.227466 10040 raft_consensus.cc:2243] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.227622 10040 raft_consensus.cc:2272] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.227895 10041 ts_tablet_manager.cc:1916] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.228506 10040 ts_tablet_manager.cc:1916] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.230362 10773 tablet.cc:1722] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:08:01.230746 10032 tablet_bootstrap.cc:492] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:01.233064 10032 ts_tablet_manager.cc:1403] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.094s user 0.011s sys 0.021s
I20251027 21:08:01.234299 10032 raft_consensus.cc:359] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.234462 10032 raft_consensus.cc:385] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.234530 10032 raft_consensus.cc:740] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.234719 10032 consensus_queue.cc:260] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:01.234797 9299 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 2f641bc4dbed4c0abd0a55b253352507 (table default.Table_cfef185a988e4d61ae6575c228abc88d [id=8a26e744abae4c5eb1d18dd994cd6bd9], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2f641bc4dbed4c0abd0a55b253352507 consensus state has no leader
I20251027 21:08:01.235193 10032 ts_tablet_manager.cc:1434] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:01.235479 10032 tablet_bootstrap.cc:492] T 53cac7817db440e79e7416d625a9b2c8 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:01.236709 10032 tablet_bootstrap.cc:654] T 53cac7817db440e79e7416d625a9b2c8 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.236694 10043 ts_tablet_manager.cc:1929] T f22f013b25074d4db9c7f13caf5b9b42 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.236786 10043 log.cc:1199] T f22f013b25074d4db9c7f13caf5b9b42 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/f22f013b25074d4db9c7f13caf5b9b42
I20251027 21:08:01.237053 10043 ts_tablet_manager.cc:1950] T f22f013b25074d4db9c7f13caf5b9b42 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:01.239648 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet f22f013b25074d4db9c7f13caf5b9b42 (table default.Table_d601b22c11c34ebd8559009cd144affe [id=b96a43739ca54dac9e598e703186606b]) successfully deleted
I20251027 21:08:01.243466 10040 ts_tablet_manager.cc:1929] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.243569 10040 log.cc:1199] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/b87e809ff897417c88e5edc19e2d591e
I20251027 21:08:01.243817 10040 ts_tablet_manager.cc:1950] T b87e809ff897417c88e5edc19e2d591e P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:01.244076 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 711) took 1359 ms. Trace:
I20251027 21:08:01.244138 10040 rpcz_store.cc:276] 1027 21:07:59.884392 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:07:59.884460 (+ 68us) service_pool.cc:225] Handling call
1027 21:08:01.226062 (+1341602us) ts_tablet_manager.cc:1151] Deleting tablet b87e809ff897417c88e5edc19e2d591e
1027 21:08:01.226072 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:01.239888 (+ 13816us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:01.243393 (+ 3505us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:01.244060 (+ 667us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1341430}
I20251027 21:08:01.244705 10029 tablet_bootstrap.cc:492] T fe24af9838954a9bb680474800642e62 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:01.244830 10029 ts_tablet_manager.cc:1403] T fe24af9838954a9bb680474800642e62 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20251027 21:08:01.245087 10031 tablet_bootstrap.cc:492] T 7aa8c538b6c849a28a28ffe706306891 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:01.245203 10031 ts_tablet_manager.cc:1403] T 7aa8c538b6c849a28a28ffe706306891 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.005s
I20251027 21:08:01.245464 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e5d8837ff3c44a66a1f933d80540ba0c
I20251027 21:08:01.246058 9305 catalog_manager.cc:6719] ReplaceTablet: tablet e5d8837ff3c44a66a1f933d80540ba0c deleted and replaced by tablet 34f7313f60c440a1aba37b61dbbabc02
I20251027 21:08:01.245975 10031 raft_consensus.cc:359] T 7aa8c538b6c849a28a28ffe706306891 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.246205 10031 raft_consensus.cc:385] T 7aa8c538b6c849a28a28ffe706306891 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.246259 10031 raft_consensus.cc:740] T 7aa8c538b6c849a28a28ffe706306891 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.246471 10031 consensus_queue.cc:260] T 7aa8c538b6c849a28a28ffe706306891 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.247059 10031 ts_tablet_manager.cc:1434] T 7aa8c538b6c849a28a28ffe706306891 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:01.247292 10031 tablet_bootstrap.cc:492] T f1aa5f7f76cb4a1ba15f016942f17067 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:08:01.247610 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 25fbfcc28776483da266bb23c221f882 consensus state has no leader
I20251027 21:08:01.248340 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e1e58f6228254e3eb25dbf627a505ca1
I20251027 21:08:01.248551 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet e5d8837ff3c44a66a1f933d80540ba0c with delete_type TABLET_DATA_DELETED (replaced by tablet 34f7313f60c440a1aba37b61dbbabc02) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.249171 10043 tablet_replica.cc:333] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:01.249318 10043 raft_consensus.cc:2243] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:01.249498 10043 pending_rounds.cc:70] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14: Trying to abort 1 pending ops.
I20251027 21:08:01.249536 10043 pending_rounds.cc:77] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510655225323520 op_type: NO_OP noop_request { }
I20251027 21:08:01.249607 10043 raft_consensus.cc:2889] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:01.249670 10043 raft_consensus.cc:2272] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.249769 10031 tablet_bootstrap.cc:654] T f1aa5f7f76cb4a1ba15f016942f17067 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.251279 10032 tablet_bootstrap.cc:492] T 53cac7817db440e79e7416d625a9b2c8 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:01.251406 10032 ts_tablet_manager.cc:1403] T 53cac7817db440e79e7416d625a9b2c8 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20251027 21:08:01.252094 10041 ts_tablet_manager.cc:1929] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.252656 10029 raft_consensus.cc:359] T fe24af9838954a9bb680474800642e62 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.252821 10029 raft_consensus.cc:385] T fe24af9838954a9bb680474800642e62 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.252869 10029 raft_consensus.cc:740] T fe24af9838954a9bb680474800642e62 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.253098 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet b87e809ff897417c88e5edc19e2d591e (table default.Table_d2bf88aca4cc44df99d13783ea2722f8 [id=7de1278fe67640ff8540392837b2eeae]) successfully deleted
I20251027 21:08:01.253365 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet e5d8837ff3c44a66a1f933d80540ba0c with delete_type TABLET_DATA_DELETED (replaced by tablet 34f7313f60c440a1aba37b61dbbabc02) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.253582 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet e1e58f6228254e3eb25dbf627a505ca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.252818 10032 raft_consensus.cc:359] T 53cac7817db440e79e7416d625a9b2c8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.253685 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet e1e58f6228254e3eb25dbf627a505ca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.253762 10032 raft_consensus.cc:385] T 53cac7817db440e79e7416d625a9b2c8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.253824 10032 raft_consensus.cc:740] T 53cac7817db440e79e7416d625a9b2c8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.253988 10032 consensus_queue.cc:260] T 53cac7817db440e79e7416d625a9b2c8 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.254302 10043 ts_tablet_manager.cc:1916] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.254627 10032 ts_tablet_manager.cc:1434] T 53cac7817db440e79e7416d625a9b2c8 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:08:01.254910 10032 tablet_bootstrap.cc:492] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:01.253329 10029 consensus_queue.cc:260] T fe24af9838954a9bb680474800642e62 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.256480 10032 tablet_bootstrap.cc:654] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.257978 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 725) took 1151 ms. Trace:
I20251027 21:08:01.258065 10029 ts_tablet_manager.cc:1434] T fe24af9838954a9bb680474800642e62 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251027 21:08:01.258065 10040 rpcz_store.cc:276] 1027 21:08:00.106400 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:00.106811 (+ 411us) service_pool.cc:225] Handling call
1027 21:08:01.257867 (+1151056us) ts_tablet_manager.cc:1151] Deleting tablet 123e6e3748574ea581f67fff49037762
1027 21:08:01.257877 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:01.257962 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1150895}
I20251027 21:08:01.258988 10040 tablet_replica.cc:333] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:01.259172 10040 raft_consensus.cc:2243] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.259301 10040 raft_consensus.cc:2272] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:01.259527 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet be33f1825a4d433ea9ed859f37fe208e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be33f1825a4d433ea9ed859f37fe208e already in progress: creating tablet
I20251027 21:08:01.259724 10040 ts_tablet_manager.cc:1916] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.260450 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet e5d8837ff3c44a66a1f933d80540ba0c with delete_type TABLET_DATA_DELETED (replaced by tablet 34f7313f60c440a1aba37b61dbbabc02) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.260653 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet e1e58f6228254e3eb25dbf627a505ca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:01.260784 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 123e6e3748574ea581f67fff49037762 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 123e6e3748574ea581f67fff49037762 already in progress: creating tablet
I20251027 21:08:01.261422 10032 tablet_bootstrap.cc:492] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:01.261541 10032 ts_tablet_manager.cc:1403] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20251027 21:08:01.262243 10029 tablet_bootstrap.cc:492] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:01.264287 10029 tablet_bootstrap.cc:654] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.264183 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c5c0ff9082504f9f99ba0d43d21acbc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.264732 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5c0ff9082504f9f99ba0d43d21acbc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.265519 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_fbc3168ad4454ab3a6a83c1226e329f5" } modify_external_catalogs: true
I20251027 21:08:01.265736 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_fbc3168ad4454ab3a6a83c1226e329f5" } modify_external_catalogs: true
I20251027 21:08:01.265995 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9cd91730e4c4415e9f65d748f4592366"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:01.266593 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_004dd356c86a4140b824cdaefda61a23" } modify_external_catalogs: true
I20251027 21:08:01.266799 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_004dd356c86a4140b824cdaefda61a23" } modify_external_catalogs: true
W20251027 21:08:01.266922 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cd91730e4c4415e9f65d748f4592366 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.268105 9308 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet a9339c6b1154400eace53d765528fdf9 (table default.Table_5a6211c8465040dbad1cc86377a370ea [id=91fb9f95fe9343fca1d8cb0d0f781afb]). Expected version 2 got 0
I20251027 21:08:01.263343 10031 tablet_bootstrap.cc:492] T f1aa5f7f76cb4a1ba15f016942f17067 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:01.268498 10031 ts_tablet_manager.cc:1403] T f1aa5f7f76cb4a1ba15f016942f17067 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.002s
I20251027 21:08:01.268970 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c59fd341300a4f2f90f1bbd06750d943" } new_table_name: "default.Table_1284db19502347cdbb233e995dcca267" modify_external_catalogs: true
I20251027 21:08:01.269349 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_82f669b5326f44efa26969b7d9c7c3f9" } modify_external_catalogs: true
I20251027 21:08:01.269512 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_82f669b5326f44efa26969b7d9c7c3f9" } modify_external_catalogs: true
I20251027 21:08:01.269641 10031 raft_consensus.cc:359] T f1aa5f7f76cb4a1ba15f016942f17067 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.269784 10031 raft_consensus.cc:385] T f1aa5f7f76cb4a1ba15f016942f17067 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.269835 10031 raft_consensus.cc:740] T f1aa5f7f76cb4a1ba15f016942f17067 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.270051 10031 consensus_queue.cc:260] T f1aa5f7f76cb4a1ba15f016942f17067 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.271353 10031 ts_tablet_manager.cc:1434] T f1aa5f7f76cb4a1ba15f016942f17067 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251027 21:08:01.272188 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 21a8fc4a69f3486e92b1a68d10faed38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.272383 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.272886 10031 tablet_bootstrap.cc:492] T a287439ea6c9469da5fb56d84ee68048 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:01.273339 10032 raft_consensus.cc:359] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.273478 10032 raft_consensus.cc:385] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.273527 10032 raft_consensus.cc:740] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.273694 10032 consensus_queue.cc:260] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.274121 10032 ts_tablet_manager.cc:1434] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251027 21:08:01.274467 10032 tablet_bootstrap.cc:492] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:01.276682 10032 tablet_bootstrap.cc:654] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.277419 9303 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 7aa8c538b6c849a28a28ffe706306891 (table default.Table_b779f2756a924ab6ba1bc1702a412e06 [id=8ed572891f9b41cba8ed984911fa6b11]). Expected version 1 got 0
I20251027 21:08:01.279891 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c27c699cc2744f40811f7b9395ca76be
I20251027 21:08:01.279896 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 53cac7817db440e79e7416d625a9b2c8 with delete_type TABLET_DATA_DELETED (replaced by tablet ed66f9eeeb2b4b4bb77062c241d2140f) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.280555 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet c27c699cc2744f40811f7b9395ca76be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.281071 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet fe24af9838954a9bb680474800642e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.281256 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet c27c699cc2744f40811f7b9395ca76be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.281889 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet c27c699cc2744f40811f7b9395ca76be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.285593 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ec2e0ba1994ab59c53977583bd2058 with delete_type TABLET_DATA_DELETED (replaced by tablet 815e12edcd3847ff8485ba768291a824) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.288329 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet f1aa5f7f76cb4a1ba15f016942f17067 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.288506 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 132a01405a63447ab27b2137b803224f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.290522 10041 log.cc:1199] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/22d861810ed846c3b77a452d05e93d72
I20251027 21:08:01.290859 10041 ts_tablet_manager.cc:1950] T 22d861810ed846c3b77a452d05e93d72 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:01.291682 10031 tablet_bootstrap.cc:654] T a287439ea6c9469da5fb56d84ee68048 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.291826 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 22d861810ed846c3b77a452d05e93d72 (table default.Table_e4b781970438453eb7f7f4b885331bfb [id=c92fda1282d7419f94b0004ab8b8403d]) successfully deleted
I20251027 21:08:01.292021 10041 tablet_replica.cc:333] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:01.292196 10041 raft_consensus.cc:2243] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.292326 10041 raft_consensus.cc:2272] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:01.292325 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet d1d16726537a473ebd731e7a1f448400 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1d16726537a473ebd731e7a1f448400 already in progress: creating tablet
W20251027 21:08:01.292568 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet fb385f17054e430bbd69a2c9d22944ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb385f17054e430bbd69a2c9d22944ca already in progress: creating tablet
I20251027 21:08:01.292832 10041 ts_tablet_manager.cc:1916] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.298674 10040 ts_tablet_manager.cc:1929] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.298774 10040 log.cc:1199] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/583fba7c750e4ac7b768da447002c211
I20251027 21:08:01.299057 10040 ts_tablet_manager.cc:1950] T 583fba7c750e4ac7b768da447002c211 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:01.299309 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 726) took 1188 ms. Trace:
I20251027 21:08:01.299386 10040 rpcz_store.cc:276] 1027 21:08:00.111206 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:00.126824 (+ 15618us) service_pool.cc:225] Handling call
1027 21:08:01.258935 (+1132111us) ts_tablet_manager.cc:1151] Deleting tablet 583fba7c750e4ac7b768da447002c211
1027 21:08:01.258943 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:01.295482 (+ 36539us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:01.298601 (+ 3119us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:01.299294 (+ 693us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1131962}
I20251027 21:08:01.299737 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 583fba7c750e4ac7b768da447002c211 (table default.Table_660eb77e69884fe5bd5e65b47ea91dde [id=14daa3e199fe4fa3ba2de06dd6be4330]) successfully deleted
I20251027 21:08:01.300185 10747 raft_consensus.cc:493] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:01.300292 10747 raft_consensus.cc:515] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.300802 10747 leader_election.cc:290] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:01.301537 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccf10ef559f74e2db28023fd520c8de9" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:01.301864 10043 ts_tablet_manager.cc:1929] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:01.301990 10043 log.cc:1199] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/c400101d2a39490e83722500839198cd
W20251027 21:08:01.302074 9860 leader_election.cc:343] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:01.301002 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 728) took 1163 ms. Trace:
I20251027 21:08:01.301353 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccf10ef559f74e2db28023fd520c8de9" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:01.302382 10040 rpcz_store.cc:276] 1027 21:08:00.137107 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:00.138220 (+ 1113us) service_pool.cc:225] Handling call
1027 21:08:01.300893 (+1162673us) ts_tablet_manager.cc:1151] Deleting tablet deb8b54e5e0f42bca4330d2160e5524a
1027 21:08:01.300902 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:01.300985 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1162509}
W20251027 21:08:01.302677 9859 leader_election.cc:343] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:01.302786 9859 leader_election.cc:304] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:01.303093 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 727) took 1169 ms. Trace:
W20251027 21:08:01.303210 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet deb8b54e5e0f42bca4330d2160e5524a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet deb8b54e5e0f42bca4330d2160e5524a already in progress: creating tablet
I20251027 21:08:01.303170 10040 rpcz_store.cc:276] 1027 21:08:00.133704 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:00.155906 (+ 22202us) service_pool.cc:225] Handling call
1027 21:08:01.302988 (+1147082us) ts_tablet_manager.cc:1151] Deleting tablet e1385beb1dc14ee0b42a2f4e1f66f532
1027 21:08:01.302998 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:01.303078 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1146922}
W20251027 21:08:01.303427 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet e1385beb1dc14ee0b42a2f4e1f66f532 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1385beb1dc14ee0b42a2f4e1f66f532 already in progress: creating tablet
I20251027 21:08:01.303910 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 731) took 1087 ms. Trace:
I20251027 21:08:01.303982 10040 rpcz_store.cc:276] 1027 21:08:00.216387 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:00.216430 (+ 43us) service_pool.cc:225] Handling call
1027 21:08:01.303821 (+1087391us) ts_tablet_manager.cc:1151] Deleting tablet cc4635c00e9b40c49ce04debc76a04bf
1027 21:08:01.303829 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:01.303895 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1087220}
I20251027 21:08:01.304085 10747 raft_consensus.cc:2749] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:01.304742 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet cc4635c00e9b40c49ce04debc76a04bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc4635c00e9b40c49ce04debc76a04bf already in progress: creating tablet
I20251027 21:08:01.304800 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 729) took 1099 ms. Trace:
I20251027 21:08:01.304867 10040 rpcz_store.cc:276] 1027 21:08:00.205056 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:00.205117 (+ 61us) service_pool.cc:225] Handling call
1027 21:08:01.304701 (+1099584us) ts_tablet_manager.cc:1151] Deleting tablet 211ac01a37f8434f95aa6a63970296fa
1027 21:08:01.304711 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:01.304785 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1072313}
I20251027 21:08:01.306293 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 730) took 1091 ms. Trace:
I20251027 21:08:01.306371 10040 rpcz_store.cc:276] 1027 21:08:00.214884 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:00.214966 (+ 82us) service_pool.cc:225] Handling call
1027 21:08:01.306195 (+1091229us) ts_tablet_manager.cc:1151] Deleting tablet c1f99ca89f924b58b1416d2945a6ba40
1027 21:08:01.306203 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:01.306277 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1072921}
I20251027 21:08:01.306608 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 9e9191649aa84b058abdc1477995ef67
W20251027 21:08:01.307000 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fdd0198d5047458aab8bf3ce08ccc0de consensus state has no leader
I20251027 21:08:01.307569 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e9191649aa84b058abdc1477995ef67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.307983 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e9191649aa84b058abdc1477995ef67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.308048 10040 tablet_replica.cc:333] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:01.308264 10040 raft_consensus.cc:2243] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.308418 10040 raft_consensus.cc:2272] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:01.308722 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 211ac01a37f8434f95aa6a63970296fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 211ac01a37f8434f95aa6a63970296fa already in progress: creating tablet
I20251027 21:08:01.308885 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e9191649aa84b058abdc1477995ef67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:08:01.308993 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c1f99ca89f924b58b1416d2945a6ba40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1f99ca89f924b58b1416d2945a6ba40 already in progress: creating tablet
I20251027 21:08:01.310479 10032 tablet_bootstrap.cc:492] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:01.310632 10040 ts_tablet_manager.cc:1916] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.310675 10032 ts_tablet_manager.cc:1403] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.002s
I20251027 21:08:01.311496 10032 raft_consensus.cc:359] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.311688 10032 raft_consensus.cc:385] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.311769 10032 raft_consensus.cc:740] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.311995 10032 consensus_queue.cc:260] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.312940 9769 raft_consensus.cc:3060] T 7aa8c538b6c849a28a28ffe706306891 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:01.314884 10032 ts_tablet_manager.cc:1434] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251027 21:08:01.315145 10032 tablet_bootstrap.cc:492] T 84b9de82d55447bba2a897edb683e5c5 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:01.318542 10032 tablet_bootstrap.cc:654] T 84b9de82d55447bba2a897edb683e5c5 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.322413 10043 ts_tablet_manager.cc:1950] T c400101d2a39490e83722500839198cd P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:01.323211 10031 tablet_bootstrap.cc:492] T a287439ea6c9469da5fb56d84ee68048 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:01.323400 10031 ts_tablet_manager.cc:1403] T a287439ea6c9469da5fb56d84ee68048 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.006s
I20251027 21:08:01.323731 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet c400101d2a39490e83722500839198cd (table default.Table_97b01c1840c543838263ca4222695f54 [id=bea51a58fda64d40916f306c4aa8ab4e]) successfully deleted
I20251027 21:08:01.324227 10031 raft_consensus.cc:359] T a287439ea6c9469da5fb56d84ee68048 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.324643 10740 tablet.cc:1722] T 7aa8c538b6c849a28a28ffe706306891 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:08:01.325140 10031 raft_consensus.cc:385] T a287439ea6c9469da5fb56d84ee68048 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251027 21:08:01.325114 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 842c1a1298234d9ea2719bab29c54ae3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 842c1a1298234d9ea2719bab29c54ae3 already in progress: creating tablet
I20251027 21:08:01.325235 10031 raft_consensus.cc:740] T a287439ea6c9469da5fb56d84ee68048 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.325525 10031 consensus_queue.cc:260] T a287439ea6c9469da5fb56d84ee68048 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:08:01.327706 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet ae0ee7cd2c1b406f84285f25420588d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae0ee7cd2c1b406f84285f25420588d0 already in progress: creating tablet
W20251027 21:08:01.329957 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet bdd382c4d376484cb7767d30681f966f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdd382c4d376484cb7767d30681f966f already in progress: creating tablet
W20251027 21:08:01.330605 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet e04f4c5cf724419ab0be0e84d30a9a57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e04f4c5cf724419ab0be0e84d30a9a57 already in progress: creating tablet
I20251027 21:08:01.330679 9310 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 7aa8c538b6c849a28a28ffe706306891 (table default.Table_b779f2756a924ab6ba1bc1702a412e06 [id=8ed572891f9b41cba8ed984911fa6b11]). Expected version 1 got 0
W20251027 21:08:01.331271 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 55a545185c0140e1aa1ddf3fb6ceba9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55a545185c0140e1aa1ddf3fb6ceba9a already in progress: creating tablet
I20251027 21:08:01.332335 10031 ts_tablet_manager.cc:1434] T a287439ea6c9469da5fb56d84ee68048 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251027 21:08:01.333401 10031 tablet_bootstrap.cc:492] T 9b04349b15f3494bb0933db56b7b24bc P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:01.334666 10043 tablet_replica.cc:333] T d8e008c868d04281beba0e5b204eb69a P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:01.334887 10043 raft_consensus.cc:2243] T d8e008c868d04281beba0e5b204eb69a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.335582 10031 tablet_bootstrap.cc:654] T 9b04349b15f3494bb0933db56b7b24bc P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.335800 10043 raft_consensus.cc:2272] T d8e008c868d04281beba0e5b204eb69a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.336439 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet a287439ea6c9469da5fb56d84ee68048 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.337087 10040 ts_tablet_manager.cc:1929] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.337169 10040 log.cc:1199] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/6db0f31c12d14e66a4ec7fc722d62c3d
I20251027 21:08:01.337425 10040 ts_tablet_manager.cc:1950] T 6db0f31c12d14e66a4ec7fc722d62c3d P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:01.337661 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 739) took 1055 ms. Trace:
I20251027 21:08:01.337715 10040 rpcz_store.cc:276] 1027 21:08:00.282496 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:00.301342 (+ 18846us) service_pool.cc:225] Handling call
1027 21:08:01.307990 (+1006648us) ts_tablet_manager.cc:1151] Deleting tablet 6db0f31c12d14e66a4ec7fc722d62c3d
1027 21:08:01.308001 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:01.316107 (+ 8106us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:01.337010 (+ 20903us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:01.337649 (+ 639us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1006446}
I20251027 21:08:01.338531 10043 ts_tablet_manager.cc:1916] T d8e008c868d04281beba0e5b204eb69a P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.338582 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 740) took 1036 ms. Trace:
I20251027 21:08:01.338668 10040 rpcz_store.cc:276] 1027 21:08:00.301760 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:00.302379 (+ 619us) service_pool.cc:225] Handling call
1027 21:08:01.338482 (+1036103us) ts_tablet_manager.cc:1151] Deleting tablet 4ba13a89a33549af921d3b55023faf15
1027 21:08:01.338490 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:01.338567 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1035890}
I20251027 21:08:01.338953 10041 ts_tablet_manager.cc:1929] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.339094 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 741) took 1036 ms. Trace:
I20251027 21:08:01.339267 10040 rpcz_store.cc:276] 1027 21:08:00.302742 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:00.303267 (+ 525us) service_pool.cc:225] Handling call
1027 21:08:01.339006 (+1035739us) ts_tablet_manager.cc:1151] Deleting tablet a8c46c2a01f946aea8b85875115f7f37
1027 21:08:01.339015 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:01.339080 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1035568}
I20251027 21:08:01.339501 10041 log.cc:1199] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/630ae4d66f99474382814cc32cc60ea0
I20251027 21:08:01.339737 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 738) took 1057 ms. Trace:
I20251027 21:08:01.339839 10040 rpcz_store.cc:276] 1027 21:08:00.282396 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:00.346706 (+ 64310us) service_pool.cc:225] Handling call
1027 21:08:01.339623 (+992917us) ts_tablet_manager.cc:1151] Deleting tablet 933b33aac5354778a5866d340fd34db4
1027 21:08:01.339635 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:01.339719 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":939644}
I20251027 21:08:01.341069 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 6db0f31c12d14e66a4ec7fc722d62c3d (table default.Table_42431224890f43859abe3dedfae4db51 [id=a6784a116865476b85fe6ff9c32bb9db]) successfully deleted
W20251027 21:08:01.341320 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 4ba13a89a33549af921d3b55023faf15 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ba13a89a33549af921d3b55023faf15 already in progress: creating tablet
W20251027 21:08:01.341514 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet a8c46c2a01f946aea8b85875115f7f37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8c46c2a01f946aea8b85875115f7f37 already in progress: creating tablet
I20251027 21:08:01.342417 10041 ts_tablet_manager.cc:1950] T 630ae4d66f99474382814cc32cc60ea0 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
W20251027 21:08:01.341718 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 933b33aac5354778a5866d340fd34db4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 933b33aac5354778a5866d340fd34db4 already in progress: creating tablet
W20251027 21:08:01.343003 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet e9b8f76c33444bcb9b488af65e4d3e6b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9b8f76c33444bcb9b488af65e4d3e6b already in progress: creating tablet
W20251027 21:08:01.343194 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c25be263facd42c5bb6fa421160fbef0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c25be263facd42c5bb6fa421160fbef0 already in progress: creating tablet
I20251027 21:08:01.347847 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 630ae4d66f99474382814cc32cc60ea0 (table default.Table_cfef185a988e4d61ae6575c228abc88d [id=8a26e744abae4c5eb1d18dd994cd6bd9]) successfully deleted
W20251027 21:08:01.348605 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 622fa1b3ff134f2d8a087ba13a85e597 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 622fa1b3ff134f2d8a087ba13a85e597 already in progress: creating tablet
I20251027 21:08:01.349221 10747 raft_consensus.cc:493] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:01.349370 10747 raft_consensus.cc:515] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.350054 10747 leader_election.cc:290] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:01.350386 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24ec2e0ba1994ab59c53977583bd2058" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:01.350545 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet be33f1825a4d433ea9ed859f37fe208e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be33f1825a4d433ea9ed859f37fe208e already in progress: creating tablet
W20251027 21:08:01.350963 9860 leader_election.cc:343] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:01.351200 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 842c1a1298234d9ea2719bab29c54ae3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 842c1a1298234d9ea2719bab29c54ae3 already in progress: creating tablet
I20251027 21:08:01.351290 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24ec2e0ba1994ab59c53977583bd2058" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:01.351671 9859 leader_election.cc:343] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:01.351776 9859 leader_election.cc:304] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:01.352219 10747 raft_consensus.cc:2749] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:01.352737 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet bdd382c4d376484cb7767d30681f966f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdd382c4d376484cb7767d30681f966f already in progress: creating tablet
I20251027 21:08:01.353821 10032 tablet_bootstrap.cc:492] T 84b9de82d55447bba2a897edb683e5c5 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:01.353948 10032 ts_tablet_manager.cc:1403] T 84b9de82d55447bba2a897edb683e5c5 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.005s
W20251027 21:08:01.354382 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet e04f4c5cf724419ab0be0e84d30a9a57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e04f4c5cf724419ab0be0e84d30a9a57 already in progress: creating tablet
W20251027 21:08:01.355065 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet ae0ee7cd2c1b406f84285f25420588d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae0ee7cd2c1b406f84285f25420588d0 already in progress: creating tablet
W20251027 21:08:01.355726 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 55a545185c0140e1aa1ddf3fb6ceba9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55a545185c0140e1aa1ddf3fb6ceba9a already in progress: creating tablet
I20251027 21:08:01.355710 10032 raft_consensus.cc:359] T 84b9de82d55447bba2a897edb683e5c5 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.355877 10032 raft_consensus.cc:385] T 84b9de82d55447bba2a897edb683e5c5 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.355943 10032 raft_consensus.cc:740] T 84b9de82d55447bba2a897edb683e5c5 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.356139 10032 consensus_queue.cc:260] T 84b9de82d55447bba2a897edb683e5c5 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:01.358059 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 7c6c2292ad0e4066958897fff01370af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c6c2292ad0e4066958897fff01370af already in progress: creating tablet
I20251027 21:08:01.358181 10040 tablet_replica.cc:333] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:01.358352 10040 raft_consensus.cc:2243] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.358506 10040 raft_consensus.cc:2272] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.358614 10032 ts_tablet_manager.cc:1434] T 84b9de82d55447bba2a897edb683e5c5 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251027 21:08:01.359006 10040 ts_tablet_manager.cc:1916] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.358815 10041 tablet_replica.cc:333] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:01.359604 10032 tablet_bootstrap.cc:492] T 26872be132fb46ee9e79a9c2e62374ef P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:01.359608 10041 raft_consensus.cc:2243] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.359920 10041 raft_consensus.cc:2272] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.360818 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_24aac5cecb8745d7be92287080a1b6e0" } new_table_name: "default.Table_b7607fa1c70d431d97f9e3f8a7c3d667" modify_external_catalogs: true
I20251027 21:08:01.361305 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8b1c94c0aec244e3950e08017ed7da7d" } new_table_name: "default.Table_6b1fd8ed70ee48709a53ab24b5baaab4" modify_external_catalogs: true
I20251027 21:08:01.361539 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5f6e0aaf15fe439c87306828e9a8cb1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.362027 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f6e0aaf15fe439c87306828e9a8cb1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.363782 10032 tablet_bootstrap.cc:654] T 26872be132fb46ee9e79a9c2e62374ef P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.365664 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2096e2f92ca2449485e81a1a68807ef2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:01.365962 10031 tablet_bootstrap.cc:492] T 9b04349b15f3494bb0933db56b7b24bc P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:01.366082 10031 ts_tablet_manager.cc:1403] T 9b04349b15f3494bb0933db56b7b24bc P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.003s
W20251027 21:08:01.366194 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2096e2f92ca2449485e81a1a68807ef2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.366858 10031 raft_consensus.cc:359] T 9b04349b15f3494bb0933db56b7b24bc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.367020 10031 raft_consensus.cc:385] T 9b04349b15f3494bb0933db56b7b24bc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.367076 10031 raft_consensus.cc:740] T 9b04349b15f3494bb0933db56b7b24bc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.367380 10032 tablet_bootstrap.cc:492] T 26872be132fb46ee9e79a9c2e62374ef P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:01.367363 10031 consensus_queue.cc:260] T 9b04349b15f3494bb0933db56b7b24bc P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.367496 10032 ts_tablet_manager.cc:1403] T 26872be132fb46ee9e79a9c2e62374ef P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20251027 21:08:01.368074 10032 raft_consensus.cc:359] T 26872be132fb46ee9e79a9c2e62374ef P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.368211 10032 raft_consensus.cc:385] T 26872be132fb46ee9e79a9c2e62374ef P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.368254 10032 raft_consensus.cc:740] T 26872be132fb46ee9e79a9c2e62374ef P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.368837 10031 ts_tablet_manager.cc:1434] T 9b04349b15f3494bb0933db56b7b24bc P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251027 21:08:01.369691 10032 consensus_queue.cc:260] T 26872be132fb46ee9e79a9c2e62374ef P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.370177 10032 ts_tablet_manager.cc:1434] T 26872be132fb46ee9e79a9c2e62374ef P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:08:01.371816 10032 tablet_bootstrap.cc:492] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:01.371905 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b04349b15f3494bb0933db56b7b24bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.372707 10031 tablet_bootstrap.cc:492] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:01.374392 10032 tablet_bootstrap.cc:654] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:01.377538 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5cb8ca3a50c148e6b602e949650353f8 consensus state has no leader
I20251027 21:08:01.379225 10747 raft_consensus.cc:493] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:01.379339 10747 raft_consensus.cc:515] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.379882 10747 leader_election.cc:290] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:01.380710 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "345d3795a37b4a40b40967cfacf0f893" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:01.381217 9860 leader_election.cc:343] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:01.381533 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "345d3795a37b4a40b40967cfacf0f893" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:01.381916 9859 leader_election.cc:343] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:01.382021 9859 leader_election.cc:304] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:01.381932 10041 ts_tablet_manager.cc:1916] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.382361 10031 tablet_bootstrap.cc:654] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.382599 10747 raft_consensus.cc:2749] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:01.386554 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8db666d9b7eb458b843ab5910cd7dff4 consensus state has no leader
I20251027 21:08:01.387019 10681 raft_consensus.cc:493] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:01.387166 10681 raft_consensus.cc:515] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.387737 10681 leader_election.cc:290] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:01.388573 9923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9339c6b1154400eace53d765528fdf9" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:01.388711 9923 raft_consensus.cc:2468] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:01.389422 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9339c6b1154400eace53d765528fdf9" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:01.390163 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c8fded5f6e8b4798a55ec84954d75574" } new_table_name: "default.Table_0cc61333cd424ee2b2f850d933fc6340" modify_external_catalogs: true
I20251027 21:08:01.390430 9568 leader_election.cc:304] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
W20251027 21:08:01.390676 9566 leader_election.cc:343] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:01.390916 10681 raft_consensus.cc:2804] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:01.390972 10681 raft_consensus.cc:493] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:01.391033 10681 raft_consensus.cc:3060] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:01.393556 10681 raft_consensus.cc:515] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.393875 10040 ts_tablet_manager.cc:1929] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.394008 10040 log.cc:1199] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/9c239216cb8846f0b1aaa360ddb83321
I20251027 21:08:01.394486 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9339c6b1154400eace53d765528fdf9" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:01.394721 9923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9339c6b1154400eace53d765528fdf9" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:01.394867 9923 raft_consensus.cc:3060] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:01.397397 9923 raft_consensus.cc:2468] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:01.397850 9568 leader_election.cc:304] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:01.398113 10529 raft_consensus.cc:2804] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:01.398317 10529 raft_consensus.cc:697] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:01.398538 10529 consensus_queue.cc:237] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:08:01.400137 9566 leader_election.cc:343] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:01.403290 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_db03a0faacfc4ecfad89dbf48def9140"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.403828 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db03a0faacfc4ecfad89dbf48def9140 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.405510 10681 leader_election.cc:290] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:01.410027 9304 master_service.cc:954] ReplaceTablet: received request to replace tablet 7425b6b0184e4d0b871777b25d904ac4 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:01.413978 10043 ts_tablet_manager.cc:1929] T d8e008c868d04281beba0e5b204eb69a P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.422889 10031 tablet_bootstrap.cc:492] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:01.423031 10031 ts_tablet_manager.cc:1403] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.003s
I20251027 21:08:01.423870 10031 raft_consensus.cc:359] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.424052 10031 raft_consensus.cc:385] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.424111 10031 raft_consensus.cc:740] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.424325 10031 consensus_queue.cc:260] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.425220 10031 ts_tablet_manager.cc:1434] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:01.426662 10031 tablet_bootstrap.cc:492] T e841c54a94d649c19b639f5021f150cd P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:08:01.429400 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b261415f48a43229fbb342ac7f7a677 consensus state has no leader
I20251027 21:08:01.430424 10031 tablet_bootstrap.cc:654] T e841c54a94d649c19b639f5021f150cd P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.432976 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet ef6f69db3d2547c88bbc8df2d52e2fb1 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:01.447499 10031 tablet_bootstrap.cc:492] T e841c54a94d649c19b639f5021f150cd P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:01.447649 10031 ts_tablet_manager.cc:1403] T e841c54a94d649c19b639f5021f150cd P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20251027 21:08:01.449095 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e12fdbe9bbc0492f85e48ceed92def03" } new_table_name: "default.Table_ff5458dda9f042e189bd8cd69c06d025" modify_external_catalogs: true
I20251027 21:08:01.450006 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_263a9587a19c451b81a428c0891f84b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.450485 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_263a9587a19c451b81a428c0891f84b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.450452 10031 raft_consensus.cc:359] T e841c54a94d649c19b639f5021f150cd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.450735 10031 raft_consensus.cc:385] T e841c54a94d649c19b639f5021f150cd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.450845 10031 raft_consensus.cc:740] T e841c54a94d649c19b639f5021f150cd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.451160 10031 consensus_queue.cc:260] T e841c54a94d649c19b639f5021f150cd P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.451563 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 0d2d977e360b4ed29c9ed63a309783d6
I20251027 21:08:01.451975 10031 ts_tablet_manager.cc:1434] T e841c54a94d649c19b639f5021f150cd P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251027 21:08:01.452661 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d2d977e360b4ed29c9ed63a309783d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.453050 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d2d977e360b4ed29c9ed63a309783d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.453565 10031 tablet_bootstrap.cc:492] T 69da799c886841a19002d3c7c5084e73 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:01.454015 10032 tablet_bootstrap.cc:492] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:01.454838 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_61f42d2bf7b941cebdb4d20f78094243" } modify_external_catalogs: true
I20251027 21:08:01.455024 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_61f42d2bf7b941cebdb4d20f78094243" } modify_external_catalogs: true
I20251027 21:08:01.455209 10031 tablet_bootstrap.cc:654] T 69da799c886841a19002d3c7c5084e73 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.456070 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_be7a9a27122348488bb59fcd53109c5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.456614 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be7a9a27122348488bb59fcd53109c5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.457634 10040 ts_tablet_manager.cc:1950] T 9c239216cb8846f0b1aaa360ddb83321 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:01.457805 10032 ts_tablet_manager.cc:1403] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.086s user 0.013s sys 0.000s
I20251027 21:08:01.458285 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 9c239216cb8846f0b1aaa360ddb83321 (table default.Table_2d758ba690f8478ba7f983e8fd770219 [id=849a0760791548a38aceb18d0533f5f4]) successfully deleted
I20251027 21:08:01.458680 10032 raft_consensus.cc:359] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:01.458879 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 7a28dc9fe75744e5aa3d486afbe29561 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a28dc9fe75744e5aa3d486afbe29561 already in progress: creating tablet
I20251027 21:08:01.458940 10032 raft_consensus.cc:385] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.459120 10032 raft_consensus.cc:740] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.459362 10032 consensus_queue.cc:260] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:01.459774 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet fb385f17054e430bbd69a2c9d22944ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb385f17054e430bbd69a2c9d22944ca already in progress: creating tablet
W20251027 21:08:01.460057 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet d1d16726537a473ebd731e7a1f448400 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1d16726537a473ebd731e7a1f448400 already in progress: creating tablet
W20251027 21:08:01.460304 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 622fa1b3ff134f2d8a087ba13a85e597 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 622fa1b3ff134f2d8a087ba13a85e597 already in progress: creating tablet
I20251027 21:08:01.462787 10040 tablet_replica.cc:333] T 78884c2be75f45f881f5424d3b3f4e08 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:01.462989 10040 raft_consensus.cc:2243] T 78884c2be75f45f881f5424d3b3f4e08 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251027 21:08:01.463071 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet be33f1825a4d433ea9ed859f37fe208e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be33f1825a4d433ea9ed859f37fe208e already in progress: creating tablet
I20251027 21:08:01.463152 10040 raft_consensus.cc:2272] T 78884c2be75f45f881f5424d3b3f4e08 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.463689 10040 ts_tablet_manager.cc:1916] T 78884c2be75f45f881f5424d3b3f4e08 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:01.464260 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 72c1dc3b9eff47ee81df821deb789307 consensus state has no leader
I20251027 21:08:01.464643 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7cf03d793a0548ef80ca9ce245e0fc1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.465092 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7cf03d793a0548ef80ca9ce245e0fc1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.467978 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5a2a0af6dbb4438195f67883f19838ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.468487 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a2a0af6dbb4438195f67883f19838ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.472645 10031 tablet_bootstrap.cc:492] T 69da799c886841a19002d3c7c5084e73 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:01.472764 10031 ts_tablet_manager.cc:1403] T 69da799c886841a19002d3c7c5084e73 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20251027 21:08:01.473361 10031 raft_consensus.cc:359] T 69da799c886841a19002d3c7c5084e73 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.473521 10031 raft_consensus.cc:385] T 69da799c886841a19002d3c7c5084e73 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.473582 10031 raft_consensus.cc:740] T 69da799c886841a19002d3c7c5084e73 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.473778 10031 consensus_queue.cc:260] T 69da799c886841a19002d3c7c5084e73 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.474995 10031 ts_tablet_manager.cc:1434] T 69da799c886841a19002d3c7c5084e73 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:01.475416 10031 tablet_bootstrap.cc:492] T ed5619d295a246379b625f35692203e4 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:01.477314 9305 master_service.cc:954] ReplaceTablet: received request to replace tablet a68bd9243b7541bbbc77014113e0106a from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:01.477505 10031 tablet_bootstrap.cc:654] T ed5619d295a246379b625f35692203e4 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.482121 10032 ts_tablet_manager.cc:1434] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.024s user 0.004s sys 0.000s
I20251027 21:08:01.482501 10032 tablet_bootstrap.cc:492] T a8328318a4f14b38a4ce36d183e99dfb P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:01.484469 10032 tablet_bootstrap.cc:654] T a8328318a4f14b38a4ce36d183e99dfb P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.484843 10040 ts_tablet_manager.cc:1929] T 78884c2be75f45f881f5424d3b3f4e08 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.484948 10040 log.cc:1199] T 78884c2be75f45f881f5424d3b3f4e08 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/78884c2be75f45f881f5424d3b3f4e08
I20251027 21:08:01.485260 10040 ts_tablet_manager.cc:1950] T 78884c2be75f45f881f5424d3b3f4e08 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:01.486112 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 78884c2be75f45f881f5424d3b3f4e08 (table default.Table_1376974f614548189208cb9cf0fd5353 [id=0eb078640c174f8fadb959306a35cada]) successfully deleted
I20251027 21:08:01.486785 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_80cde2bc1ed84de29cec9255f33b03d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.487469 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80cde2bc1ed84de29cec9255f33b03d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.489006 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_728cbf4f936c4df79d77156e2d3ac59c" } modify_external_catalogs: true
I20251027 21:08:01.489210 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_728cbf4f936c4df79d77156e2d3ac59c" } modify_external_catalogs: true
I20251027 21:08:01.503224 10031 tablet_bootstrap.cc:492] T ed5619d295a246379b625f35692203e4 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:01.503378 10031 ts_tablet_manager.cc:1403] T ed5619d295a246379b625f35692203e4 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.002s
I20251027 21:08:01.503991 10031 raft_consensus.cc:359] T ed5619d295a246379b625f35692203e4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.504175 10031 raft_consensus.cc:385] T ed5619d295a246379b625f35692203e4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.504244 10031 raft_consensus.cc:740] T ed5619d295a246379b625f35692203e4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.504455 10031 consensus_queue.cc:260] T ed5619d295a246379b625f35692203e4 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.505946 10031 ts_tablet_manager.cc:1434] T ed5619d295a246379b625f35692203e4 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:01.507185 10031 tablet_bootstrap.cc:492] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:01.510004 10031 tablet_bootstrap.cc:654] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.512326 10040 tablet_replica.cc:333] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:01.512369 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e86d94359267418f85f5fda8a5ff79d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:01.512622 10040 raft_consensus.cc:2243] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:01.512821 10040 pending_rounds.cc:70] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
I20251027 21:08:01.512868 10040 pending_rounds.cc:77] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510655255146496 op_type: NO_OP noop_request { }
W20251027 21:08:01.512873 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e86d94359267418f85f5fda8a5ff79d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.512941 10040 raft_consensus.cc:2889] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:01.513008 10040 raft_consensus.cc:2272] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.513769 10040 ts_tablet_manager.cc:1916] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.517980 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2c4482a14aa14f498e7c8ab78d9fe9ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.518476 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c4482a14aa14f498e7c8ab78d9fe9ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.519471 10031 tablet_bootstrap.cc:492] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:01.519598 10031 ts_tablet_manager.cc:1403] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20251027 21:08:01.520285 10031 raft_consensus.cc:359] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.520418 10031 raft_consensus.cc:385] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.520478 10031 raft_consensus.cc:740] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.520704 10031 consensus_queue.cc:260] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.522001 10031 ts_tablet_manager.cc:1434] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:01.523123 10031 tablet_bootstrap.cc:492] T 8fc2d7d6601d4cc2849064f4cdddc90d P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:01.523236 9310 catalog_manager.cc:5649] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:01.523617 9310 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet a9339c6b1154400eace53d765528fdf9 (table default.Table_5a6211c8465040dbad1cc86377a370ea [id=91fb9f95fe9343fca1d8cb0d0f781afb]). Expected version 2 got 0
I20251027 21:08:01.524971 10031 tablet_bootstrap.cc:654] T 8fc2d7d6601d4cc2849064f4cdddc90d P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.526945 10040 ts_tablet_manager.cc:1929] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:01.527052 10040 log.cc:1199] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/f22f013b25074d4db9c7f13caf5b9b42
I20251027 21:08:01.415890 10029 tablet_bootstrap.cc:492] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:01.527227 10029 ts_tablet_manager.cc:1403] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.265s user 0.014s sys 0.022s
I20251027 21:08:01.527375 10040 ts_tablet_manager.cc:1950] T f22f013b25074d4db9c7f13caf5b9b42 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:01.527844 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet f22f013b25074d4db9c7f13caf5b9b42 (table default.Table_d601b22c11c34ebd8559009cd144affe [id=b96a43739ca54dac9e598e703186606b]) successfully deleted
I20251027 21:08:01.527894 10040 tablet_replica.cc:333] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:01.527909 10029 raft_consensus.cc:359] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.528043 10040 raft_consensus.cc:2243] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.528086 10029 raft_consensus.cc:385] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.528151 10029 raft_consensus.cc:740] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.528170 10040 raft_consensus.cc:2272] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.528338 10029 consensus_queue.cc:260] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.528604 10040 ts_tablet_manager.cc:1916] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.528762 10029 ts_tablet_manager.cc:1434] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251027 21:08:01.529016 10029 tablet_bootstrap.cc:492] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:01.453573 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d2d977e360b4ed29c9ed63a309783d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.534587 10029 tablet_bootstrap.cc:654] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.536396 10029 tablet_bootstrap.cc:492] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:01.536561 10029 ts_tablet_manager.cc:1403] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20251027 21:08:01.537277 10029 raft_consensus.cc:359] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.537468 10029 raft_consensus.cc:385] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.537572 10029 raft_consensus.cc:740] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.537848 10029 consensus_queue.cc:260] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.538329 10029 ts_tablet_manager.cc:1434] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 21:08:01.538617 10029 tablet_bootstrap.cc:492] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:01.539151 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 356a04cc6b7444df9ae17c12d5bff727
I20251027 21:08:01.539860 10040 ts_tablet_manager.cc:1929] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.539942 10040 log.cc:1199] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/ccaf5c2f8e544842880ba8ba27300ae6
I20251027 21:08:01.540004 10029 tablet_bootstrap.cc:654] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.540218 10040 ts_tablet_manager.cc:1950] T ccaf5c2f8e544842880ba8ba27300ae6 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:01.541224 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 356a04cc6b7444df9ae17c12d5bff727 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.541699 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 292192b6d0d646de9144bbd6adbd4fbe
I20251027 21:08:01.543495 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 292192b6d0d646de9144bbd6adbd4fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.540526 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 356a04cc6b7444df9ae17c12d5bff727 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.542894 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 292192b6d0d646de9144bbd6adbd4fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.544534 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet ccaf5c2f8e544842880ba8ba27300ae6 (table default.Table_9c6b1e43fc5f4f4389620f1ee244db12 [id=2b0d4e085a8e42879c67963231a24b03]) successfully deleted
I20251027 21:08:01.544881 10040 tablet_replica.cc:333] T c400101d2a39490e83722500839198cd P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:01.545053 10040 raft_consensus.cc:2243] T c400101d2a39490e83722500839198cd P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.545194 10040 raft_consensus.cc:2272] T c400101d2a39490e83722500839198cd P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.545095 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8f8eb3a5bb404ca59da6c90fea1abfb1" } new_table_name: "default.Table_50b757fa69504e57872e1ecaee13583c" modify_external_catalogs: true
I20251027 21:08:01.545622 10040 ts_tablet_manager.cc:1916] T c400101d2a39490e83722500839198cd P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.549548 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 84b9de82d55447bba2a897edb683e5c5 with delete_type TABLET_DATA_DELETED (replaced by tablet befdd3a485a1426b8a3adf340d4eae4f) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.550335 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 2532bf7190064443a898e47e843fd9ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.551262 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 26872be132fb46ee9e79a9c2e62374ef with delete_type TABLET_DATA_DELETED (replaced by tablet 6409f45a907b4389bac84093cdc3c30f) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.551486 10040 ts_tablet_manager.cc:1929] T c400101d2a39490e83722500839198cd P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.551584 10040 log.cc:1199] T c400101d2a39490e83722500839198cd P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/c400101d2a39490e83722500839198cd
I20251027 21:08:01.551855 10040 ts_tablet_manager.cc:1950] T c400101d2a39490e83722500839198cd P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:01.552399 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet c400101d2a39490e83722500839198cd (table default.Table_97b01c1840c543838263ca4222695f54 [id=bea51a58fda64d40916f306c4aa8ab4e]) successfully deleted
I20251027 21:08:01.552781 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_dd85b9fbcede45888d66f6226d2c5f2f" } modify_external_catalogs: true
I20251027 21:08:01.553031 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_dd85b9fbcede45888d66f6226d2c5f2f" } modify_external_catalogs: true
I20251027 21:08:01.555471 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 356a04cc6b7444df9ae17c12d5bff727 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.556051 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 292192b6d0d646de9144bbd6adbd4fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.537479 10043 log.cc:1199] T d8e008c868d04281beba0e5b204eb69a P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/d8e008c868d04281beba0e5b204eb69a
W20251027 21:08:01.560853 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 842c1a1298234d9ea2719bab29c54ae3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 842c1a1298234d9ea2719bab29c54ae3 already in progress: creating tablet
W20251027 21:08:01.561566 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet bdd382c4d376484cb7767d30681f966f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdd382c4d376484cb7767d30681f966f already in progress: creating tablet
I20251027 21:08:01.560550 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 8e07797daedc47ee9dfb5daa36b15eef (DEFAULT_TABLE table=default.Table_e9721dd5a67f48559827c11b5944ba0b [id=01c891709ba34d3c9f2cb11576a97527]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.562244 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e07797daedc47ee9dfb5daa36b15eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.564098 10040 tablet_replica.cc:333] T d8e008c868d04281beba0e5b204eb69a P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:01.564788 10040 raft_consensus.cc:2243] T d8e008c868d04281beba0e5b204eb69a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.565058 10040 raft_consensus.cc:2272] T d8e008c868d04281beba0e5b204eb69a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.566002 10040 ts_tablet_manager.cc:1916] T d8e008c868d04281beba0e5b204eb69a P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:01.566540 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet e04f4c5cf724419ab0be0e84d30a9a57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e04f4c5cf724419ab0be0e84d30a9a57 already in progress: creating tablet
W20251027 21:08:01.566771 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet ae0ee7cd2c1b406f84285f25420588d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae0ee7cd2c1b406f84285f25420588d0 already in progress: creating tablet
W20251027 21:08:01.566995 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 55a545185c0140e1aa1ddf3fb6ceba9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55a545185c0140e1aa1ddf3fb6ceba9a already in progress: creating tablet
I20251027 21:08:01.571257 10043 ts_tablet_manager.cc:1950] T d8e008c868d04281beba0e5b204eb69a P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:01.575340 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet d8e008c868d04281beba0e5b204eb69a (table default.Table_04ef6b3afcc34d84804cd6e9613d7bc2 [id=8a1cda11e1b3436cb8ffa2616d543a29]) successfully deleted
I20251027 21:08:01.577289 10747 raft_consensus.cc:493] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:01.577520 10747 raft_consensus.cc:515] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.578716 10747 leader_election.cc:290] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:01.578979 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2532bf7190064443a898e47e843fd9ac" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:01.579586 9859 leader_election.cc:343] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:01.581082 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 8e07797daedc47ee9dfb5daa36b15eef (DEFAULT_TABLE table=default.Table_e9721dd5a67f48559827c11b5944ba0b [id=01c891709ba34d3c9f2cb11576a97527]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.581404 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e07797daedc47ee9dfb5daa36b15eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.581792 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 8e07797daedc47ee9dfb5daa36b15eef (DEFAULT_TABLE table=default.Table_e9721dd5a67f48559827c11b5944ba0b [id=01c891709ba34d3c9f2cb11576a97527]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.582281 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e07797daedc47ee9dfb5daa36b15eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.591231 10032 tablet_bootstrap.cc:492] T a8328318a4f14b38a4ce36d183e99dfb P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:01.591384 10032 ts_tablet_manager.cc:1403] T a8328318a4f14b38a4ce36d183e99dfb P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.109s user 0.011s sys 0.039s
I20251027 21:08:01.580602 10040 ts_tablet_manager.cc:1929] T d8e008c868d04281beba0e5b204eb69a P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.591820 10040 log.cc:1199] T d8e008c868d04281beba0e5b204eb69a P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/d8e008c868d04281beba0e5b204eb69a
I20251027 21:08:01.591898 10041 ts_tablet_manager.cc:1929] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.592944 10029 tablet_bootstrap.cc:492] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:01.593153 10041 log.cc:1199] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/d872cf3ef39042ff9a615b55e7195afa
I20251027 21:08:01.593454 10041 ts_tablet_manager.cc:1950] T d872cf3ef39042ff9a615b55e7195afa P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:01.593612 10029 ts_tablet_manager.cc:1403] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.016s
I20251027 21:08:01.594342 10029 raft_consensus.cc:359] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.595438 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet d872cf3ef39042ff9a615b55e7195afa (table default.Table_2c2ec3f9d23142c9b8c2d66d1c979525 [id=59de715802f547bca16f181917fcc418]) successfully deleted
I20251027 21:08:01.596464 10029 raft_consensus.cc:385] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.596714 10041 tablet_replica.cc:333] T dd7029986a5246ee823e5fdb50683118 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:01.596870 10029 raft_consensus.cc:740] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.597288 10041 raft_consensus.cc:2243] T dd7029986a5246ee823e5fdb50683118 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.597582 10029 consensus_queue.cc:260] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.598210 10041 raft_consensus.cc:2272] T dd7029986a5246ee823e5fdb50683118 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.598537 10040 ts_tablet_manager.cc:1950] T d8e008c868d04281beba0e5b204eb69a P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:01.599002 10031 tablet_bootstrap.cc:492] T 8fc2d7d6601d4cc2849064f4cdddc90d P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:01.599145 10040 tablet_replica.cc:333] T e5d8837ff3c44a66a1f933d80540ba0c P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:01.599140 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet d8e008c868d04281beba0e5b204eb69a (table default.Table_04ef6b3afcc34d84804cd6e9613d7bc2 [id=8a1cda11e1b3436cb8ffa2616d543a29]) successfully deleted
I20251027 21:08:01.599368 10040 raft_consensus.cc:2243] T e5d8837ff3c44a66a1f933d80540ba0c P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.599674 10029 ts_tablet_manager.cc:1434] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251027 21:08:01.599165 10031 ts_tablet_manager.cc:1403] T 8fc2d7d6601d4cc2849064f4cdddc90d P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.076s user 0.009s sys 0.018s
I20251027 21:08:01.600512 10031 raft_consensus.cc:359] T 8fc2d7d6601d4cc2849064f4cdddc90d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.600696 10031 raft_consensus.cc:385] T 8fc2d7d6601d4cc2849064f4cdddc90d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.600756 10031 raft_consensus.cc:740] T 8fc2d7d6601d4cc2849064f4cdddc90d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.600996 10041 ts_tablet_manager.cc:1916] T dd7029986a5246ee823e5fdb50683118 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.601123 10031 consensus_queue.cc:260] T 8fc2d7d6601d4cc2849064f4cdddc90d P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.597846 10043 tablet_replica.cc:333] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:01.601861 10043 raft_consensus.cc:2243] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:01.602442 10031 ts_tablet_manager.cc:1434] T 8fc2d7d6601d4cc2849064f4cdddc90d P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:08:01.603641 10031 tablet_bootstrap.cc:492] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:01.604990 10031 tablet_bootstrap.cc:654] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.605160 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet a68bd9243b7541bbbc77014113e0106a
I20251027 21:08:01.605655 9305 catalog_manager.cc:6719] ReplaceTablet: tablet a68bd9243b7541bbbc77014113e0106a deleted and replaced by tablet 784df834432244e8812d1802f9e1a1bc
I20251027 21:08:01.605639 10040 raft_consensus.cc:2272] T e5d8837ff3c44a66a1f933d80540ba0c P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.605891 9305 master_service.cc:954] ReplaceTablet: received request to replace tablet 85ab6ac53e604ca0bfe051373606fe71 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:01.606092 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 7425b6b0184e4d0b871777b25d904ac4
I20251027 21:08:01.606123 10032 raft_consensus.cc:359] T a8328318a4f14b38a4ce36d183e99dfb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.606303 10032 raft_consensus.cc:385] T a8328318a4f14b38a4ce36d183e99dfb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.606362 10032 raft_consensus.cc:740] T a8328318a4f14b38a4ce36d183e99dfb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.606367 10040 ts_tablet_manager.cc:1916] T e5d8837ff3c44a66a1f933d80540ba0c P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.606566 9304 catalog_manager.cc:6719] ReplaceTablet: tablet 7425b6b0184e4d0b871777b25d904ac4 deleted and replaced by tablet 070167be7d8d4ddf8da561df14448360
I20251027 21:08:01.606546 10032 consensus_queue.cc:260] T a8328318a4f14b38a4ce36d183e99dfb P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.606879 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 7425b6b0184e4d0b871777b25d904ac4 with delete_type TABLET_DATA_DELETED (replaced by tablet 070167be7d8d4ddf8da561df14448360) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.606963 10032 ts_tablet_manager.cc:1434] T a8328318a4f14b38a4ce36d183e99dfb P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251027 21:08:01.607048 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet a68bd9243b7541bbbc77014113e0106a with delete_type TABLET_DATA_DELETED (replaced by tablet 784df834432244e8812d1802f9e1a1bc) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.607190 10032 tablet_bootstrap.cc:492] T 6600774e4c7b48479a2fdb02614ba505 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:01.607585 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet a68bd9243b7541bbbc77014113e0106a with delete_type TABLET_DATA_DELETED (replaced by tablet 784df834432244e8812d1802f9e1a1bc) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.609257 10032 tablet_bootstrap.cc:654] T 6600774e4c7b48479a2fdb02614ba505 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.610610 10041 ts_tablet_manager.cc:1929] T dd7029986a5246ee823e5fdb50683118 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.610749 10041 log.cc:1199] T dd7029986a5246ee823e5fdb50683118 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/dd7029986a5246ee823e5fdb50683118
I20251027 21:08:01.611100 10041 ts_tablet_manager.cc:1950] T dd7029986a5246ee823e5fdb50683118 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:01.611984 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet dd7029986a5246ee823e5fdb50683118 (table default.Table_79211c9993c54e4ea0455d01fec68b8e [id=421a64f7fb6f499bbf6f20fb38f736d0]) successfully deleted
I20251027 21:08:01.612284 10041 tablet_replica.cc:333] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:01.612471 10041 raft_consensus.cc:2243] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.612696 10041 raft_consensus.cc:2272] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.612859 10040 ts_tablet_manager.cc:1929] T e5d8837ff3c44a66a1f933d80540ba0c P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 21:08:01.613011 10040 log.cc:1199] T e5d8837ff3c44a66a1f933d80540ba0c P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/e5d8837ff3c44a66a1f933d80540ba0c
I20251027 21:08:01.613384 10040 ts_tablet_manager.cc:1950] T e5d8837ff3c44a66a1f933d80540ba0c P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:01.613862 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet e5d8837ff3c44a66a1f933d80540ba0c (table default.Table_2592614ec6a74410b7050171ee4d6d1b [id=7168e50c0645436b8c706086190b1736]) successfully deleted
I20251027 21:08:01.614252 10041 ts_tablet_manager.cc:1916] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.615840 10031 tablet_bootstrap.cc:492] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:01.615950 10031 ts_tablet_manager.cc:1403] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20251027 21:08:01.616492 10031 raft_consensus.cc:359] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.616636 10031 raft_consensus.cc:385] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.616690 10031 raft_consensus.cc:740] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.616891 10031 consensus_queue.cc:260] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.618100 10031 ts_tablet_manager.cc:1434] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:01.619122 10031 tablet_bootstrap.cc:492] T dc589852710342f7bd017193e4075303 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:01.620440 10031 tablet_bootstrap.cc:654] T dc589852710342f7bd017193e4075303 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.620982 10041 ts_tablet_manager.cc:1929] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 21:08:01.621119 10041 log.cc:1199] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/e5d8837ff3c44a66a1f933d80540ba0c
I20251027 21:08:01.621482 10041 ts_tablet_manager.cc:1950] T e5d8837ff3c44a66a1f933d80540ba0c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:01.622466 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet e5d8837ff3c44a66a1f933d80540ba0c (table default.Table_2592614ec6a74410b7050171ee4d6d1b [id=7168e50c0645436b8c706086190b1736]) successfully deleted
W20251027 21:08:01.623061 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet e1e58f6228254e3eb25dbf627a505ca1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1e58f6228254e3eb25dbf627a505ca1 already in progress: creating tablet
W20251027 21:08:01.623339 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet e1e58f6228254e3eb25dbf627a505ca1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1e58f6228254e3eb25dbf627a505ca1 already in progress: creating tablet
I20251027 21:08:01.623997 10041 tablet_replica.cc:333] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:01.624047 10040 tablet_replica.cc:333] T 21a8fc4a69f3486e92b1a68d10faed38 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:01.624195 10040 raft_consensus.cc:2243] T 21a8fc4a69f3486e92b1a68d10faed38 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.624322 10040 raft_consensus.cc:2272] T 21a8fc4a69f3486e92b1a68d10faed38 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.624451 10041 raft_consensus.cc:2243] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.624751 10041 raft_consensus.cc:2272] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.624805 10040 ts_tablet_manager.cc:1916] T 21a8fc4a69f3486e92b1a68d10faed38 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.625416 10041 ts_tablet_manager.cc:1916] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.626395 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b5b5182f58364f46849e1af83b27c6d8" } new_table_name: "default.Table_2a8ec86d85e7427597fe25a3c76c28b0" modify_external_catalogs: true
I20251027 21:08:01.629431 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c1eb65a428444379825e1d29557bbc23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.630281 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1eb65a428444379825e1d29557bbc23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.625389 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 7425b6b0184e4d0b871777b25d904ac4 with delete_type TABLET_DATA_DELETED (replaced by tablet 070167be7d8d4ddf8da561df14448360) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.632431 10041 ts_tablet_manager.cc:1929] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.632578 10041 log.cc:1199] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/71dc6b7a6ca146c6b0c58d9ad0ecbefa
I20251027 21:08:01.632910 10041 ts_tablet_manager.cc:1950] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:01.633803 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa (table default.Table_8fa71eed37b040fcb5f63d829b4a6315 [id=de5974f2ae38479fa2d403d5ac6e1dc5]) successfully deleted
I20251027 21:08:01.633947 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4ba3ecc07745498c97bd07deef4525c6" } new_table_name: "default.Table_991d9a1334ba4a7eb2a5d5ccee552553" modify_external_catalogs: true
I20251027 21:08:01.634150 10041 tablet_replica.cc:333] T 53cac7817db440e79e7416d625a9b2c8 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:01.634352 10041 raft_consensus.cc:2243] T 53cac7817db440e79e7416d625a9b2c8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.634462 10041 raft_consensus.cc:2272] T 53cac7817db440e79e7416d625a9b2c8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.635015 10041 ts_tablet_manager.cc:1916] T 53cac7817db440e79e7416d625a9b2c8 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.636183 10031 tablet_bootstrap.cc:492] T dc589852710342f7bd017193e4075303 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:01.636319 10031 ts_tablet_manager.cc:1403] T dc589852710342f7bd017193e4075303 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20251027 21:08:01.637351 10031 raft_consensus.cc:359] T dc589852710342f7bd017193e4075303 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.637496 10031 raft_consensus.cc:385] T dc589852710342f7bd017193e4075303 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.637564 10031 raft_consensus.cc:740] T dc589852710342f7bd017193e4075303 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.637799 10031 consensus_queue.cc:260] T dc589852710342f7bd017193e4075303 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.603907 10029 tablet_bootstrap.cc:492] T 343eaeca24dc4816ae032c7e674e9412 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:01.639168 10031 ts_tablet_manager.cc:1434] T dc589852710342f7bd017193e4075303 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:08:01.639393 10043 raft_consensus.cc:2272] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.603931 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2532bf7190064443a898e47e843fd9ac" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:01.640625 10029 tablet_bootstrap.cc:654] T 343eaeca24dc4816ae032c7e674e9412 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.640666 10681 raft_consensus.cc:493] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251027 21:08:01.640686 9860 leader_election.cc:343] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:01.640820 9860 leader_election.cc:304] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:01.640784 10681 raft_consensus.cc:515] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.641214 10800 raft_consensus.cc:2749] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:01.641453 10681 leader_election.cc:290] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:01.617344 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet 7425b6b0184e4d0b871777b25d904ac4 with delete_type TABLET_DATA_DELETED (replaced by tablet 070167be7d8d4ddf8da561df14448360) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.623529 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet a68bd9243b7541bbbc77014113e0106a with delete_type TABLET_DATA_DELETED (replaced by tablet 784df834432244e8812d1802f9e1a1bc) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.641911 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "132a01405a63447ab27b2137b803224f" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:01.642086 9769 raft_consensus.cc:2468] T 132a01405a63447ab27b2137b803224f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:01.641662 10031 tablet_bootstrap.cc:492] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:08:01.642547 9568 leader_election.cc:343] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 132a01405a63447ab27b2137b803224f
I20251027 21:08:01.642786 9566 leader_election.cc:304] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:01.643555 10681 raft_consensus.cc:2804] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:01.643596 10031 tablet_bootstrap.cc:654] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.643647 10681 raft_consensus.cc:493] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:01.643734 10681 raft_consensus.cc:3060] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:01.646878 10681 raft_consensus.cc:515] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.647600 10041 ts_tablet_manager.cc:1929] T 53cac7817db440e79e7416d625a9b2c8 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.647831 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "132a01405a63447ab27b2137b803224f" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:01.648002 9769 raft_consensus.cc:3060] T 132a01405a63447ab27b2137b803224f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:01.648463 9568 leader_election.cc:343] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 132a01405a63447ab27b2137b803224f
I20251027 21:08:01.648751 10681 leader_election.cc:290] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:01.649946 10041 log.cc:1199] T 53cac7817db440e79e7416d625a9b2c8 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/53cac7817db440e79e7416d625a9b2c8
I20251027 21:08:01.650310 10041 ts_tablet_manager.cc:1950] T 53cac7817db440e79e7416d625a9b2c8 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:01.650933 9769 raft_consensus.cc:2468] T 132a01405a63447ab27b2137b803224f P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:01.651263 9566 leader_election.cc:304] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:01.651690 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 1ca0ebb3fbf94ca9a532b209b6581729 (DEFAULT_TABLE table=default.Table_398fb57a71854dccb9a50f46010e7d5c [id=7aec81e01ae8493eac1efbd4b68156ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.651897 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 53cac7817db440e79e7416d625a9b2c8 (table default.Table_79a3420c8b6542868231c740293ab1fb [id=f3babbbf131543d2813ed9d5882e3d00]) successfully deleted
I20251027 21:08:01.652191 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca0ebb3fbf94ca9a532b209b6581729. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:01.652977 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet c27c699cc2744f40811f7b9395ca76be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c27c699cc2744f40811f7b9395ca76be already in progress: creating tablet
I20251027 21:08:01.653446 10041 tablet_replica.cc:333] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:01.653695 10040 ts_tablet_manager.cc:1929] T 21a8fc4a69f3486e92b1a68d10faed38 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.653789 10040 log.cc:1199] T 21a8fc4a69f3486e92b1a68d10faed38 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/21a8fc4a69f3486e92b1a68d10faed38
I20251027 21:08:01.653945 10041 raft_consensus.cc:2243] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.654045 10040 ts_tablet_manager.cc:1950] T 21a8fc4a69f3486e92b1a68d10faed38 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:01.654212 10041 raft_consensus.cc:2272] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.654495 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 21a8fc4a69f3486e92b1a68d10faed38 (table default.Table_ba1d973c05e14c70944d76ff1ecbc1bc [id=490e0e5ec83b4f76bf71eb41c900ac06]) successfully deleted
I20251027 21:08:01.654991 10041 ts_tablet_manager.cc:1916] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.657296 10681 raft_consensus.cc:2804] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
W20251027 21:08:01.657599 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c27c699cc2744f40811f7b9395ca76be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c27c699cc2744f40811f7b9395ca76be already in progress: creating tablet
I20251027 21:08:01.657689 10040 tablet_replica.cc:333] T f1aa5f7f76cb4a1ba15f016942f17067 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:01.657884 10040 raft_consensus.cc:2243] T f1aa5f7f76cb4a1ba15f016942f17067 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.658023 10040 raft_consensus.cc:2272] T f1aa5f7f76cb4a1ba15f016942f17067 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.658372 10681 raft_consensus.cc:697] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:01.658651 10681 consensus_queue.cc:237] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.659296 10040 ts_tablet_manager.cc:1916] T f1aa5f7f76cb4a1ba15f016942f17067 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.660691 10043 ts_tablet_manager.cc:1916] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.663025 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 1ca0ebb3fbf94ca9a532b209b6581729 (DEFAULT_TABLE table=default.Table_398fb57a71854dccb9a50f46010e7d5c [id=7aec81e01ae8493eac1efbd4b68156ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.663345 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca0ebb3fbf94ca9a532b209b6581729. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.665040 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 1ca0ebb3fbf94ca9a532b209b6581729 (DEFAULT_TABLE table=default.Table_398fb57a71854dccb9a50f46010e7d5c [id=7aec81e01ae8493eac1efbd4b68156ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.665378 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca0ebb3fbf94ca9a532b209b6581729. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.669941 9615 tablet_service.cc:1505] Processing CreateTablet for tablet b68c838a2d054604814ccf58765c233f (DEFAULT_TABLE table=default.Table_4aebdc4cec714b2cb02e19c014b27779 [id=3253fbfc6a9b47b2b20970cf1b25814e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.670271 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b68c838a2d054604814ccf58765c233f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.672161 10041 ts_tablet_manager.cc:1929] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.672247 10041 log.cc:1199] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/24ec2e0ba1994ab59c53977583bd2058
I20251027 21:08:01.672497 10041 ts_tablet_manager.cc:1950] T 24ec2e0ba1994ab59c53977583bd2058 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:01.673264 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 24ec2e0ba1994ab59c53977583bd2058 (table default.Table_2992b8e207024e06957ce7befb657acb [id=decffeb1481e4d798cffd8c9883be59b]) successfully deleted
W20251027 21:08:01.675580 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 9e9191649aa84b058abdc1477995ef67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e9191649aa84b058abdc1477995ef67 already in progress: creating tablet
W20251027 21:08:01.676347 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 0d2d977e360b4ed29c9ed63a309783d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d2d977e360b4ed29c9ed63a309783d6 already in progress: creating tablet
W20251027 21:08:01.677227 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 356a04cc6b7444df9ae17c12d5bff727 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 356a04cc6b7444df9ae17c12d5bff727 already in progress: creating tablet
W20251027 21:08:01.678212 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 292192b6d0d646de9144bbd6adbd4fbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 292192b6d0d646de9144bbd6adbd4fbe already in progress: creating tablet
I20251027 21:08:01.678730 10041 tablet_replica.cc:333] T 84b9de82d55447bba2a897edb683e5c5 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:01.679072 10041 raft_consensus.cc:2243] T 84b9de82d55447bba2a897edb683e5c5 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.679414 10041 raft_consensus.cc:2272] T 84b9de82d55447bba2a897edb683e5c5 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.690706 10040 ts_tablet_manager.cc:1929] T f1aa5f7f76cb4a1ba15f016942f17067 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.690816 10040 log.cc:1199] T f1aa5f7f76cb4a1ba15f016942f17067 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/f1aa5f7f76cb4a1ba15f016942f17067
I20251027 21:08:01.691114 10040 ts_tablet_manager.cc:1950] T f1aa5f7f76cb4a1ba15f016942f17067 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:01.692152 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet f1aa5f7f76cb4a1ba15f016942f17067 (table default.Table_3e0f707f8e2844bfaf937150b747a935 [id=d7dcec1e03494fd5a219a97ea8b1ab1d]) successfully deleted
I20251027 21:08:01.695787 10040 tablet_replica.cc:333] T 132a01405a63447ab27b2137b803224f P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:01.695952 10040 raft_consensus.cc:2243] T 132a01405a63447ab27b2137b803224f P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.696084 10040 raft_consensus.cc:2272] T 132a01405a63447ab27b2137b803224f P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.696520 10040 ts_tablet_manager.cc:1916] T 132a01405a63447ab27b2137b803224f P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.701061 9750 tablet_service.cc:1505] Processing CreateTablet for tablet b68c838a2d054604814ccf58765c233f (DEFAULT_TABLE table=default.Table_4aebdc4cec714b2cb02e19c014b27779 [id=3253fbfc6a9b47b2b20970cf1b25814e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.701493 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b68c838a2d054604814ccf58765c233f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.703853 9895 tablet_service.cc:1505] Processing CreateTablet for tablet b68c838a2d054604814ccf58765c233f (DEFAULT_TABLE table=default.Table_4aebdc4cec714b2cb02e19c014b27779 [id=3253fbfc6a9b47b2b20970cf1b25814e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.704106 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 473cc9f814c943a8967d92e58ed7dfd2 (DEFAULT_TABLE table=default.Table_feecf5aa70d7443293e8b3af119d90e4 [id=7149fc3eb515408dbb8df478907e31f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.704226 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b68c838a2d054604814ccf58765c233f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.704427 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 473cc9f814c943a8967d92e58ed7dfd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.711304 9309 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet a9339c6b1154400eace53d765528fdf9 (table default.Table_5a6211c8465040dbad1cc86377a370ea [id=91fb9f95fe9343fca1d8cb0d0f781afb]). Expected version 2 got 0
W20251027 21:08:01.711944 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 83f5fbf5757a49df93ab949b7657c61f consensus state has no leader
I20251027 21:08:01.717052 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet ef6f69db3d2547c88bbc8df2d52e2fb1
I20251027 21:08:01.717579 9311 catalog_manager.cc:6719] ReplaceTablet: tablet ef6f69db3d2547c88bbc8df2d52e2fb1 deleted and replaced by tablet eaa869a2e41b4f83978b7e51c4717edd
I20251027 21:08:01.718434 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet ef6f69db3d2547c88bbc8df2d52e2fb1 with delete_type TABLET_DATA_DELETED (replaced by tablet eaa869a2e41b4f83978b7e51c4717edd) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.718590 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_263a9587a19c451b81a428c0891f84b9" } new_table_name: "default.Table_9581624c205746a19f66b0619cfbd6cf" modify_external_catalogs: true
I20251027 21:08:01.719928 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet edbf5f0db6fa40c7b24f1cd20ee80dd4
I20251027 21:08:01.719951 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet ef6f69db3d2547c88bbc8df2d52e2fb1 with delete_type TABLET_DATA_DELETED (replaced by tablet eaa869a2e41b4f83978b7e51c4717edd) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.720031 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_342bf4fd6c7f4ffbb80cc0f87baf3768"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.720515 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_342bf4fd6c7f4ffbb80cc0f87baf3768 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.721113 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c3f4e692c08244afa2c79e0a075bbcdb
I20251027 21:08:01.722065 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e832c43c91c74891aaed15942e4b96dd" } new_table_name: "default.Table_9e349c15adbf444f96f4d83d3acb5990" modify_external_catalogs: true
I20251027 21:08:01.722705 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet edbf5f0db6fa40c7b24f1cd20ee80dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.722909 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet c3f4e692c08244afa2c79e0a075bbcdb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.724943 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet ef6f69db3d2547c88bbc8df2d52e2fb1 with delete_type TABLET_DATA_DELETED (replaced by tablet eaa869a2e41b4f83978b7e51c4717edd) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.725147 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet edbf5f0db6fa40c7b24f1cd20ee80dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.725297 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet c3f4e692c08244afa2c79e0a075bbcdb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.725605 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c6b87d4cea6a4f8eb5226bbdd13f1434"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.726078 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6b87d4cea6a4f8eb5226bbdd13f1434 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.728581 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet a8328318a4f14b38a4ce36d183e99dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.728708 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet c3f4e692c08244afa2c79e0a075bbcdb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.720537 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6b452ea1c8f14d038a841d02b164c04a" } modify_external_catalogs: true
I20251027 21:08:01.728879 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet edbf5f0db6fa40c7b24f1cd20ee80dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.728919 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6b452ea1c8f14d038a841d02b164c04a" } modify_external_catalogs: true
I20251027 21:08:01.729887 10040 ts_tablet_manager.cc:1929] T 132a01405a63447ab27b2137b803224f P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.730005 10040 log.cc:1199] T 132a01405a63447ab27b2137b803224f P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/132a01405a63447ab27b2137b803224f
I20251027 21:08:01.730854 10040 ts_tablet_manager.cc:1950] T 132a01405a63447ab27b2137b803224f P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:01.731832 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet e841c54a94d649c19b639f5021f150cd with delete_type TABLET_DATA_DELETED (replaced by tablet 666842b8e54c4561989fd9401210464b) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.731860 9754 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bf9c2a78aac42528d67846dc58adc40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.732002 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 8fc2d7d6601d4cc2849064f4cdddc90d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.732170 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet dc589852710342f7bd017193e4075303 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.732452 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet ed5619d295a246379b625f35692203e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.733016 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 69da799c886841a19002d3c7c5084e73 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.733189 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet d52d71f4976143e4a514350621ebe309 with delete_type TABLET_DATA_DELETED (replaced by tablet e419e588b31d48c8bc19d7948081cc81) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.735088 10040 tablet_replica.cc:333] T a287439ea6c9469da5fb56d84ee68048 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:01.735275 10040 raft_consensus.cc:2243] T a287439ea6c9469da5fb56d84ee68048 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.735301 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 132a01405a63447ab27b2137b803224f (table default.Table_673503358b54481c8877034c26349796 [id=c6aed7584a0b49d89bf9a8f8ef0cb429]) successfully deleted
I20251027 21:08:01.735410 10040 raft_consensus.cc:2272] T a287439ea6c9469da5fb56d84ee68048 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:01.735556 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 9e9191649aa84b058abdc1477995ef67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e9191649aa84b058abdc1477995ef67 already in progress: creating tablet
I20251027 21:08:01.736218 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e8c83b501d8426fa0cad6a89f9a0b9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.736726 10040 ts_tablet_manager.cc:1916] T a287439ea6c9469da5fb56d84ee68048 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.736739 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 132a01405a63447ab27b2137b803224f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.736888 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a58cd07e1bb4ffdbf39e7e16a46022a with delete_type TABLET_DATA_DELETED (replaced by tablet 4d88aacbf47144cb837b99d68f6331e2) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.738814 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_32b31b9bf26542e69fb03fea9edd07bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.739442 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32b31b9bf26542e69fb03fea9edd07bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.742156 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 473cc9f814c943a8967d92e58ed7dfd2 (DEFAULT_TABLE table=default.Table_feecf5aa70d7443293e8b3af119d90e4 [id=7149fc3eb515408dbb8df478907e31f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.742468 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 473cc9f814c943a8967d92e58ed7dfd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.742522 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 473cc9f814c943a8967d92e58ed7dfd2 (DEFAULT_TABLE table=default.Table_feecf5aa70d7443293e8b3af119d90e4 [id=7149fc3eb515408dbb8df478907e31f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.742875 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 473cc9f814c943a8967d92e58ed7dfd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.747014 10032 tablet_bootstrap.cc:492] T 6600774e4c7b48479a2fdb02614ba505 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:01.747130 10032 ts_tablet_manager.cc:1403] T 6600774e4c7b48479a2fdb02614ba505 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.140s user 0.008s sys 0.007s
I20251027 21:08:01.747520 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 1ea10027bc7246e98092b894241d9d61 (DEFAULT_TABLE table=default.Table_ec8494b7496d4d08828dce88b99c18f2 [id=3880c4f98ea04180bb7415e117ed6373]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.747820 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ea10027bc7246e98092b894241d9d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.747874 10032 raft_consensus.cc:359] T 6600774e4c7b48479a2fdb02614ba505 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.748025 10032 raft_consensus.cc:385] T 6600774e4c7b48479a2fdb02614ba505 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.748095 10032 raft_consensus.cc:740] T 6600774e4c7b48479a2fdb02614ba505 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.748342 10032 consensus_queue.cc:260] T 6600774e4c7b48479a2fdb02614ba505 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.750365 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 11e6720681324de0b724ffac7f816d30 (DEFAULT_TABLE table=default.Table_3f1fa06443254429a4f8cd54e07636cd [id=7f6eed67cfe3454e8589072361c04910]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.750613 10043 ts_tablet_manager.cc:1929] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 21:08:01.750661 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11e6720681324de0b724ffac7f816d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.751137 10041 ts_tablet_manager.cc:1916] T 84b9de82d55447bba2a897edb683e5c5 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.754333 9898 tablet_service.cc:1505] Processing CreateTablet for tablet d106dacad8fb48ceab13e9b1eb63d82f (DEFAULT_TABLE table=default.Table_fb6d994c828141ddb43073cb17d9fad6 [id=179e87ee0bc14ba5972beac3c84d1bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.754640 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d106dacad8fb48ceab13e9b1eb63d82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.754689 10032 ts_tablet_manager.cc:1434] T 6600774e4c7b48479a2fdb02614ba505 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251027 21:08:01.756560 9611 tablet_service.cc:1505] Processing CreateTablet for tablet aadae0184a8e454cbe40d0c8491d9bd5 (DEFAULT_TABLE table=default.Table_46c804a5818a4a4ca815f2449ccbe842 [id=86ff92656faf49768be876994d62045f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.756892 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aadae0184a8e454cbe40d0c8491d9bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.757903 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 915f250f9fd046cfa9c6590bd8e4f33c (DEFAULT_TABLE table=default.Table_86b3f1d7a96f4c788e7ab443853fd5cd [id=70eb7e3452dc438cbdd9f12917bb8c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.758056 10031 tablet_bootstrap.cc:492] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:01.758198 10031 ts_tablet_manager.cc:1403] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.117s user 0.010s sys 0.014s
I20251027 21:08:01.758345 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 7e740ddf3b394d3a823bc95bad48fa99 (DEFAULT_TABLE table=default.Table_ec0b0d1ae0774aa598e7311546fcedd4 [id=0216c82db6554eb88e26cce5db5598c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.758476 10032 tablet_bootstrap.cc:492] T a339c0dee0994788b8644912f74c1b82 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:01.758867 10031 raft_consensus.cc:359] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.759025 10031 raft_consensus.cc:385] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.759073 10031 raft_consensus.cc:740] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.759265 9601 tablet_service.cc:1505] Processing CreateTablet for tablet d106dacad8fb48ceab13e9b1eb63d82f (DEFAULT_TABLE table=default.Table_fb6d994c828141ddb43073cb17d9fad6 [id=179e87ee0bc14ba5972beac3c84d1bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.759244 10031 consensus_queue.cc:260] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.759543 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d106dacad8fb48ceab13e9b1eb63d82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.759960 10031 ts_tablet_manager.cc:1434] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:01.761096 10040 ts_tablet_manager.cc:1929] T a287439ea6c9469da5fb56d84ee68048 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.761191 10040 log.cc:1199] T a287439ea6c9469da5fb56d84ee68048 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/a287439ea6c9469da5fb56d84ee68048
I20251027 21:08:01.761478 10040 ts_tablet_manager.cc:1950] T a287439ea6c9469da5fb56d84ee68048 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:01.761890 10031 tablet_bootstrap.cc:492] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:01.762357 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 915f250f9fd046cfa9c6590bd8e4f33c (DEFAULT_TABLE table=default.Table_86b3f1d7a96f4c788e7ab443853fd5cd [id=70eb7e3452dc438cbdd9f12917bb8c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.762617 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 915f250f9fd046cfa9c6590bd8e4f33c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.762750 10032 tablet_bootstrap.cc:654] T a339c0dee0994788b8644912f74c1b82 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:01.764712 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet efbabe357ae947e09b1902b1fbc0ee40 consensus state has no leader
I20251027 21:08:01.764828 10031 tablet_bootstrap.cc:654] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.764923 9304 master_service.cc:954] ReplaceTablet: received request to replace tablet 33c5bc33130a424fb63e3f92a2cc49fc from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:01.766378 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 7e740ddf3b394d3a823bc95bad48fa99 (DEFAULT_TABLE table=default.Table_ec0b0d1ae0774aa598e7311546fcedd4 [id=0216c82db6554eb88e26cce5db5598c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.766695 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e740ddf3b394d3a823bc95bad48fa99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.768844 10040 tablet_replica.cc:333] T 9b04349b15f3494bb0933db56b7b24bc P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:01.769023 10040 raft_consensus.cc:2243] T 9b04349b15f3494bb0933db56b7b24bc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.769160 10040 raft_consensus.cc:2272] T 9b04349b15f3494bb0933db56b7b24bc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.771126 10040 ts_tablet_manager.cc:1916] T 9b04349b15f3494bb0933db56b7b24bc P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.772325 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 915f250f9fd046cfa9c6590bd8e4f33c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.774281 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e740ddf3b394d3a823bc95bad48fa99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.779042 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet a287439ea6c9469da5fb56d84ee68048 (table default.Table_1ecaac632a024248b04e7c40e2c01d2c [id=4e2210c84e4c464da017d1b36ddb0a37]) successfully deleted
I20251027 21:08:01.779253 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 1ea10027bc7246e98092b894241d9d61 (DEFAULT_TABLE table=default.Table_ec8494b7496d4d08828dce88b99c18f2 [id=3880c4f98ea04180bb7415e117ed6373]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.779726 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ea10027bc7246e98092b894241d9d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.779435 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 11e6720681324de0b724ffac7f816d30 (DEFAULT_TABLE table=default.Table_3f1fa06443254429a4f8cd54e07636cd [id=7f6eed67cfe3454e8589072361c04910]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.780148 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11e6720681324de0b724ffac7f816d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.785508 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 11e6720681324de0b724ffac7f816d30 (DEFAULT_TABLE table=default.Table_3f1fa06443254429a4f8cd54e07636cd [id=7f6eed67cfe3454e8589072361c04910]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.779386 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 1ea10027bc7246e98092b894241d9d61 (DEFAULT_TABLE table=default.Table_ec8494b7496d4d08828dce88b99c18f2 [id=3880c4f98ea04180bb7415e117ed6373]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.785799 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11e6720681324de0b724ffac7f816d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.785928 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ea10027bc7246e98092b894241d9d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.790755 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 915f250f9fd046cfa9c6590bd8e4f33c (DEFAULT_TABLE table=default.Table_86b3f1d7a96f4c788e7ab443853fd5cd [id=70eb7e3452dc438cbdd9f12917bb8c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.791122 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 915f250f9fd046cfa9c6590bd8e4f33c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.791478 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 85ab6ac53e604ca0bfe051373606fe71
I20251027 21:08:01.791937 9305 catalog_manager.cc:6719] ReplaceTablet: tablet 85ab6ac53e604ca0bfe051373606fe71 deleted and replaced by tablet 312dc1e8caea4627b268536a6ce6344d
I20251027 21:08:01.793928 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c564272cb5b445acbb5a98d619938f00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:01.794420 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 7e740ddf3b394d3a823bc95bad48fa99 (DEFAULT_TABLE table=default.Table_ec0b0d1ae0774aa598e7311546fcedd4 [id=0216c82db6554eb88e26cce5db5598c7]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:01.794692 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d8224d24b1144ff0b8bc593fc4e244de consensus state has no leader
I20251027 21:08:01.794737 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e740ddf3b394d3a823bc95bad48fa99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.795089 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_233d6aa47dba4a72beb99b61ba19983b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.795775 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca12dafba2ec4243812969c1b3e3d6c0 has no consensus state
I20251027 21:08:01.796142 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_368edee03055430fafa818f57694096b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:01.796370 9305 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 6600774e4c7b48479a2fdb02614ba505 (table default.Table_d7530773668f42c5a07a921df02a95ab [id=914b728e358e4fdba6d38d6c7beadaca]). Expected version 1 got 0
W20251027 21:08:01.796619 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_368edee03055430fafa818f57694096b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.797065 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_01b9845ac6a54380a39c0a76e0db231e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.797466 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01b9845ac6a54380a39c0a76e0db231e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:01.797866 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ada34d5086484ef18e2050ac05b71725 has no consensus state
I20251027 21:08:01.798197 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3c0c40b875fc45969c1a916c5545c4cb" } modify_external_catalogs: true
I20251027 21:08:01.798362 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3c0c40b875fc45969c1a916c5545c4cb" } modify_external_catalogs: true
I20251027 21:08:01.799032 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 85ab6ac53e604ca0bfe051373606fe71 with delete_type TABLET_DATA_DELETED (replaced by tablet 312dc1e8caea4627b268536a6ce6344d) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.799211 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f1129eecfb54448965222269d9c32b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:08:01.800567 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_233d6aa47dba4a72beb99b61ba19983b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:01.801826 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c564272cb5b445acbb5a98d619938f00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.802094 10718 raft_consensus.cc:493] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:01.802253 10718 raft_consensus.cc:515] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.802444 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet 85ab6ac53e604ca0bfe051373606fe71 with delete_type TABLET_DATA_DELETED (replaced by tablet 312dc1e8caea4627b268536a6ce6344d) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.802810 10718 leader_election.cc:290] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:01.803668 9710 leader_election.cc:343] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 6bf9c2a78aac42528d67846dc58adc40
I20251027 21:08:01.804122 9310 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 6600774e4c7b48479a2fdb02614ba505 (table default.Table_d7530773668f42c5a07a921df02a95ab [id=914b728e358e4fdba6d38d6c7beadaca]). Expected version 1 got 0
I20251027 21:08:01.804720 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ec8494b7496d4d08828dce88b99c18f2" } modify_external_catalogs: true
I20251027 21:08:01.804852 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ec8494b7496d4d08828dce88b99c18f2" } modify_external_catalogs: true
W20251027 21:08:01.806205 9711 leader_election.cc:343] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 6bf9c2a78aac42528d67846dc58adc40
I20251027 21:08:01.806308 9711 leader_election.cc:304] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:01.806644 10718 raft_consensus.cc:2749] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:01.808022 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 85ab6ac53e604ca0bfe051373606fe71 with delete_type TABLET_DATA_DELETED (replaced by tablet 312dc1e8caea4627b268536a6ce6344d) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.808332 9755 tablet_service.cc:1505] Processing CreateTablet for tablet d106dacad8fb48ceab13e9b1eb63d82f (DEFAULT_TABLE table=default.Table_fb6d994c828141ddb43073cb17d9fad6 [id=179e87ee0bc14ba5972beac3c84d1bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.808609 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d106dacad8fb48ceab13e9b1eb63d82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.814889 10043 log.cc:1199] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/e5d8837ff3c44a66a1f933d80540ba0c
I20251027 21:08:01.815342 10043 ts_tablet_manager.cc:1950] T e5d8837ff3c44a66a1f933d80540ba0c P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:01.819581 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet e5d8837ff3c44a66a1f933d80540ba0c (table default.Table_2592614ec6a74410b7050171ee4d6d1b [id=7168e50c0645436b8c706086190b1736]) successfully deleted
I20251027 21:08:01.821915 10040 ts_tablet_manager.cc:1929] T 9b04349b15f3494bb0933db56b7b24bc P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.822028 10040 log.cc:1199] T 9b04349b15f3494bb0933db56b7b24bc P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/9b04349b15f3494bb0933db56b7b24bc
I20251027 21:08:01.822358 10040 ts_tablet_manager.cc:1950] T 9b04349b15f3494bb0933db56b7b24bc P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
W20251027 21:08:01.825677 9296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20251027 21:08:01.828944 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet e1e58f6228254e3eb25dbf627a505ca1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1e58f6228254e3eb25dbf627a505ca1 already in progress: creating tablet
I20251027 21:08:01.831259 10043 tablet_replica.cc:333] T fe24af9838954a9bb680474800642e62 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:01.831437 10043 raft_consensus.cc:2243] T fe24af9838954a9bb680474800642e62 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.831784 10040 tablet_replica.cc:333] T e841c54a94d649c19b639f5021f150cd P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:01.831753 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 9b04349b15f3494bb0933db56b7b24bc (table default.Table_2ff5a5b701c540b8a9eb8b87aace0e77 [id=69f10c0f3f3a4227ab4c35e939085634]) successfully deleted
I20251027 21:08:01.831954 10040 raft_consensus.cc:2243] T e841c54a94d649c19b639f5021f150cd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.832124 10040 raft_consensus.cc:2272] T e841c54a94d649c19b639f5021f150cd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:01.832103 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 0d2d977e360b4ed29c9ed63a309783d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d2d977e360b4ed29c9ed63a309783d6 already in progress: creating tablet
W20251027 21:08:01.832401 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 356a04cc6b7444df9ae17c12d5bff727 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 356a04cc6b7444df9ae17c12d5bff727 already in progress: creating tablet
I20251027 21:08:01.832641 10040 ts_tablet_manager.cc:1916] T e841c54a94d649c19b639f5021f150cd P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.832703 10043 raft_consensus.cc:2272] T fe24af9838954a9bb680474800642e62 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.833187 10043 ts_tablet_manager.cc:1916] T fe24af9838954a9bb680474800642e62 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:01.834932 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 292192b6d0d646de9144bbd6adbd4fbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 292192b6d0d646de9144bbd6adbd4fbe already in progress: creating tablet
I20251027 21:08:01.835006 10031 tablet_bootstrap.cc:492] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:01.835140 10031 ts_tablet_manager.cc:1403] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.001s
W20251027 21:08:01.835310 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 7425b6b0184e4d0b871777b25d904ac4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7425b6b0184e4d0b871777b25d904ac4 already in progress: creating tablet
W20251027 21:08:01.835548 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet a68bd9243b7541bbbc77014113e0106a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a68bd9243b7541bbbc77014113e0106a already in progress: creating tablet
I20251027 21:08:01.835619 9755 tablet_service.cc:1505] Processing CreateTablet for tablet aadae0184a8e454cbe40d0c8491d9bd5 (DEFAULT_TABLE table=default.Table_46c804a5818a4a4ca815f2449ccbe842 [id=86ff92656faf49768be876994d62045f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:01.835834 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet ef6f69db3d2547c88bbc8df2d52e2fb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef6f69db3d2547c88bbc8df2d52e2fb1 already in progress: creating tablet
I20251027 21:08:01.835932 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aadae0184a8e454cbe40d0c8491d9bd5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:01.836128 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c3f4e692c08244afa2c79e0a075bbcdb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3f4e692c08244afa2c79e0a075bbcdb already in progress: creating tablet
W20251027 21:08:01.836442 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet edbf5f0db6fa40c7b24f1cd20ee80dd4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edbf5f0db6fa40c7b24f1cd20ee80dd4 already in progress: creating tablet
I20251027 21:08:01.836666 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 42d5df2a47b744ffb0bf3a34201903d7 (DEFAULT_TABLE table=default.Table_090167e6602f4bff86b275f39d9a58cb [id=dcf7afada025495aaa91d6548b9162aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.836915 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 42d5df2a47b744ffb0bf3a34201903d7 (DEFAULT_TABLE table=default.Table_090167e6602f4bff86b275f39d9a58cb [id=dcf7afada025495aaa91d6548b9162aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.837013 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42d5df2a47b744ffb0bf3a34201903d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.837294 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42d5df2a47b744ffb0bf3a34201903d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.839298 10041 ts_tablet_manager.cc:1929] T 84b9de82d55447bba2a897edb683e5c5 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.839421 10041 log.cc:1199] T 84b9de82d55447bba2a897edb683e5c5 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/84b9de82d55447bba2a897edb683e5c5
I20251027 21:08:01.839685 10041 ts_tablet_manager.cc:1950] T 84b9de82d55447bba2a897edb683e5c5 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:01.840601 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 84b9de82d55447bba2a897edb683e5c5 (table default.Table_4755cbfcd1464ec1b6f0b909b99f0c3f [id=6b70bbbf477d4b788b3ffb10c9c7d7c9]) successfully deleted
I20251027 21:08:01.840981 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 42d5df2a47b744ffb0bf3a34201903d7 (DEFAULT_TABLE table=default.Table_090167e6602f4bff86b275f39d9a58cb [id=dcf7afada025495aaa91d6548b9162aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.841373 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42d5df2a47b744ffb0bf3a34201903d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.843623 9600 tablet_service.cc:1505] Processing CreateTablet for tablet aeae07c3906145149cea31e5a7c466fe (DEFAULT_TABLE table=default.Table_7a535e8483da496498c4c7f7e76d6670 [id=53f813baffa448c990d467d8267929ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.843963 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeae07c3906145149cea31e5a7c466fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.844003 9609 tablet_service.cc:1505] Processing CreateTablet for tablet ee0c598904004dd18df058da483aa45c (DEFAULT_TABLE table=default.Table_94b3855384ef4019bf69b5ff2ccc6dd6 [id=017a3899f6a34e0ea04892c0a8bc4225]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.844383 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee0c598904004dd18df058da483aa45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.846441 9898 tablet_service.cc:1505] Processing CreateTablet for tablet ee0c598904004dd18df058da483aa45c (DEFAULT_TABLE table=default.Table_94b3855384ef4019bf69b5ff2ccc6dd6 [id=017a3899f6a34e0ea04892c0a8bc4225]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.846802 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee0c598904004dd18df058da483aa45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.848811 9899 tablet_service.cc:1505] Processing CreateTablet for tablet aeae07c3906145149cea31e5a7c466fe (DEFAULT_TABLE table=default.Table_7a535e8483da496498c4c7f7e76d6670 [id=53f813baffa448c990d467d8267929ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.849105 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeae07c3906145149cea31e5a7c466fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.849531 10031 raft_consensus.cc:359] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.849678 10031 raft_consensus.cc:385] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.849730 10031 raft_consensus.cc:740] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.849901 10031 consensus_queue.cc:260] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.850409 10031 ts_tablet_manager.cc:1434] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251027 21:08:01.850433 9905 tablet_service.cc:1505] Processing CreateTablet for tablet aadae0184a8e454cbe40d0c8491d9bd5 (DEFAULT_TABLE table=default.Table_46c804a5818a4a4ca815f2449ccbe842 [id=86ff92656faf49768be876994d62045f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.850837 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aadae0184a8e454cbe40d0c8491d9bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.851419 10031 tablet_bootstrap.cc:492] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:01.854748 9749 tablet_service.cc:1505] Processing CreateTablet for tablet ee0c598904004dd18df058da483aa45c (DEFAULT_TABLE table=default.Table_94b3855384ef4019bf69b5ff2ccc6dd6 [id=017a3899f6a34e0ea04892c0a8bc4225]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.855080 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee0c598904004dd18df058da483aa45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.857616 10041 tablet_replica.cc:333] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:01.858346 9750 tablet_service.cc:1505] Processing CreateTablet for tablet aeae07c3906145149cea31e5a7c466fe (DEFAULT_TABLE table=default.Table_7a535e8483da496498c4c7f7e76d6670 [id=53f813baffa448c990d467d8267929ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:01.858438 10041 raft_consensus.cc:2243] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.858640 10041 raft_consensus.cc:2272] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.858659 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeae07c3906145149cea31e5a7c466fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:01.867594 10041 ts_tablet_manager.cc:1916] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.869306 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_52c44a2a153c46ec823f80c2c4fbb600"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:01.869794 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52c44a2a153c46ec823f80c2c4fbb600 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.872551 10031 tablet_bootstrap.cc:654] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.875700 10041 ts_tablet_manager.cc:1929] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.875806 10041 log.cc:1199] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/2532bf7190064443a898e47e843fd9ac
I20251027 21:08:01.866971 10029 tablet_bootstrap.cc:492] T 343eaeca24dc4816ae032c7e674e9412 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:01.876120 10041 ts_tablet_manager.cc:1950] T 2532bf7190064443a898e47e843fd9ac P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:01.876184 10029 ts_tablet_manager.cc:1403] T 343eaeca24dc4816ae032c7e674e9412 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.272s user 0.025s sys 0.022s
I20251027 21:08:01.876899 10029 raft_consensus.cc:359] T 343eaeca24dc4816ae032c7e674e9412 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.876972 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 2532bf7190064443a898e47e843fd9ac (table default.Table_89e2320f599341e882a2d09c0985b220 [id=c6f20d8985134d9a8d3c5bb4a51b45fb]) successfully deleted
I20251027 21:08:01.877059 10029 raft_consensus.cc:385] T 343eaeca24dc4816ae032c7e674e9412 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.877035 10041 tablet_replica.cc:333] T 26872be132fb46ee9e79a9c2e62374ef P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:01.877125 10029 raft_consensus.cc:740] T 343eaeca24dc4816ae032c7e674e9412 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.877260 10041 raft_consensus.cc:2243] T 26872be132fb46ee9e79a9c2e62374ef P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.877396 10041 raft_consensus.cc:2272] T 26872be132fb46ee9e79a9c2e62374ef P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.877343 10029 consensus_queue.cc:260] T 343eaeca24dc4816ae032c7e674e9412 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.877892 10041 ts_tablet_manager.cc:1916] T 26872be132fb46ee9e79a9c2e62374ef P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.879276 10040 ts_tablet_manager.cc:1929] T e841c54a94d649c19b639f5021f150cd P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.879410 10040 log.cc:1199] T e841c54a94d649c19b639f5021f150cd P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/e841c54a94d649c19b639f5021f150cd
I20251027 21:08:01.879706 10040 ts_tablet_manager.cc:1950] T e841c54a94d649c19b639f5021f150cd P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:01.868284 10043 ts_tablet_manager.cc:1929] T fe24af9838954a9bb680474800642e62 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.880803 10043 log.cc:1199] T fe24af9838954a9bb680474800642e62 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/fe24af9838954a9bb680474800642e62
I20251027 21:08:01.881129 10043 ts_tablet_manager.cc:1950] T fe24af9838954a9bb680474800642e62 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:01.882047 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet fe24af9838954a9bb680474800642e62 (table default.Table_fc695aa989384c1ba355a09381428926 [id=dae1114f6b50443f8f6b79d18619d868]) successfully deleted
I20251027 21:08:01.882954 10043 tablet_replica.cc:333] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
W20251027 21:08:01.882994 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet c27c699cc2744f40811f7b9395ca76be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c27c699cc2744f40811f7b9395ca76be already in progress: creating tablet
I20251027 21:08:01.883131 10043 raft_consensus.cc:2243] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251027 21:08:01.883209 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 9e9191649aa84b058abdc1477995ef67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e9191649aa84b058abdc1477995ef67 already in progress: creating tablet
I20251027 21:08:01.883242 10043 raft_consensus.cc:2272] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:01.883404 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 0d2d977e360b4ed29c9ed63a309783d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d2d977e360b4ed29c9ed63a309783d6 already in progress: creating tablet
W20251027 21:08:01.883569 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 356a04cc6b7444df9ae17c12d5bff727 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 356a04cc6b7444df9ae17c12d5bff727 already in progress: creating tablet
W20251027 21:08:01.883756 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 292192b6d0d646de9144bbd6adbd4fbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 292192b6d0d646de9144bbd6adbd4fbe already in progress: creating tablet
W20251027 21:08:01.883944 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 7425b6b0184e4d0b871777b25d904ac4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7425b6b0184e4d0b871777b25d904ac4 already in progress: creating tablet
I20251027 21:08:01.884061 10041 ts_tablet_manager.cc:1929] T 26872be132fb46ee9e79a9c2e62374ef P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.884153 10041 log.cc:1199] T 26872be132fb46ee9e79a9c2e62374ef P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/26872be132fb46ee9e79a9c2e62374ef
W20251027 21:08:01.884169 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet a68bd9243b7541bbbc77014113e0106a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a68bd9243b7541bbbc77014113e0106a already in progress: creating tablet
I20251027 21:08:01.884395 10041 ts_tablet_manager.cc:1950] T 26872be132fb46ee9e79a9c2e62374ef P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
W20251027 21:08:01.884908 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet ef6f69db3d2547c88bbc8df2d52e2fb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef6f69db3d2547c88bbc8df2d52e2fb1 already in progress: creating tablet
W20251027 21:08:01.885198 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet edbf5f0db6fa40c7b24f1cd20ee80dd4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edbf5f0db6fa40c7b24f1cd20ee80dd4 already in progress: creating tablet
I20251027 21:08:01.885249 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 26872be132fb46ee9e79a9c2e62374ef (table default.Table_b23313ec4d7b4e32a4283dd404578616 [id=6a545a959eb74d0eb589f1be92a0bc0a]) successfully deleted
W20251027 21:08:01.885452 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet c3f4e692c08244afa2c79e0a075bbcdb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3f4e692c08244afa2c79e0a075bbcdb already in progress: creating tablet
I20251027 21:08:01.885774 10043 ts_tablet_manager.cc:1916] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:01.886457 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet a68bd9243b7541bbbc77014113e0106a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a68bd9243b7541bbbc77014113e0106a already in progress: creating tablet
W20251027 21:08:01.887056 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 7425b6b0184e4d0b871777b25d904ac4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7425b6b0184e4d0b871777b25d904ac4 already in progress: creating tablet
W20251027 21:08:01.887590 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet ef6f69db3d2547c88bbc8df2d52e2fb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef6f69db3d2547c88bbc8df2d52e2fb1 already in progress: creating tablet
I20251027 21:08:01.888381 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet e841c54a94d649c19b639f5021f150cd (table default.Table_5cdd921f5bdf47bab7ec19ac00c6292f [id=d392262a11e24cf0991bc200475b0ed6]) successfully deleted
I20251027 21:08:01.888765 10040 tablet_replica.cc:333] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:01.888929 10040 raft_consensus.cc:2243] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251027 21:08:01.888923 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet edbf5f0db6fa40c7b24f1cd20ee80dd4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edbf5f0db6fa40c7b24f1cd20ee80dd4 already in progress: creating tablet
I20251027 21:08:01.889056 10040 raft_consensus.cc:2272] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.890313 10040 ts_tablet_manager.cc:1916] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:01.890478 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet c3f4e692c08244afa2c79e0a075bbcdb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3f4e692c08244afa2c79e0a075bbcdb already in progress: creating tablet
I20251027 21:08:01.890995 10041 tablet_replica.cc:333] T a8328318a4f14b38a4ce36d183e99dfb P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:01.891177 10041 raft_consensus.cc:2243] T a8328318a4f14b38a4ce36d183e99dfb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.891325 10041 raft_consensus.cc:2272] T a8328318a4f14b38a4ce36d183e99dfb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.893644 10029 ts_tablet_manager.cc:1434] T 343eaeca24dc4816ae032c7e674e9412 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20251027 21:08:01.893782 10041 ts_tablet_manager.cc:1916] T a8328318a4f14b38a4ce36d183e99dfb P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.894565 10029 tablet_bootstrap.cc:492] T f34512059f0c47fc9af7db2645f86b40 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:01.896039 10029 tablet_bootstrap.cc:654] T f34512059f0c47fc9af7db2645f86b40 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.900633 10043 ts_tablet_manager.cc:1929] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.902266 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 431313b4c80940239536aea40721b635
I20251027 21:08:01.903558 10032 tablet_bootstrap.cc:492] T a339c0dee0994788b8644912f74c1b82 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:01.903672 10032 ts_tablet_manager.cc:1403] T a339c0dee0994788b8644912f74c1b82 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.145s user 0.001s sys 0.016s
I20251027 21:08:01.904909 10032 raft_consensus.cc:359] T a339c0dee0994788b8644912f74c1b82 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.905064 10032 raft_consensus.cc:385] T a339c0dee0994788b8644912f74c1b82 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.905114 10032 raft_consensus.cc:740] T a339c0dee0994788b8644912f74c1b82 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.905469 10032 consensus_queue.cc:260] T a339c0dee0994788b8644912f74c1b82 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.905745 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6f780e2be9194c1e9ad52558415f7043"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:01.906102 10032 ts_tablet_manager.cc:1434] T a339c0dee0994788b8644912f74c1b82 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 21:08:01.906235 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 0b4bf2b20e7c416aad123c6f96355cac
I20251027 21:08:01.906831 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 33c5bc33130a424fb63e3f92a2cc49fc
W20251027 21:08:01.907060 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f780e2be9194c1e9ad52558415f7043 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:01.907235 9304 catalog_manager.cc:6719] ReplaceTablet: tablet 33c5bc33130a424fb63e3f92a2cc49fc deleted and replaced by tablet f838fb6e148c4b16b062e831c57cd454
I20251027 21:08:01.907817 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 1ea10027bc7246e98092b894241d9d61
I20251027 21:08:01.908257 9896 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b4bf2b20e7c416aad123c6f96355cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.908485 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 431313b4c80940239536aea40721b635 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.909376 9896 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ea10027bc7246e98092b894241d9d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.909580 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 33c5bc33130a424fb63e3f92a2cc49fc with delete_type TABLET_DATA_DELETED (replaced by tablet f838fb6e148c4b16b062e831c57cd454) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.911942 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet a339c0dee0994788b8644912f74c1b82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.913568 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6b1fd8ed70ee48709a53ab24b5baaab4" } new_table_name: "default.Table_782e39a9c1114d0590c00c48ee3946c6" modify_external_catalogs: true
I20251027 21:08:01.913201 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet 33c5bc33130a424fb63e3f92a2cc49fc with delete_type TABLET_DATA_DELETED (replaced by tablet f838fb6e148c4b16b062e831c57cd454) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.914103 10032 tablet_bootstrap.cc:492] T e9edb62a8e4b4dfea6a31a2202545a6e P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:01.914163 10043 log.cc:1199] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/4e8c83b501d8426fa0cad6a89f9a0b9b
I20251027 21:08:01.913353 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ea10027bc7246e98092b894241d9d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.914449 10043 ts_tablet_manager.cc:1950] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:01.915212 10043 tablet_replica.cc:333] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:01.915393 10043 raft_consensus.cc:2243] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:01.915616 10043 pending_rounds.cc:70] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14: Trying to abort 1 pending ops.
I20251027 21:08:01.915676 10043 pending_rounds.cc:77] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510657678184448 op_type: NO_OP noop_request { }
I20251027 21:08:01.915772 10043 raft_consensus.cc:2889] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:01.915859 10043 raft_consensus.cc:2272] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.916543 10029 tablet_bootstrap.cc:492] T f34512059f0c47fc9af7db2645f86b40 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:01.916667 10029 ts_tablet_manager.cc:1403] T f34512059f0c47fc9af7db2645f86b40 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.001s
I20251027 21:08:01.916674 10043 ts_tablet_manager.cc:1916] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.917493 10718 raft_consensus.cc:493] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:01.917618 10718 raft_consensus.cc:515] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.917791 10029 raft_consensus.cc:359] T f34512059f0c47fc9af7db2645f86b40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.917966 10029 raft_consensus.cc:385] T f34512059f0c47fc9af7db2645f86b40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.918018 10029 raft_consensus.cc:740] T f34512059f0c47fc9af7db2645f86b40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.918231 10718 leader_election.cc:290] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:01.919234 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 4e8c83b501d8426fa0cad6a89f9a0b9b (table default.Table_4f6664b7160e4549a50589454ff10313 [id=67b71acdadc64445bb2492e5d424cb2d]) successfully deleted
I20251027 21:08:01.922232 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b4bf2b20e7c416aad123c6f96355cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.922255 10029 consensus_queue.cc:260] T f34512059f0c47fc9af7db2645f86b40 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.922917 10029 ts_tablet_manager.cc:1434] T f34512059f0c47fc9af7db2645f86b40 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251027 21:08:01.923393 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_da0e93f761894c259f883d648d7f25d8" } modify_external_catalogs: true
I20251027 21:08:01.923548 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_da0e93f761894c259f883d648d7f25d8" } modify_external_catalogs: true
I20251027 21:08:01.924065 10029 tablet_bootstrap.cc:492] T d52d71f4976143e4a514350621ebe309 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:01.924925 10032 tablet_bootstrap.cc:654] T e9edb62a8e4b4dfea6a31a2202545a6e P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.929575 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 431313b4c80940239536aea40721b635 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:01.929633 9710 leader_election.cc:343] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 6f1129eecfb54448965222269d9c32b7
I20251027 21:08:01.930285 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 92a2393cff8044028c86c8cd2bccd930 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.930456 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet 431313b4c80940239536aea40721b635 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.930564 9754 tablet_service.cc:1552] Processing DeleteTablet for tablet 33c5bc33130a424fb63e3f92a2cc49fc with delete_type TABLET_DATA_DELETED (replaced by tablet f838fb6e148c4b16b062e831c57cd454) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.930661 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b4bf2b20e7c416aad123c6f96355cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.930886 10041 ts_tablet_manager.cc:1929] T a8328318a4f14b38a4ce36d183e99dfb P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.930980 10041 log.cc:1199] T a8328318a4f14b38a4ce36d183e99dfb P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/a8328318a4f14b38a4ce36d183e99dfb
I20251027 21:08:01.931268 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ea10027bc7246e98092b894241d9d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.931485 10041 ts_tablet_manager.cc:1950] T a8328318a4f14b38a4ce36d183e99dfb P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
W20251027 21:08:01.931586 9711 leader_election.cc:343] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 6f1129eecfb54448965222269d9c32b7
I20251027 21:08:01.931669 9711 leader_election.cc:304] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:01.932516 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet a8328318a4f14b38a4ce36d183e99dfb (table default.Table_526d1cbb28784221b43a6719685c5e2a [id=287175ad40544f73a586c736fcce910b]) successfully deleted
W20251027 21:08:01.932925 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 85ab6ac53e604ca0bfe051373606fe71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85ab6ac53e604ca0bfe051373606fe71 already in progress: creating tablet
I20251027 21:08:01.933210 10718 raft_consensus.cc:2749] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:01.934688 10800 raft_consensus.cc:493] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:01.934747 10031 tablet_bootstrap.cc:492] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:01.934849 10031 ts_tablet_manager.cc:1403] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.002s
I20251027 21:08:01.934813 10800 raft_consensus.cc:515] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:01.934909 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 0b4bf2b20e7c416aad123c6f96355cac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b4bf2b20e7c416aad123c6f96355cac already in progress: creating tablet
W20251027 21:08:01.935110 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 431313b4c80940239536aea40721b635 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 431313b4c80940239536aea40721b635 already in progress: creating tablet
W20251027 21:08:01.935339 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 1ea10027bc7246e98092b894241d9d61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ea10027bc7246e98092b894241d9d61 already in progress: creating tablet
I20251027 21:08:01.935402 10800 leader_election.cc:290] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:01.935559 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 33c5bc33130a424fb63e3f92a2cc49fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33c5bc33130a424fb63e3f92a2cc49fc already in progress: creating tablet
I20251027 21:08:01.935545 10031 raft_consensus.cc:359] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.935827 10041 tablet_replica.cc:333] T a339c0dee0994788b8644912f74c1b82 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:01.935942 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f1d1bd52556463fa306e51eb253af40" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:01.935987 10041 raft_consensus.cc:2243] T a339c0dee0994788b8644912f74c1b82 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.936136 10041 raft_consensus.cc:2272] T a339c0dee0994788b8644912f74c1b82 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.936300 10031 raft_consensus.cc:385] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251027 21:08:01.936311 9859 leader_election.cc:343] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:01.936374 10031 raft_consensus.cc:740] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.936579 10031 consensus_queue.cc:260] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.937047 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a473b10ab1e94ca885768527b521172b" } modify_external_catalogs: true
I20251027 21:08:01.937186 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a473b10ab1e94ca885768527b521172b" } modify_external_catalogs: true
I20251027 21:08:01.937886 10031 ts_tablet_manager.cc:1434] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:08:01.942565 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f1d1bd52556463fa306e51eb253af40" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:01.943147 10041 ts_tablet_manager.cc:1916] T a339c0dee0994788b8644912f74c1b82 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:01.943173 9860 leader_election.cc:343] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:01.943270 9860 leader_election.cc:304] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:01.943594 10800 raft_consensus.cc:2749] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:01.944114 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet 343eaeca24dc4816ae032c7e674e9412 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.944322 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet f34512059f0c47fc9af7db2645f86b40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:01.946771 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8db666d9b7eb458b843ab5910cd7dff4 consensus state has no leader
I20251027 21:08:01.947494 10029 tablet_bootstrap.cc:654] T d52d71f4976143e4a514350621ebe309 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.948248 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet d8691aa22fe84cf29893ed4e79d156cc
I20251027 21:08:01.949497 9754 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e8c83b501d8426fa0cad6a89f9a0b9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:49 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.949654 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet d8691aa22fe84cf29893ed4e79d156cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:01.949672 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet d8691aa22fe84cf29893ed4e79d156cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.950475 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet d8691aa22fe84cf29893ed4e79d156cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.952203 10031 tablet_bootstrap.cc:492] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:01.956075 10041 ts_tablet_manager.cc:1929] T a339c0dee0994788b8644912f74c1b82 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.956164 10041 log.cc:1199] T a339c0dee0994788b8644912f74c1b82 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/a339c0dee0994788b8644912f74c1b82
I20251027 21:08:01.956431 10041 ts_tablet_manager.cc:1950] T a339c0dee0994788b8644912f74c1b82 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:01.957234 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet a339c0dee0994788b8644912f74c1b82 (table default.Table_4a455030bbe945c391d97696ee50010e [id=fb030c6e79b44af4b3469a20e09a8e47]) successfully deleted
W20251027 21:08:01.957505 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet d8691aa22fe84cf29893ed4e79d156cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8691aa22fe84cf29893ed4e79d156cc already in progress: creating tablet
I20251027 21:08:01.960024 10031 tablet_bootstrap.cc:654] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.960279 10032 tablet_bootstrap.cc:492] T e9edb62a8e4b4dfea6a31a2202545a6e P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:01.960387 10032 ts_tablet_manager.cc:1403] T e9edb62a8e4b4dfea6a31a2202545a6e P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.003s
I20251027 21:08:01.961103 10032 raft_consensus.cc:359] T e9edb62a8e4b4dfea6a31a2202545a6e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.961278 10032 raft_consensus.cc:385] T e9edb62a8e4b4dfea6a31a2202545a6e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.961330 10032 raft_consensus.cc:740] T e9edb62a8e4b4dfea6a31a2202545a6e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.961534 10032 consensus_queue.cc:260] T e9edb62a8e4b4dfea6a31a2202545a6e P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.964720 10032 ts_tablet_manager.cc:1434] T e9edb62a8e4b4dfea6a31a2202545a6e P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251027 21:08:01.964999 10032 tablet_bootstrap.cc:492] T d56d323ef0e74344a479940b93fea11a P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:01.965137 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet e9edb62a8e4b4dfea6a31a2202545a6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:01.965379 10041 tablet_replica.cc:333] T e9edb62a8e4b4dfea6a31a2202545a6e P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:01.965633 10041 raft_consensus.cc:2243] T e9edb62a8e4b4dfea6a31a2202545a6e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.965804 10041 raft_consensus.cc:2272] T e9edb62a8e4b4dfea6a31a2202545a6e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.967882 10032 tablet_bootstrap.cc:654] T d56d323ef0e74344a479940b93fea11a P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.968381 10041 ts_tablet_manager.cc:1916] T e9edb62a8e4b4dfea6a31a2202545a6e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:01.969322 10029 tablet_bootstrap.cc:492] T d52d71f4976143e4a514350621ebe309 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:01.969475 10029 ts_tablet_manager.cc:1403] T d52d71f4976143e4a514350621ebe309 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.002s
I20251027 21:08:01.970391 10029 raft_consensus.cc:359] T d52d71f4976143e4a514350621ebe309 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.970538 10029 raft_consensus.cc:385] T d52d71f4976143e4a514350621ebe309 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.970592 10029 raft_consensus.cc:740] T d52d71f4976143e4a514350621ebe309 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.970798 10029 consensus_queue.cc:260] T d52d71f4976143e4a514350621ebe309 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.973181 10029 ts_tablet_manager.cc:1434] T d52d71f4976143e4a514350621ebe309 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251027 21:08:01.973698 10029 tablet_bootstrap.cc:492] T 533f34d4872444278477abcce89da9ec P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:01.974006 10040 ts_tablet_manager.cc:1929] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.974102 10040 log.cc:1199] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/6bf9c2a78aac42528d67846dc58adc40
I20251027 21:08:01.977440 10041 ts_tablet_manager.cc:1929] T e9edb62a8e4b4dfea6a31a2202545a6e P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.977751 10041 log.cc:1199] T e9edb62a8e4b4dfea6a31a2202545a6e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/e9edb62a8e4b4dfea6a31a2202545a6e
I20251027 21:08:01.978165 10041 ts_tablet_manager.cc:1950] T e9edb62a8e4b4dfea6a31a2202545a6e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:01.980418 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet d52d71f4976143e4a514350621ebe309 with delete_type TABLET_DATA_DELETED (replaced by tablet e419e588b31d48c8bc19d7948081cc81) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:01.981352 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet e9edb62a8e4b4dfea6a31a2202545a6e (table default.Table_2d874641d88e434c8022d7026802972f [id=7b94fb706c8e429b911e1e19265a3883]) successfully deleted
I20251027 21:08:01.980881 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5d4d055be5ca4beb9199ff43126de81c" } new_table_name: "default.Table_9821265dde764aee9571790dd10b4910" modify_external_catalogs: true
I20251027 21:08:01.985654 9308 master_service.cc:954] ReplaceTablet: received request to replace tablet e419e588b31d48c8bc19d7948081cc81 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:01.988297 10043 ts_tablet_manager.cc:1929] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:01.988413 10043 log.cc:1199] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/132a01405a63447ab27b2137b803224f
I20251027 21:08:01.988785 10043 ts_tablet_manager.cc:1950] T 132a01405a63447ab27b2137b803224f P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:01.989600 10043 tablet_replica.cc:333] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:01.989979 10043 raft_consensus.cc:2243] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:01.990120 10043 raft_consensus.cc:2272] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:01.990227 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 132a01405a63447ab27b2137b803224f (table default.Table_673503358b54481c8877034c26349796 [id=c6aed7584a0b49d89bf9a8f8ef0cb429]) successfully deleted
I20251027 21:08:01.991174 10043 ts_tablet_manager.cc:1916] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:01.991642 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9558d53008640739ac6fb45f1352fc0 consensus state has no leader
I20251027 21:08:01.990841 10031 tablet_bootstrap.cc:492] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:01.991974 10031 ts_tablet_manager.cc:1403] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.002s
I20251027 21:08:01.992626 10031 raft_consensus.cc:359] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.992770 10031 raft_consensus.cc:385] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.992813 10031 raft_consensus.cc:740] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.993239 10040 ts_tablet_manager.cc:1950] T 6bf9c2a78aac42528d67846dc58adc40 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:01.993363 10031 consensus_queue.cc:260] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:01.994526 10031 ts_tablet_manager.cc:1434] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:01.995134 10032 tablet_bootstrap.cc:492] T d56d323ef0e74344a479940b93fea11a P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:01.996201 10029 tablet_bootstrap.cc:654] T 533f34d4872444278477abcce89da9ec P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:01.995254 10032 ts_tablet_manager.cc:1403] T d56d323ef0e74344a479940b93fea11a P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.001s
I20251027 21:08:01.997570 10032 raft_consensus.cc:359] T d56d323ef0e74344a479940b93fea11a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:01.997745 10032 raft_consensus.cc:385] T d56d323ef0e74344a479940b93fea11a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:01.997792 10032 raft_consensus.cc:740] T d56d323ef0e74344a479940b93fea11a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:01.999279 10043 ts_tablet_manager.cc:1929] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:01.999372 10043 log.cc:1199] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/0a58cd07e1bb4ffdbf39e7e16a46022a
I20251027 21:08:01.999650 10043 ts_tablet_manager.cc:1950] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:01.999681 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 6bf9c2a78aac42528d67846dc58adc40 (table default.Table_e9fd9bee6a2a40d7ad92d2d31b03afce [id=af6cb904f8cd4a929be7a978f9bc2001]) successfully deleted
I20251027 21:08:02.000150 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_32eeb5fb70df48eb8c27a0a3bc95cee8" } new_table_name: "default.Table_06ef936a484b4bfe88cee764e5aeb190" modify_external_catalogs: true
I20251027 21:08:02.000367 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_fe40346a4d3641159b11c6b9ec566eca" } new_table_name: "default.Table_24fe105409a8463fa475647295017db9" modify_external_catalogs: true
I20251027 21:08:02.000595 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 0a58cd07e1bb4ffdbf39e7e16a46022a (table default.Table_bc5ec745ebe54017b6bc9635581c813d [id=31172964ef544c5f99cade30a42b340d]) successfully deleted
W20251027 21:08:02.001382 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 85ab6ac53e604ca0bfe051373606fe71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85ab6ac53e604ca0bfe051373606fe71 already in progress: creating tablet
I20251027 21:08:02.001681 10031 tablet_bootstrap.cc:492] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:02.001785 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_56aebdec46ea4837a4a42ff7c45aebba" } new_table_name: "default.Table_3fb758abf8b749b8a099e066f6efd218" modify_external_catalogs: true
I20251027 21:08:01.997967 10032 consensus_queue.cc:260] T d56d323ef0e74344a479940b93fea11a P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:02.002008 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 33c5bc33130a424fb63e3f92a2cc49fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33c5bc33130a424fb63e3f92a2cc49fc already in progress: creating tablet
I20251027 21:08:02.003232 10032 ts_tablet_manager.cc:1434] T d56d323ef0e74344a479940b93fea11a P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
W20251027 21:08:02.003942 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 1ea10027bc7246e98092b894241d9d61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ea10027bc7246e98092b894241d9d61 already in progress: creating tablet
I20251027 21:08:02.004194 10032 tablet_bootstrap.cc:492] T a26ded6012174348b7295fc7c0e05e6e P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:08:02.004969 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 0b4bf2b20e7c416aad123c6f96355cac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b4bf2b20e7c416aad123c6f96355cac already in progress: creating tablet
I20251027 21:08:02.005468 10032 tablet_bootstrap.cc:654] T a26ded6012174348b7295fc7c0e05e6e P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:02.005929 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 431313b4c80940239536aea40721b635 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 431313b4c80940239536aea40721b635 already in progress: creating tablet
I20251027 21:08:02.006001 10031 tablet_bootstrap.cc:654] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.006582 10043 tablet_replica.cc:333] T 343eaeca24dc4816ae032c7e674e9412 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:02.007043 10043 raft_consensus.cc:2243] T 343eaeca24dc4816ae032c7e674e9412 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.007118 10040 tablet_replica.cc:333] T 8fc2d7d6601d4cc2849064f4cdddc90d P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:02.007196 10043 raft_consensus.cc:2272] T 343eaeca24dc4816ae032c7e674e9412 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.007279 10040 raft_consensus.cc:2243] T 8fc2d7d6601d4cc2849064f4cdddc90d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.007436 10040 raft_consensus.cc:2272] T 8fc2d7d6601d4cc2849064f4cdddc90d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.007768 10043 ts_tablet_manager.cc:1916] T 343eaeca24dc4816ae032c7e674e9412 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.008127 10040 ts_tablet_manager.cc:1916] T 8fc2d7d6601d4cc2849064f4cdddc90d P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.008467 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6a6205df84c04bf8a9fb02bef162312c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:02.008930 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a6205df84c04bf8a9fb02bef162312c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.011638 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet c1f99ca89f924b58b1416d2945a6ba40 with delete_type TABLET_DATA_DELETED (replaced by tablet d25b493ab8ca401bb1cbe01e355d7bcd) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.013298 10043 ts_tablet_manager.cc:1929] T 343eaeca24dc4816ae032c7e674e9412 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.013401 10043 log.cc:1199] T 343eaeca24dc4816ae032c7e674e9412 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/343eaeca24dc4816ae032c7e674e9412
I20251027 21:08:02.013635 10043 ts_tablet_manager.cc:1950] T 343eaeca24dc4816ae032c7e674e9412 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:02.014683 10029 tablet_bootstrap.cc:492] T 533f34d4872444278477abcce89da9ec P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.014920 10029 ts_tablet_manager.cc:1403] T 533f34d4872444278477abcce89da9ec P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
W20251027 21:08:02.015491 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3c0d0c5263140728f3b8f0b11f46aab consensus state has no leader
I20251027 21:08:02.016294 10029 raft_consensus.cc:359] T 533f34d4872444278477abcce89da9ec P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.016487 10029 raft_consensus.cc:385] T 533f34d4872444278477abcce89da9ec P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.016546 10029 raft_consensus.cc:740] T 533f34d4872444278477abcce89da9ec P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
W20251027 21:08:02.016577 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 31c6aeeac212490ea9b13aa772765277 consensus state has no leader
I20251027 21:08:02.016785 10029 consensus_queue.cc:260] T 533f34d4872444278477abcce89da9ec P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.017361 10029 ts_tablet_manager.cc:1434] T 533f34d4872444278477abcce89da9ec P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251027 21:08:02.018504 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 699c6a8dd8d44d729df463b20cdcfdda consensus state has no leader
I20251027 21:08:02.018548 10029 tablet_bootstrap.cc:492] T 9b04349b15f3494bb0933db56b7b24bc P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:02.020028 10029 tablet_bootstrap.cc:654] T 9b04349b15f3494bb0933db56b7b24bc P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.021493 10032 tablet_bootstrap.cc:492] T a26ded6012174348b7295fc7c0e05e6e P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:02.021612 10032 ts_tablet_manager.cc:1403] T a26ded6012174348b7295fc7c0e05e6e P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20251027 21:08:02.022238 10032 raft_consensus.cc:359] T a26ded6012174348b7295fc7c0e05e6e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.022382 10032 raft_consensus.cc:385] T a26ded6012174348b7295fc7c0e05e6e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.022428 10032 raft_consensus.cc:740] T a26ded6012174348b7295fc7c0e05e6e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.022601 10032 consensus_queue.cc:260] T a26ded6012174348b7295fc7c0e05e6e P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.023164 10032 ts_tablet_manager.cc:1434] T a26ded6012174348b7295fc7c0e05e6e P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251027 21:08:02.023384 10032 tablet_bootstrap.cc:492] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:02.024610 10032 tablet_bootstrap.cc:654] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.025640 10043 tablet_replica.cc:333] T f34512059f0c47fc9af7db2645f86b40 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:02.025808 10043 raft_consensus.cc:2243] T f34512059f0c47fc9af7db2645f86b40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.025938 10043 raft_consensus.cc:2272] T f34512059f0c47fc9af7db2645f86b40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.026924 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 343eaeca24dc4816ae032c7e674e9412 (table default.Table_546a5293975148e1841c8fff9eeac42a [id=b4569b8a023945989439915acb6097e4]) successfully deleted
I20251027 21:08:02.027374 10031 tablet_bootstrap.cc:492] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:02.027500 10043 ts_tablet_manager.cc:1916] T f34512059f0c47fc9af7db2645f86b40 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.027626 10031 ts_tablet_manager.cc:1403] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20251027 21:08:02.028812 10031 raft_consensus.cc:359] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.028959 10031 raft_consensus.cc:385] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.029024 10031 raft_consensus.cc:740] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.029196 10031 consensus_queue.cc:260] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.029850 10031 ts_tablet_manager.cc:1434] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:02.030445 10031 tablet_bootstrap.cc:492] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:02.031718 10029 tablet_bootstrap.cc:492] T 9b04349b15f3494bb0933db56b7b24bc P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.031826 10029 ts_tablet_manager.cc:1403] T 9b04349b15f3494bb0933db56b7b24bc P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20251027 21:08:02.032505 10029 raft_consensus.cc:359] T 9b04349b15f3494bb0933db56b7b24bc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.032656 10029 raft_consensus.cc:385] T 9b04349b15f3494bb0933db56b7b24bc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.032680 10031 tablet_bootstrap.cc:654] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.032708 10029 raft_consensus.cc:740] T 9b04349b15f3494bb0933db56b7b24bc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.032900 10029 consensus_queue.cc:260] T 9b04349b15f3494bb0933db56b7b24bc P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.034616 10029 ts_tablet_manager.cc:1434] T 9b04349b15f3494bb0933db56b7b24bc P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:08:02.034859 10029 tablet_bootstrap.cc:492] T 288b88e13fdf4427aae53f64b87733bf P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:02.036460 10029 tablet_bootstrap.cc:654] T 288b88e13fdf4427aae53f64b87733bf P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.040084 10043 ts_tablet_manager.cc:1929] T f34512059f0c47fc9af7db2645f86b40 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.040170 10043 log.cc:1199] T f34512059f0c47fc9af7db2645f86b40 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/f34512059f0c47fc9af7db2645f86b40
I20251027 21:08:02.040478 10043 ts_tablet_manager.cc:1950] T f34512059f0c47fc9af7db2645f86b40 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:02.040541 10040 ts_tablet_manager.cc:1929] T 8fc2d7d6601d4cc2849064f4cdddc90d P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.040638 10040 log.cc:1199] T 8fc2d7d6601d4cc2849064f4cdddc90d P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/8fc2d7d6601d4cc2849064f4cdddc90d
I20251027 21:08:02.040964 10040 ts_tablet_manager.cc:1950] T 8fc2d7d6601d4cc2849064f4cdddc90d P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:02.041460 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 8fc2d7d6601d4cc2849064f4cdddc90d (table default.Table_162a38d7fe14491d8dae6d2e097d3e3a [id=6f39910d23dd455f87a4ac847917b87f]) successfully deleted
I20251027 21:08:02.041817 10718 raft_consensus.cc:493] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:02.042022 10718 raft_consensus.cc:515] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.042515 10032 tablet_bootstrap.cc:492] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:02.042564 10718 leader_election.cc:290] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:02.042652 10032 ts_tablet_manager.cc:1403] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20251027 21:08:02.043438 10032 raft_consensus.cc:359] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:02.043501 9711 leader_election.cc:343] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: d52d71f4976143e4a514350621ebe309
I20251027 21:08:02.043579 10032 raft_consensus.cc:385] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.043628 10032 raft_consensus.cc:740] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.043838 10032 consensus_queue.cc:260] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.044145 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d52d71f4976143e4a514350621ebe309" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:02.044353 9632 raft_consensus.cc:2468] T d52d71f4976143e4a514350621ebe309 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:08:02.044399 10032 ts_tablet_manager.cc:1434] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:02.044301 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet f34512059f0c47fc9af7db2645f86b40 (table default.Table_660eb77e69884fe5bd5e65b47ea91dde [id=14daa3e199fe4fa3ba2de06dd6be4330]) successfully deleted
I20251027 21:08:02.044626 10032 tablet_bootstrap.cc:492] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:02.044742 10043 tablet_replica.cc:333] T d52d71f4976143e4a514350621ebe309 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
W20251027 21:08:02.044662 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet d8691aa22fe84cf29893ed4e79d156cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8691aa22fe84cf29893ed4e79d156cc already in progress: creating tablet
I20251027 21:08:02.044973 10043 raft_consensus.cc:2243] T d52d71f4976143e4a514350621ebe309 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.045331 10043 raft_consensus.cc:2272] T d52d71f4976143e4a514350621ebe309 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.045945 9710 leader_election.cc:304] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:02.046375 10032 tablet_bootstrap.cc:654] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.046643 10718 raft_consensus.cc:2804] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:02.046741 10718 raft_consensus.cc:493] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:02.046851 10718 raft_consensus.cc:3060] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:02.047040 10043 ts_tablet_manager.cc:1916] T d52d71f4976143e4a514350621ebe309 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.049314 10718 raft_consensus.cc:515] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.050305 10718 leader_election.cc:290] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:02.050765 9711 leader_election.cc:343] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: d52d71f4976143e4a514350621ebe309
I20251027 21:08:02.051775 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d52d71f4976143e4a514350621ebe309" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
W20251027 21:08:02.055673 9710 leader_election.cc:343] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: Tablet not RUNNING: STOPPED
I20251027 21:08:02.059259 10043 ts_tablet_manager.cc:1929] T d52d71f4976143e4a514350621ebe309 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.061337 10043 log.cc:1199] T d52d71f4976143e4a514350621ebe309 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/d52d71f4976143e4a514350621ebe309
I20251027 21:08:02.061767 10043 ts_tablet_manager.cc:1950] T d52d71f4976143e4a514350621ebe309 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:02.062942 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet d52d71f4976143e4a514350621ebe309 (table default.Table_c305a5155cce4754bcbc792f3900b690 [id=82b0b6e6d2e145a7870a13ed5b407b47]) successfully deleted
I20251027 21:08:02.075260 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7d0f803e9bb44ef6b7f03e11e0dae838"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:02.075979 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d0f803e9bb44ef6b7f03e11e0dae838 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.079315 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 533f34d4872444278477abcce89da9ec with delete_type TABLET_DATA_DELETED (replaced by tablet 5054bbc3e27243298b4df6b0a87af8be) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.079614 10043 tablet_replica.cc:333] T 533f34d4872444278477abcce89da9ec P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:02.079839 10043 raft_consensus.cc:2243] T 533f34d4872444278477abcce89da9ec P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.080024 10043 raft_consensus.cc:2272] T 533f34d4872444278477abcce89da9ec P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.081516 10043 ts_tablet_manager.cc:1916] T 533f34d4872444278477abcce89da9ec P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.082293 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b04349b15f3494bb0933db56b7b24bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.087741 10043 ts_tablet_manager.cc:1929] T 533f34d4872444278477abcce89da9ec P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.087872 10043 log.cc:1199] T 533f34d4872444278477abcce89da9ec P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/533f34d4872444278477abcce89da9ec
I20251027 21:08:02.088150 10043 ts_tablet_manager.cc:1950] T 533f34d4872444278477abcce89da9ec P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:02.089275 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 533f34d4872444278477abcce89da9ec (table default.Table_61bbf6b5c1e740128b80465ef3b9a50e [id=439f48256cbd43f785f57857e83ae25f]) successfully deleted
I20251027 21:08:02.089579 10043 tablet_replica.cc:333] T 9b04349b15f3494bb0933db56b7b24bc P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:02.089623 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet 4c023428437f45b4bae1374bd3dd36a7 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:02.089762 10043 raft_consensus.cc:2243] T 9b04349b15f3494bb0933db56b7b24bc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.089939 10043 raft_consensus.cc:2272] T 9b04349b15f3494bb0933db56b7b24bc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.091050 10043 ts_tablet_manager.cc:1916] T 9b04349b15f3494bb0933db56b7b24bc P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.093495 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7aab2b63c44043aa9aea339ec0b5763e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:02.094386 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7aab2b63c44043aa9aea339ec0b5763e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.095659 9710 leader_election.cc:304] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:02.097082 10032 tablet_bootstrap.cc:492] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:02.097203 10032 ts_tablet_manager.cc:1403] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.000s
I20251027 21:08:02.097864 10032 raft_consensus.cc:359] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.098024 10032 raft_consensus.cc:385] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.098067 10032 raft_consensus.cc:740] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.098253 10032 consensus_queue.cc:260] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.098675 10032 ts_tablet_manager.cc:1434] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251027 21:08:02.098905 10032 tablet_bootstrap.cc:492] T da26c83976df40328cf9b11734fcfa44 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:02.100389 10032 tablet_bootstrap.cc:654] T da26c83976df40328cf9b11734fcfa44 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.101722 10029 tablet_bootstrap.cc:492] T 288b88e13fdf4427aae53f64b87733bf P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.101864 10029 ts_tablet_manager.cc:1403] T 288b88e13fdf4427aae53f64b87733bf P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.067s user 0.014s sys 0.014s
I20251027 21:08:02.102581 10029 raft_consensus.cc:359] T 288b88e13fdf4427aae53f64b87733bf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.102746 10029 raft_consensus.cc:385] T 288b88e13fdf4427aae53f64b87733bf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.102803 10029 raft_consensus.cc:740] T 288b88e13fdf4427aae53f64b87733bf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.103132 10029 consensus_queue.cc:260] T 288b88e13fdf4427aae53f64b87733bf P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.105329 10029 ts_tablet_manager.cc:1434] T 288b88e13fdf4427aae53f64b87733bf P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:08:02.105621 10029 tablet_bootstrap.cc:492] T 94f2dbf47a5c4458bee54b26f272156e P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:02.106770 10718 raft_consensus.cc:493] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:02.106901 10718 raft_consensus.cc:515] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.107280 10029 tablet_bootstrap.cc:654] T 94f2dbf47a5c4458bee54b26f272156e P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.107466 10718 leader_election.cc:290] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:02.107599 10718 raft_consensus.cc:2749] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251027 21:08:02.108088 9923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9339c6b1154400eace53d765528fdf9" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:02.108285 9923 raft_consensus.cc:2393] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77b0513f17c0472088901a1ce8b08218 in current term 1: Already voted for candidate 51902b6b61ff4d5584ff21896e847e14 in this term.
I20251027 21:08:02.109069 10043 ts_tablet_manager.cc:1929] T 9b04349b15f3494bb0933db56b7b24bc P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.109196 10043 log.cc:1199] T 9b04349b15f3494bb0933db56b7b24bc P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/9b04349b15f3494bb0933db56b7b24bc
I20251027 21:08:02.109568 10043 ts_tablet_manager.cc:1950] T 9b04349b15f3494bb0933db56b7b24bc P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:02.110776 10032 tablet_bootstrap.cc:492] T da26c83976df40328cf9b11734fcfa44 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:02.110898 10032 ts_tablet_manager.cc:1403] T da26c83976df40328cf9b11734fcfa44 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20251027 21:08:02.112843 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_dbd96c7982f647c6990d16cbec0a6a04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:02.113380 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbd96c7982f647c6990d16cbec0a6a04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.113776 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c51243cadc5c403d84acc583c03edaba" } modify_external_catalogs: true
I20251027 21:08:02.113947 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c51243cadc5c403d84acc583c03edaba" } modify_external_catalogs: true
I20251027 21:08:02.114595 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 288b88e13fdf4427aae53f64b87733bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.114852 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f1103102a7544791874c67fa7cadbbbe" } modify_external_catalogs: true
I20251027 21:08:02.115033 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f1103102a7544791874c67fa7cadbbbe" } modify_external_catalogs: true
I20251027 21:08:02.114985 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_54ec9385aca7411ba2d3d140802ee661"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:02.115548 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54ec9385aca7411ba2d3d140802ee661 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.116261 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 9b04349b15f3494bb0933db56b7b24bc (table default.Table_2ff5a5b701c540b8a9eb8b87aace0e77 [id=69f10c0f3f3a4227ab4c35e939085634]) successfully deleted
I20251027 21:08:02.116504 10032 raft_consensus.cc:359] T da26c83976df40328cf9b11734fcfa44 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.116860 10043 tablet_replica.cc:333] T 288b88e13fdf4427aae53f64b87733bf P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:02.117061 10043 raft_consensus.cc:2243] T 288b88e13fdf4427aae53f64b87733bf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.117216 10043 raft_consensus.cc:2272] T 288b88e13fdf4427aae53f64b87733bf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.117502 10032 raft_consensus.cc:385] T da26c83976df40328cf9b11734fcfa44 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.117578 10032 raft_consensus.cc:740] T da26c83976df40328cf9b11734fcfa44 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.117806 10032 consensus_queue.cc:260] T da26c83976df40328cf9b11734fcfa44 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.119324 10043 ts_tablet_manager.cc:1916] T 288b88e13fdf4427aae53f64b87733bf P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.122594 10032 ts_tablet_manager.cc:1434] T da26c83976df40328cf9b11734fcfa44 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251027 21:08:02.122859 10032 tablet_bootstrap.cc:492] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:02.123911 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e186a89da49149d3921fe0646ac3f5c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:02.124380 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e186a89da49149d3921fe0646ac3f5c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.126053 10032 tablet_bootstrap.cc:654] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.126868 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4469fd75838845e9b60fd07ddc845f5c" } modify_external_catalogs: true
I20251027 21:08:02.127199 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4469fd75838845e9b60fd07ddc845f5c" } modify_external_catalogs: true
I20251027 21:08:02.127648 10040 tablet_replica.cc:333] T dc589852710342f7bd017193e4075303 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:02.127913 10040 raft_consensus.cc:2243] T dc589852710342f7bd017193e4075303 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.128091 10040 raft_consensus.cc:2272] T dc589852710342f7bd017193e4075303 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.132858 10043 ts_tablet_manager.cc:1929] T 288b88e13fdf4427aae53f64b87733bf P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.132995 10043 log.cc:1199] T 288b88e13fdf4427aae53f64b87733bf P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/288b88e13fdf4427aae53f64b87733bf
I20251027 21:08:02.133308 10043 ts_tablet_manager.cc:1950] T 288b88e13fdf4427aae53f64b87733bf P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:02.134197 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 288b88e13fdf4427aae53f64b87733bf (table default.Table_a061b11818fc4a2db61e51a5972d69db [id=92ff9768853d403fb15db41e79fe77bc]) successfully deleted
I20251027 21:08:02.135193 10029 tablet_bootstrap.cc:492] T 94f2dbf47a5c4458bee54b26f272156e P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.135349 10029 ts_tablet_manager.cc:1403] T 94f2dbf47a5c4458bee54b26f272156e P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.000s
I20251027 21:08:02.135525 10040 ts_tablet_manager.cc:1916] T dc589852710342f7bd017193e4075303 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.136037 10029 raft_consensus.cc:359] T 94f2dbf47a5c4458bee54b26f272156e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.136198 10029 raft_consensus.cc:385] T 94f2dbf47a5c4458bee54b26f272156e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.136288 10029 raft_consensus.cc:740] T 94f2dbf47a5c4458bee54b26f272156e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.136499 10029 consensus_queue.cc:260] T 94f2dbf47a5c4458bee54b26f272156e P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.137137 10029 ts_tablet_manager.cc:1434] T 94f2dbf47a5c4458bee54b26f272156e P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:02.137462 10029 tablet_bootstrap.cc:492] T abc7458fcfb94985acf76d77c0292b08 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
W20251027 21:08:02.139209 9710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.65:44531, user_credentials={real_user=slave}} blocked reactor thread for 83548.6us
I20251027 21:08:02.147024 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_92fa3ef69db74af5964d6b6f170b03c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:02.147573 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92fa3ef69db74af5964d6b6f170b03c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.150625 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4e44eb5928a2489baf1bf9d68ea91d18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:02.151116 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e44eb5928a2489baf1bf9d68ea91d18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.152292 10029 tablet_bootstrap.cc:654] T abc7458fcfb94985acf76d77c0292b08 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.152498 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 288b88e13fdf4427aae53f64b87733bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.152747 10031 tablet_bootstrap.cc:492] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:02.152899 10031 ts_tablet_manager.cc:1403] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.123s user 0.002s sys 0.002s
I20251027 21:08:02.153013 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9339c6b1154400eace53d765528fdf9" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:02.153519 9298 catalog_manager.cc:4952] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 288b88e13fdf4427aae53f64b87733bf because the tablet was not found. No further retry: Not found: Tablet not found: 288b88e13fdf4427aae53f64b87733bf
I20251027 21:08:02.153563 10031 raft_consensus.cc:359] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.153712 10031 raft_consensus.cc:385] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.153772 10031 raft_consensus.cc:740] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.153941 10031 consensus_queue.cc:260] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.154317 9710 leader_election.cc:304] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:02.154605 10718 raft_consensus.cc:3060] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:02.155678 10639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 21:08:02.156088 10639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20251027 21:08:02.157686 10718 raft_consensus.cc:2749] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:02.158032 10639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251027 21:08:02.158699 10031 ts_tablet_manager.cc:1434] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251027 21:08:02.159056 10031 tablet_bootstrap.cc:492] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:02.160655 10031 tablet_bootstrap.cc:654] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.162338 10032 tablet_bootstrap.cc:492] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:02.162469 10032 ts_tablet_manager.cc:1403] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.040s user 0.014s sys 0.006s
I20251027 21:08:02.163158 10032 raft_consensus.cc:359] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.163307 10032 raft_consensus.cc:385] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.163353 10032 raft_consensus.cc:740] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.163517 10032 consensus_queue.cc:260] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.164675 10800 raft_consensus.cc:493] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:02.164808 10800 raft_consensus.cc:515] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.165153 10040 ts_tablet_manager.cc:1929] T dc589852710342f7bd017193e4075303 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.165261 10040 log.cc:1199] T dc589852710342f7bd017193e4075303 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/dc589852710342f7bd017193e4075303
I20251027 21:08:02.165513 10040 ts_tablet_manager.cc:1950] T dc589852710342f7bd017193e4075303 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:02.165484 10800 leader_election.cc:290] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:02.166067 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9efffa9766e46aca8eb286fc2ad48d2" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:02.166397 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9efffa9766e46aca8eb286fc2ad48d2" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:02.166638 9859 leader_election.cc:343] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:02.166916 9860 leader_election.cc:343] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:02.167023 9860 leader_election.cc:304] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:02.167429 10800 raft_consensus.cc:2749] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:02.167958 10829 raft_consensus.cc:493] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:02.168074 10829 raft_consensus.cc:515] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.168684 10829 leader_election.cc:290] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:02.169353 9711 leader_election.cc:343] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 288b88e13fdf4427aae53f64b87733bf
W20251027 21:08:02.170392 9710 leader_election.cc:343] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 288b88e13fdf4427aae53f64b87733bf
I20251027 21:08:02.170559 9710 leader_election.cc:304] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:02.171146 10829 raft_consensus.cc:2749] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:02.172318 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 94f2dbf47a5c4458bee54b26f272156e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.172565 10043 tablet_replica.cc:333] T 94f2dbf47a5c4458bee54b26f272156e P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:02.172744 10043 raft_consensus.cc:2243] T 94f2dbf47a5c4458bee54b26f272156e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.172928 10043 raft_consensus.cc:2272] T 94f2dbf47a5c4458bee54b26f272156e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.173650 10043 ts_tablet_manager.cc:1916] T 94f2dbf47a5c4458bee54b26f272156e P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.175798 10029 tablet_bootstrap.cc:492] T abc7458fcfb94985acf76d77c0292b08 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.175966 10029 ts_tablet_manager.cc:1403] T abc7458fcfb94985acf76d77c0292b08 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.005s
I20251027 21:08:02.177248 10029 raft_consensus.cc:359] T abc7458fcfb94985acf76d77c0292b08 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.177459 10029 raft_consensus.cc:385] T abc7458fcfb94985acf76d77c0292b08 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.177546 10029 raft_consensus.cc:740] T abc7458fcfb94985acf76d77c0292b08 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.177879 10029 consensus_queue.cc:260] T abc7458fcfb94985acf76d77c0292b08 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.178438 10029 ts_tablet_manager.cc:1434] T abc7458fcfb94985acf76d77c0292b08 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20251027 21:08:02.178720 10029 tablet_bootstrap.cc:492] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:02.180377 10029 tablet_bootstrap.cc:654] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.180406 10043 ts_tablet_manager.cc:1929] T 94f2dbf47a5c4458bee54b26f272156e P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.180523 10043 log.cc:1199] T 94f2dbf47a5c4458bee54b26f272156e P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/94f2dbf47a5c4458bee54b26f272156e
I20251027 21:08:02.180920 10043 ts_tablet_manager.cc:1950] T 94f2dbf47a5c4458bee54b26f272156e P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:02.181893 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 94f2dbf47a5c4458bee54b26f272156e (table default.Table_7954e74d079b416cac73f9a77979c8b3 [id=0103a75b0a714d3c9a533d94b6bda41f]) successfully deleted
I20251027 21:08:02.182050 10031 tablet_bootstrap.cc:492] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:02.182219 10031 ts_tablet_manager.cc:1403] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20251027 21:08:02.182888 10031 raft_consensus.cc:359] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.183041 10031 raft_consensus.cc:385] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.183095 10031 raft_consensus.cc:740] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.183291 10031 consensus_queue.cc:260] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.183712 10031 ts_tablet_manager.cc:1434] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 21:08:02.183971 10031 tablet_bootstrap.cc:492] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:02.185187 10031 tablet_bootstrap.cc:654] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.185405 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ec0b0d1ae0774aa598e7311546fcedd4" } new_table_name: "default.Table_1a6d4a92fb514fdf8f76dcf2757f2b70" modify_external_catalogs: true
I20251027 21:08:02.194648 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c9eb751ca4b7420a9d7557aa95e30ed6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:02.195258 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9eb751ca4b7420a9d7557aa95e30ed6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.196164 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_279dfb819b974d2085826df64b14d411"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:02.196630 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_279dfb819b974d2085826df64b14d411 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.197046 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet abc7458fcfb94985acf76d77c0292b08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.197314 10043 tablet_replica.cc:333] T abc7458fcfb94985acf76d77c0292b08 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:02.197526 10043 raft_consensus.cc:2243] T abc7458fcfb94985acf76d77c0292b08 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.197683 10043 raft_consensus.cc:2272] T abc7458fcfb94985acf76d77c0292b08 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.198645 10032 ts_tablet_manager.cc:1434] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.036s user 0.001s sys 0.000s
I20251027 21:08:02.199011 10032 tablet_bootstrap.cc:492] T 2a4aa1fe05114c1287ca580c1f411d3f P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:02.200647 10043 ts_tablet_manager.cc:1916] T abc7458fcfb94985acf76d77c0292b08 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.209332 10032 tablet_bootstrap.cc:654] T 2a4aa1fe05114c1287ca580c1f411d3f P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.210469 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet dc589852710342f7bd017193e4075303 (table default.Table_d2ff301b3e2b4602a2483859973a02d6 [id=ccc88d98bcd74bde873cca8c9ce3e134]) successfully deleted
I20251027 21:08:02.210518 9311 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet d56d323ef0e74344a479940b93fea11a (table default.Table_cf11a8c344c544c5ba54672fee38e4aa [id=71e02b21c9654e45a41d3e450ac3fbe1]). Expected version 2 got 0
I20251027 21:08:02.210791 10040 tablet_replica.cc:333] T ed5619d295a246379b625f35692203e4 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:02.210992 10040 raft_consensus.cc:2243] T ed5619d295a246379b625f35692203e4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251027 21:08:02.211066 10639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 21:08:02.211162 10639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20251027 21:08:02.211136 10040 raft_consensus.cc:2272] T ed5619d295a246379b625f35692203e4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:02.211251 10639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 21:08:02.211320 10639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 21:08:02.211629 10040 ts_tablet_manager.cc:1916] T ed5619d295a246379b625f35692203e4 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.213449 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e53258cb321b4ae099133b8176b9c0a1" } modify_external_catalogs: true
I20251027 21:08:02.214293 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 288b88e13fdf4427aae53f64b87733bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.214473 9754 tablet_service.cc:1552] Processing DeleteTablet for tablet e0968c2f25b64f6685f38a999ee9d60e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.214603 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a58cd07e1bb4ffdbf39e7e16a46022a with delete_type TABLET_DATA_DELETED (replaced by tablet 4d88aacbf47144cb837b99d68f6331e2) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.216279 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e53258cb321b4ae099133b8176b9c0a1" } modify_external_catalogs: true
I20251027 21:08:02.219410 10029 tablet_bootstrap.cc:492] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.219826 10029 ts_tablet_manager.cc:1403] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.002s
I20251027 21:08:02.220168 9609 tablet_service.cc:1505] Processing CreateTablet for tablet f7fbffe7cd3c45d1b8fecec85f2bbf92 (DEFAULT_TABLE table=default.Table_342bf4fd6c7f4ffbb80cc0f87baf3768 [id=f4fba6101e0b42f5b5815f6f6b786501]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.220507 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7fbffe7cd3c45d1b8fecec85f2bbf92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.220582 10029 raft_consensus.cc:359] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.220731 10029 raft_consensus.cc:385] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.220825 10029 raft_consensus.cc:740] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.220969 10032 tablet_bootstrap.cc:492] T 2a4aa1fe05114c1287ca580c1f411d3f P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:02.221086 10032 ts_tablet_manager.cc:1403] T 2a4aa1fe05114c1287ca580c1f411d3f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.001s
I20251027 21:08:02.221071 10029 consensus_queue.cc:260] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.221652 10029 ts_tablet_manager.cc:1434] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 21:08:02.221916 10029 tablet_bootstrap.cc:492] T 92a2393cff8044028c86c8cd2bccd930 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:02.222034 10032 raft_consensus.cc:359] T 2a4aa1fe05114c1287ca580c1f411d3f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.222214 10032 raft_consensus.cc:385] T 2a4aa1fe05114c1287ca580c1f411d3f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.222276 10032 raft_consensus.cc:740] T 2a4aa1fe05114c1287ca580c1f411d3f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.222714 10032 consensus_queue.cc:260] T 2a4aa1fe05114c1287ca580c1f411d3f P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.223187 10032 ts_tablet_manager.cc:1434] T 2a4aa1fe05114c1287ca580c1f411d3f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:02.223464 10029 tablet_bootstrap.cc:654] T 92a2393cff8044028c86c8cd2bccd930 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.223440 10032 tablet_bootstrap.cc:492] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:02.225042 10031 tablet_bootstrap.cc:492] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:02.225169 10031 ts_tablet_manager.cc:1403] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.000s
I20251027 21:08:02.225240 10032 tablet_bootstrap.cc:654] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.225785 10031 raft_consensus.cc:359] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.225921 10031 raft_consensus.cc:385] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.225961 10031 raft_consensus.cc:740] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.226178 10031 consensus_queue.cc:260] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.226613 10031 ts_tablet_manager.cc:1434] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251027 21:08:02.227594 10031 tablet_bootstrap.cc:492] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:02.229348 10800 raft_consensus.cc:493] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:02.229481 10800 raft_consensus.cc:515] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.229681 10031 tablet_bootstrap.cc:654] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.230178 10800 leader_election.cc:290] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:02.230659 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b30052ff19744a9bf4bfaaa1bc77b36" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:02.231204 9859 leader_election.cc:343] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:02.231856 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b30052ff19744a9bf4bfaaa1bc77b36" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:02.233420 9860 leader_election.cc:343] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:02.233543 9860 leader_election.cc:304] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:02.233937 10800 raft_consensus.cc:2749] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:02.242129 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e740ddf3b394d3a823bc95bad48fa99 consensus state has no leader
I20251027 21:08:02.246091 9305 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet b8856c5dc8f144358445165324bfb477 (table default.Table_203191e192be4e3a9a7477b4313ef4f4 [id=5da6bd4463e24c03accd9985f72b6ebb]). Expected version 3 got 0
I20251027 21:08:02.246929 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_991d9a1334ba4a7eb2a5d5ccee552553" } new_table_name: "default.Table_f994ff2c05ed4f129533763dfa6ddc34" modify_external_catalogs: true
I20251027 21:08:02.247562 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a05ec8bfd95c4f82bd16fbd66e110afc" } new_table_name: "default.Table_c31d7158e88c44b18707ae0ad36fe482" modify_external_catalogs: true
I20251027 21:08:02.248773 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c9c86253ce914916bee6584675a82067
I20251027 21:08:02.249078 9890 tablet_service.cc:1505] Processing CreateTablet for tablet f7fbffe7cd3c45d1b8fecec85f2bbf92 (DEFAULT_TABLE table=default.Table_342bf4fd6c7f4ffbb80cc0f87baf3768 [id=f4fba6101e0b42f5b5815f6f6b786501]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.249435 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7fbffe7cd3c45d1b8fecec85f2bbf92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.249461 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_45436efc9fd041c0a813930bd895b3dd" } new_table_name: "default.Table_bde50fa24ef54301a7b55d76cf5a9bda" modify_external_catalogs: true
I20251027 21:08:02.250183 10040 ts_tablet_manager.cc:1929] T ed5619d295a246379b625f35692203e4 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.250281 10040 log.cc:1199] T ed5619d295a246379b625f35692203e4 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/ed5619d295a246379b625f35692203e4
I20251027 21:08:02.250545 10040 ts_tablet_manager.cc:1950] T ed5619d295a246379b625f35692203e4 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:02.255429 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet ed5619d295a246379b625f35692203e4 (table default.Table_32f38256207d4fa79ed533151449afa7 [id=e3c04aba1dbf46928a0c36444dd69d02]) successfully deleted
I20251027 21:08:02.255985 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet f34512059f0c47fc9af7db2645f86b40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.256043 9745 tablet_service.cc:1505] Processing CreateTablet for tablet d9e8d0ad43154e4daaa513c50bbdef9b (DEFAULT_TABLE table=default.Table_be7a9a27122348488bb59fcd53109c5d [id=5320b33899a34920b367dbb12aa75629]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.256356 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9e8d0ad43154e4daaa513c50bbdef9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.256564 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 1391ad8a22944ab7a8f45aceec19008b (DEFAULT_TABLE table=default.Table_c1eb65a428444379825e1d29557bbc23 [id=a905a7b376174f9c8d76bd42690aad77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.256844 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1391ad8a22944ab7a8f45aceec19008b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.257560 9898 tablet_service.cc:1505] Processing CreateTablet for tablet c0857f89bc74495081d35ee55e785835 (DEFAULT_TABLE table=default.Table_2c4482a14aa14f498e7c8ab78d9fe9ee [id=5ee4cd6429584ac68a9c3f3ce0f61faf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.257900 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 1391ad8a22944ab7a8f45aceec19008b (DEFAULT_TABLE table=default.Table_c1eb65a428444379825e1d29557bbc23 [id=a905a7b376174f9c8d76bd42690aad77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.256294 9754 tablet_service.cc:1505] Processing CreateTablet for tablet f7fbffe7cd3c45d1b8fecec85f2bbf92 (DEFAULT_TABLE table=default.Table_342bf4fd6c7f4ffbb80cc0f87baf3768 [id=f4fba6101e0b42f5b5815f6f6b786501]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.258112 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 38e6a73ff9314e47a16a6c5164650828 (DEFAULT_TABLE table=default.Table_80cde2bc1ed84de29cec9255f33b03d3 [id=919b2c0c6fef4b1a950dbe8c52e11b0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.258335 9895 tablet_service.cc:1505] Processing CreateTablet for tablet d9e8d0ad43154e4daaa513c50bbdef9b (DEFAULT_TABLE table=default.Table_be7a9a27122348488bb59fcd53109c5d [id=5320b33899a34920b367dbb12aa75629]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.258402 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7fbffe7cd3c45d1b8fecec85f2bbf92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.258556 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 138a5d18566240858b5c92c3c03b8087 (DEFAULT_TABLE table=default.Table_7cf03d793a0548ef80ca9ce245e0fc1f [id=d7fb62804ebe44bcacccc1a18d936653]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.258759 9888 tablet_service.cc:1505] Processing CreateTablet for tablet ca12dafba2ec4243812969c1b3e3d6c0 (DEFAULT_TABLE table=default.Table_9581624c205746a19f66b0619cfbd6cf [id=22ee23b9b08e413e99a26a4a621a7434]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.257915 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0857f89bc74495081d35ee55e785835. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.259641 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9e8d0ad43154e4daaa513c50bbdef9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.262252 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca12dafba2ec4243812969c1b3e3d6c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.262248 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet c9c86253ce914916bee6584675a82067 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.262405 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 138a5d18566240858b5c92c3c03b8087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.262542 9736 tablet_service.cc:1505] Processing CreateTablet for tablet c0857f89bc74495081d35ee55e785835 (DEFAULT_TABLE table=default.Table_2c4482a14aa14f498e7c8ab78d9fe9ee [id=5ee4cd6429584ac68a9c3f3ce0f61faf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.262849 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0857f89bc74495081d35ee55e785835. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.263120 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1391ad8a22944ab7a8f45aceec19008b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:02.266348 9299 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 7e740ddf3b394d3a823bc95bad48fa99 (table default.Table_1a6d4a92fb514fdf8f76dcf2757f2b70 [id=0216c82db6554eb88e26cce5db5598c7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7e740ddf3b394d3a823bc95bad48fa99 consensus state has no leader
I20251027 21:08:02.266343 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 38e6a73ff9314e47a16a6c5164650828 (DEFAULT_TABLE table=default.Table_80cde2bc1ed84de29cec9255f33b03d3 [id=919b2c0c6fef4b1a950dbe8c52e11b0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.266623 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38e6a73ff9314e47a16a6c5164650828. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.266724 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38e6a73ff9314e47a16a6c5164650828. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.269975 9748 tablet_service.cc:1505] Processing CreateTablet for tablet ca12dafba2ec4243812969c1b3e3d6c0 (DEFAULT_TABLE table=default.Table_9581624c205746a19f66b0619cfbd6cf [id=22ee23b9b08e413e99a26a4a621a7434]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.270279 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca12dafba2ec4243812969c1b3e3d6c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.271629 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 4c023428437f45b4bae1374bd3dd36a7
I20251027 21:08:02.272092 9307 catalog_manager.cc:6719] ReplaceTablet: tablet 4c023428437f45b4bae1374bd3dd36a7 deleted and replaced by tablet 511fdda39953415aaa65d88cb75ad382
I20251027 21:08:02.274575 9739 tablet_service.cc:1505] Processing CreateTablet for tablet 138a5d18566240858b5c92c3c03b8087 (DEFAULT_TABLE table=default.Table_7cf03d793a0548ef80ca9ce245e0fc1f [id=d7fb62804ebe44bcacccc1a18d936653]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.274870 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 138a5d18566240858b5c92c3c03b8087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.276327 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c023428437f45b4bae1374bd3dd36a7 with delete_type TABLET_DATA_DELETED (replaced by tablet 511fdda39953415aaa65d88cb75ad382) from {username='slave'} at 127.0.0.1:34644
W20251027 21:08:02.278517 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 4c023428437f45b4bae1374bd3dd36a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c023428437f45b4bae1374bd3dd36a7 already in progress: creating tablet
I20251027 21:08:02.282253 9745 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c023428437f45b4bae1374bd3dd36a7 with delete_type TABLET_DATA_DELETED (replaced by tablet 511fdda39953415aaa65d88cb75ad382) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.282809 9309 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet d56d323ef0e74344a479940b93fea11a (table default.Table_cf11a8c344c544c5ba54672fee38e4aa [id=71e02b21c9654e45a41d3e450ac3fbe1]). Expected version 2 got 0
I20251027 21:08:02.284361 10639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.14.84:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.14.126:34057,127.9.14.125:46313,127.9.14.124:45071
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.14.124:45071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.14.124
--webserver_port=41907
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 27 Oct 2025 20:43:25 UTC on 24a791456cd2
build id 8748
ASAN enabled
I20251027 21:08:02.284667 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 4da0bb391aba49cbaeb72983553362b2
I20251027 21:08:02.285337 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c22259490f91403e8b54952e22c9924b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:02.285846 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c22259490f91403e8b54952e22c9924b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.286171 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet c9c86253ce914916bee6584675a82067 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:02.287004 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 0bee9d4601c94b138d68f7663769ccbe
I20251027 21:08:02.287693 10639 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 21:08:02.288266 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5a2a0af6dbb4438195f67883f19838ce" } new_table_name: "default.Table_937452027956409d907e8d44579e9d66" modify_external_catalogs: true
I20251027 21:08:02.288569 10031 tablet_bootstrap.cc:492] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:02.288690 10031 ts_tablet_manager.cc:1403] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.061s user 0.001s sys 0.001s
I20251027 21:08:02.289389 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 217fbfb1f58f420f8f258b4178901370
I20251027 21:08:02.289376 10031 raft_consensus.cc:359] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.289530 10031 raft_consensus.cc:385] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.289587 10031 raft_consensus.cc:740] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.289770 10031 consensus_queue.cc:260] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.290685 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0668ac874959449eb530dd484f0c1d73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:02.291158 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0668ac874959449eb530dd484f0c1d73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.291962 10639 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251027 21:08:02.293869 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet 0f90e09e74b6495db1aa2df85de21181 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:02.298070 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 1391ad8a22944ab7a8f45aceec19008b (DEFAULT_TABLE table=default.Table_c1eb65a428444379825e1d29557bbc23 [id=a905a7b376174f9c8d76bd42690aad77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.298525 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1391ad8a22944ab7a8f45aceec19008b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.298871 9602 tablet_service.cc:1505] Processing CreateTablet for tablet ca12dafba2ec4243812969c1b3e3d6c0 (DEFAULT_TABLE table=default.Table_9581624c205746a19f66b0619cfbd6cf [id=22ee23b9b08e413e99a26a4a621a7434]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.299180 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 2e65fecdeb104706bc6ffe7d7a038a7e (DEFAULT_TABLE table=default.Table_2096e2f92ca2449485e81a1a68807ef2 [id=a733fa5973a74f3d961bdda216316529]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.299995 9600 tablet_service.cc:1505] Processing CreateTablet for tablet c0857f89bc74495081d35ee55e785835 (DEFAULT_TABLE table=default.Table_2c4482a14aa14f498e7c8ab78d9fe9ee [id=5ee4cd6429584ac68a9c3f3ce0f61faf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.298838 9611 tablet_service.cc:1505] Processing CreateTablet for tablet d9e8d0ad43154e4daaa513c50bbdef9b (DEFAULT_TABLE table=default.Table_be7a9a27122348488bb59fcd53109c5d [id=5320b33899a34920b367dbb12aa75629]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:02.300329 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet c9c86253ce914916bee6584675a82067 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9c86253ce914916bee6584675a82067 already in progress: creating tablet
I20251027 21:08:02.300449 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 38e6a73ff9314e47a16a6c5164650828 (DEFAULT_TABLE table=default.Table_80cde2bc1ed84de29cec9255f33b03d3 [id=919b2c0c6fef4b1a950dbe8c52e11b0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.300812 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 138a5d18566240858b5c92c3c03b8087 (DEFAULT_TABLE table=default.Table_7cf03d793a0548ef80ca9ce245e0fc1f [id=d7fb62804ebe44bcacccc1a18d936653]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.301541 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet 4da0bb391aba49cbaeb72983553362b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:02.301762 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b30052ff19744a9bf4bfaaa1bc77b36 with delete_type TABLET_DATA_DELETED (replaced by tablet 283b72f626644ac2b9b61186387b3839) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:02.301792 9896 tablet_service.cc:1552] Processing DeleteTablet for tablet c9efffa9766e46aca8eb286fc2ad48d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:02.301914 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 0bee9d4601c94b138d68f7663769ccbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:02.301944 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet da26c83976df40328cf9b11734fcfa44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:02.302052 9888 tablet_service.cc:1552] Processing DeleteTablet for tablet a26ded6012174348b7295fc7c0e05e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:02.302198 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 217fbfb1f58f420f8f258b4178901370 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:02.302335 10041 tablet_replica.cc:333] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:02.302548 10041 raft_consensus.cc:2243] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.302536 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 217fbfb1f58f420f8f258b4178901370 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.302716 10041 raft_consensus.cc:2272] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.303308 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 0bee9d4601c94b138d68f7663769ccbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.303627 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet abc7458fcfb94985acf76d77c0292b08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.304031 10041 ts_tablet_manager.cc:1916] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.309583 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 4da0bb391aba49cbaeb72983553362b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.309829 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca12dafba2ec4243812969c1b3e3d6c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.309854 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet c9c86253ce914916bee6584675a82067 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.310181 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c023428437f45b4bae1374bd3dd36a7 with delete_type TABLET_DATA_DELETED (replaced by tablet 511fdda39953415aaa65d88cb75ad382) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.310333 9608 tablet_service.cc:1552] Processing DeleteTablet for tablet 0bee9d4601c94b138d68f7663769ccbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:02.310602 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 4da0bb391aba49cbaeb72983553362b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4da0bb391aba49cbaeb72983553362b2 already in progress: creating tablet
I20251027 21:08:02.310849 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e65fecdeb104706bc6ffe7d7a038a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.311667 9595 tablet_service.cc:1552] Processing DeleteTablet for tablet 4da0bb391aba49cbaeb72983553362b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.311929 10755 consensus_queue.cc:1048] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251027 21:08:02.312064 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a4aa1fe05114c1287ca580c1f411d3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:02.314379 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 138a5d18566240858b5c92c3c03b8087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.314780 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9e8d0ad43154e4daaa513c50bbdef9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.315313 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 217fbfb1f58f420f8f258b4178901370 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.315869 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e419e588b31d48c8bc19d7948081cc81
I20251027 21:08:02.315975 10043 ts_tablet_manager.cc:1929] T abc7458fcfb94985acf76d77c0292b08 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.316285 9308 catalog_manager.cc:6719] ReplaceTablet: tablet e419e588b31d48c8bc19d7948081cc81 deleted and replaced by tablet cb5ac0fff70b491e8c8842f7a331005e
I20251027 21:08:02.319551 9595 tablet_service.cc:1552] Processing DeleteTablet for tablet e419e588b31d48c8bc19d7948081cc81 with delete_type TABLET_DATA_DELETED (replaced by tablet cb5ac0fff70b491e8c8842f7a331005e) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.319725 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38e6a73ff9314e47a16a6c5164650828. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.319900 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet e419e588b31d48c8bc19d7948081cc81 with delete_type TABLET_DATA_DELETED (replaced by tablet cb5ac0fff70b491e8c8842f7a331005e) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:02.322283 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0857f89bc74495081d35ee55e785835. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.325783 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet e419e588b31d48c8bc19d7948081cc81 with delete_type TABLET_DATA_DELETED (replaced by tablet cb5ac0fff70b491e8c8842f7a331005e) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.327018 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_439c4d078d484b7da5f47d76aca5a4a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:02.327520 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_439c4d078d484b7da5f47d76aca5a4a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.327955 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_47c1b3ca9dc541a6895e918264fd993f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:02.328610 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47c1b3ca9dc541a6895e918264fd993f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.330109 10041 ts_tablet_manager.cc:1929] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.330237 10041 log.cc:1199] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/8b30052ff19744a9bf4bfaaa1bc77b36
I20251027 21:08:02.330489 10041 ts_tablet_manager.cc:1950] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:02.333822 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 8b30052ff19744a9bf4bfaaa1bc77b36 (table default.Table_5dc88891b75b4f1ca7b3ffba0d2b1fda [id=b46f1e4f1e6d4bd8b4a3edd1d6787612]) successfully deleted
I20251027 21:08:02.334647 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet fb7fdc91156e41de80872b5c941c5d95
I20251027 21:08:02.335606 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet fb7fdc91156e41de80872b5c941c5d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:02.335805 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet fb7fdc91156e41de80872b5c941c5d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.336126 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet fb7fdc91156e41de80872b5c941c5d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.336324 10041 tablet_replica.cc:333] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:02.336547 10041 raft_consensus.cc:2243] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.336704 10041 raft_consensus.cc:2272] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.337945 10041 ts_tablet_manager.cc:1916] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.338162 10043 log.cc:1199] T abc7458fcfb94985acf76d77c0292b08 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/abc7458fcfb94985acf76d77c0292b08
I20251027 21:08:02.338485 10043 ts_tablet_manager.cc:1950] T abc7458fcfb94985acf76d77c0292b08 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:02.339520 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet abc7458fcfb94985acf76d77c0292b08 (table default.Table_9c027ff744824da9b9cac5574017e7fd [id=40a6848701284864b34f727da289347b]) successfully deleted
W20251027 21:08:02.339792 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet c9c86253ce914916bee6584675a82067 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9c86253ce914916bee6584675a82067 already in progress: creating tablet
W20251027 21:08:02.340018 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 4c023428437f45b4bae1374bd3dd36a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c023428437f45b4bae1374bd3dd36a7 already in progress: creating tablet
W20251027 21:08:02.340258 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 0bee9d4601c94b138d68f7663769ccbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bee9d4601c94b138d68f7663769ccbe already in progress: creating tablet
W20251027 21:08:02.340456 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 4da0bb391aba49cbaeb72983553362b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4da0bb391aba49cbaeb72983553362b2 already in progress: creating tablet
W20251027 21:08:02.340677 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet e419e588b31d48c8bc19d7948081cc81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e419e588b31d48c8bc19d7948081cc81 already in progress: creating tablet
W20251027 21:08:02.343385 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d8224d24b1144ff0b8bc593fc4e244de consensus state has no leader
W20251027 21:08:02.344061 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet fb7fdc91156e41de80872b5c941c5d95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb7fdc91156e41de80872b5c941c5d95 already in progress: creating tablet
I20251027 21:08:02.344355 10032 tablet_bootstrap.cc:492] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:02.344499 10032 ts_tablet_manager.cc:1403] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.121s user 0.017s sys 0.007s
I20251027 21:08:02.345201 10032 raft_consensus.cc:359] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.345371 10032 raft_consensus.cc:385] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.345417 10032 raft_consensus.cc:740] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.345587 10032 consensus_queue.cc:260] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.346078 10032 ts_tablet_manager.cc:1434] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:02.346809 10032 tablet_bootstrap.cc:492] T 32f80614e30d41128f9fb2e0ea165223 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:02.348312 10032 tablet_bootstrap.cc:654] T 32f80614e30d41128f9fb2e0ea165223 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.350198 9306 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet a9339c6b1154400eace53d765528fdf9 (table default.Table_5a6211c8465040dbad1cc86377a370ea [id=91fb9f95fe9343fca1d8cb0d0f781afb]). Expected version 2 got 0
I20251027 21:08:02.350883 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d3be7b0de84549a38b43f01c7efd058a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:02.351347 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3be7b0de84549a38b43f01c7efd058a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.356863 9306 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 980eb6755aee4bcfb2ea44d84009021f (table default.Table_bb56319b6fe0405991a4d171477621e7 [id=42c8fedcabe0435dbaa11427141228ed]). Expected version 1 got 0
I20251027 21:08:02.359941 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 2e65fecdeb104706bc6ffe7d7a038a7e (DEFAULT_TABLE table=default.Table_2096e2f92ca2449485e81a1a68807ef2 [id=a733fa5973a74f3d961bdda216316529]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.360345 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e65fecdeb104706bc6ffe7d7a038a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.362942 10029 tablet_bootstrap.cc:492] T 92a2393cff8044028c86c8cd2bccd930 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.363078 10029 ts_tablet_manager.cc:1403] T 92a2393cff8044028c86c8cd2bccd930 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.141s user 0.001s sys 0.003s
I20251027 21:08:02.363972 10029 raft_consensus.cc:359] T 92a2393cff8044028c86c8cd2bccd930 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.364159 10029 raft_consensus.cc:385] T 92a2393cff8044028c86c8cd2bccd930 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.364212 10029 raft_consensus.cc:740] T 92a2393cff8044028c86c8cd2bccd930 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.364430 10029 consensus_queue.cc:260] T 92a2393cff8044028c86c8cd2bccd930 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.368604 10031 ts_tablet_manager.cc:1434] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.080s user 0.001s sys 0.001s
I20251027 21:08:02.369961 9306 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet a9339c6b1154400eace53d765528fdf9 (table default.Table_5a6211c8465040dbad1cc86377a370ea [id=91fb9f95fe9343fca1d8cb0d0f781afb]). Expected version 2 got 0
I20251027 21:08:02.371022 10029 ts_tablet_manager.cc:1434] T 92a2393cff8044028c86c8cd2bccd930 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251027 21:08:02.371268 10029 tablet_bootstrap.cc:492] T c1f99ca89f924b58b1416d2945a6ba40 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:02.373378 10031 tablet_bootstrap.cc:492] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:02.374493 10040 tablet_replica.cc:333] T 69da799c886841a19002d3c7c5084e73 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
W20251027 21:08:02.374608 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc97e55cfbcb41ec8c6c7410bbf14fb3 consensus state has no leader
W20251027 21:08:02.374922 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 217fbfb1f58f420f8f258b4178901370 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 217fbfb1f58f420f8f258b4178901370 already in progress: creating tablet
I20251027 21:08:02.375720 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 4ae59aace74b48199219ef2252d4c7eb (DEFAULT_TABLE table=default.Table_5f6e0aaf15fe439c87306828e9a8cb1b [id=933d2ef428b24d2eb9c18d89c15e7819]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.376060 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ae59aace74b48199219ef2252d4c7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.377209 10029 tablet_bootstrap.cc:654] T c1f99ca89f924b58b1416d2945a6ba40 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.378346 9739 tablet_service.cc:1505] Processing CreateTablet for tablet 4ae59aace74b48199219ef2252d4c7eb (DEFAULT_TABLE table=default.Table_5f6e0aaf15fe439c87306828e9a8cb1b [id=933d2ef428b24d2eb9c18d89c15e7819]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.378659 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ae59aace74b48199219ef2252d4c7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.378743 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 2e65fecdeb104706bc6ffe7d7a038a7e (DEFAULT_TABLE table=default.Table_2096e2f92ca2449485e81a1a68807ef2 [id=a733fa5973a74f3d961bdda216316529]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.379108 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e65fecdeb104706bc6ffe7d7a038a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.379999 10755 consensus_queue.cc:1048] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:02.379124 10040 raft_consensus.cc:2243] T 69da799c886841a19002d3c7c5084e73 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.381613 10040 raft_consensus.cc:2272] T 69da799c886841a19002d3c7c5084e73 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.382131 10040 ts_tablet_manager.cc:1916] T 69da799c886841a19002d3c7c5084e73 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.382486 10041 ts_tablet_manager.cc:1929] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.383816 10041 log.cc:1199] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/c9efffa9766e46aca8eb286fc2ad48d2
I20251027 21:08:02.384301 10041 ts_tablet_manager.cc:1950] T c9efffa9766e46aca8eb286fc2ad48d2 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:02.385708 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet c9efffa9766e46aca8eb286fc2ad48d2 (table default.Table_97b01c1840c543838263ca4222695f54 [id=bea51a58fda64d40916f306c4aa8ab4e]) successfully deleted
W20251027 21:08:02.387423 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 0bee9d4601c94b138d68f7663769ccbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bee9d4601c94b138d68f7663769ccbe already in progress: creating tablet
I20251027 21:08:02.388116 10041 tablet_replica.cc:333] T a26ded6012174348b7295fc7c0e05e6e P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:02.388720 10041 raft_consensus.cc:2243] T a26ded6012174348b7295fc7c0e05e6e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.389175 10041 raft_consensus.cc:2272] T a26ded6012174348b7295fc7c0e05e6e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.390195 10041 ts_tablet_manager.cc:1916] T a26ded6012174348b7295fc7c0e05e6e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.390894 10029 tablet_bootstrap.cc:492] T c1f99ca89f924b58b1416d2945a6ba40 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.391330 10032 tablet_bootstrap.cc:492] T 32f80614e30d41128f9fb2e0ea165223 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:02.391364 10029 ts_tablet_manager.cc:1403] T c1f99ca89f924b58b1416d2945a6ba40 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
I20251027 21:08:02.391448 10032 ts_tablet_manager.cc:1403] T 32f80614e30d41128f9fb2e0ea165223 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.045s user 0.000s sys 0.003s
I20251027 21:08:02.392105 10029 raft_consensus.cc:359] T c1f99ca89f924b58b1416d2945a6ba40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.392246 10029 raft_consensus.cc:385] T c1f99ca89f924b58b1416d2945a6ba40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.392299 10029 raft_consensus.cc:740] T c1f99ca89f924b58b1416d2945a6ba40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.392549 10029 consensus_queue.cc:260] T c1f99ca89f924b58b1416d2945a6ba40 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.393092 10029 ts_tablet_manager.cc:1434] T c1f99ca89f924b58b1416d2945a6ba40 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:02.393085 10829 raft_consensus.cc:493] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:02.393185 10829 raft_consensus.cc:515] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.393663 10829 leader_election.cc:290] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:02.394469 9711 leader_election.cc:343] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: f34512059f0c47fc9af7db2645f86b40
I20251027 21:08:02.394564 10032 raft_consensus.cc:359] T 32f80614e30d41128f9fb2e0ea165223 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.394716 10032 raft_consensus.cc:385] T 32f80614e30d41128f9fb2e0ea165223 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.394767 10032 raft_consensus.cc:740] T 32f80614e30d41128f9fb2e0ea165223 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.394949 10032 consensus_queue.cc:260] T 32f80614e30d41128f9fb2e0ea165223 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.395390 10032 ts_tablet_manager.cc:1434] T 32f80614e30d41128f9fb2e0ea165223 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251027 21:08:02.395824 10032 tablet_bootstrap.cc:492] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:08:02.396903 9710 leader_election.cc:343] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: f34512059f0c47fc9af7db2645f86b40
I20251027 21:08:02.396996 9710 leader_election.cc:304] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:02.397328 10829 raft_consensus.cc:2749] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:02.398603 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 87612b8bd8a04fd9b32f2df3f9e0d997 consensus state has no leader
I20251027 21:08:02.398717 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_563019081d2f4beebbe8dce71dca1b65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:02.400054 9304 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet a9339c6b1154400eace53d765528fdf9 (table default.Table_5a6211c8465040dbad1cc86377a370ea [id=91fb9f95fe9343fca1d8cb0d0f781afb]). Expected version 2 got 0
W20251027 21:08:02.400902 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_563019081d2f4beebbe8dce71dca1b65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.401942 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b00669c899a945d283e7e68bcbc4384b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:02.402525 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b00669c899a945d283e7e68bcbc4384b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.402263 10031 tablet_bootstrap.cc:654] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.402060 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_555cf66116b243de86d5cdd0daf42151"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:02.403978 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_555cf66116b243de86d5cdd0daf42151 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.404342 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet 92a2393cff8044028c86c8cd2bccd930 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.404587 10043 tablet_replica.cc:333] T 92a2393cff8044028c86c8cd2bccd930 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:02.404754 10043 raft_consensus.cc:2243] T 92a2393cff8044028c86c8cd2bccd930 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.404882 10043 raft_consensus.cc:2272] T 92a2393cff8044028c86c8cd2bccd930 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.405349 10029 tablet_bootstrap.cc:492] T 9882b234e79940088be5f604c41576b7 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:02.405582 10043 ts_tablet_manager.cc:1916] T 92a2393cff8044028c86c8cd2bccd930 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.407768 10029 tablet_bootstrap.cc:654] T 9882b234e79940088be5f604c41576b7 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:02.408706 10848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 21:08:02.413285 10043 ts_tablet_manager.cc:1929] T 92a2393cff8044028c86c8cd2bccd930 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.413470 10043 log.cc:1199] T 92a2393cff8044028c86c8cd2bccd930 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/92a2393cff8044028c86c8cd2bccd930
I20251027 21:08:02.413794 10043 ts_tablet_manager.cc:1950] T 92a2393cff8044028c86c8cd2bccd930 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:02.414316 10029 tablet_bootstrap.cc:492] T 9882b234e79940088be5f604c41576b7 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.414556 10029 ts_tablet_manager.cc:1403] T 9882b234e79940088be5f604c41576b7 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20251027 21:08:02.414980 10041 ts_tablet_manager.cc:1929] T a26ded6012174348b7295fc7c0e05e6e P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.415201 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 92a2393cff8044028c86c8cd2bccd930 (table default.Table_1d766c40686340d59cc4680fbf8a2dbb [id=e5fba763a46b488cb19c738cee468ad1]) successfully deleted
I20251027 21:08:02.415331 10029 raft_consensus.cc:359] T 9882b234e79940088be5f604c41576b7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.415525 10029 raft_consensus.cc:385] T 9882b234e79940088be5f604c41576b7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.415581 10029 raft_consensus.cc:740] T 9882b234e79940088be5f604c41576b7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.415853 10029 consensus_queue.cc:260] T 9882b234e79940088be5f604c41576b7 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.419620 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 999a737767824f03b857caa3d7cd9cdf (DEFAULT_TABLE table=default.Table_c5c0ff9082504f9f99ba0d43d21acbc2 [id=23e1c6d92bc84af09aa5a4c375773e44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.419966 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 999a737767824f03b857caa3d7cd9cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.420137 10041 log.cc:1199] T a26ded6012174348b7295fc7c0e05e6e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/a26ded6012174348b7295fc7c0e05e6e
I20251027 21:08:02.420440 10029 ts_tablet_manager.cc:1434] T 9882b234e79940088be5f604c41576b7 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251027 21:08:02.421149 10040 ts_tablet_manager.cc:1929] T 69da799c886841a19002d3c7c5084e73 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.421242 10040 log.cc:1199] T 69da799c886841a19002d3c7c5084e73 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/69da799c886841a19002d3c7c5084e73
I20251027 21:08:02.421192 10041 ts_tablet_manager.cc:1950] T a26ded6012174348b7295fc7c0e05e6e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:02.421475 10040 ts_tablet_manager.cc:1950] T 69da799c886841a19002d3c7c5084e73 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:02.422012 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 69da799c886841a19002d3c7c5084e73 (table default.Table_c2e56e5918cb4007b681c71977325233 [id=0cf4bae5f10d45c1a41880b45fa0b0f0]) successfully deleted
W20251027 21:08:02.422490 10844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 21:08:02.423254 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet a26ded6012174348b7295fc7c0e05e6e (table default.Table_69f618a7f3c74a1d87cb5e0c17f38f94 [id=d37bad05f12e4d0da8934baeb7627de6]) successfully deleted
I20251027 21:08:02.424026 10041 tablet_replica.cc:333] T da26c83976df40328cf9b11734fcfa44 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:02.424412 10041 raft_consensus.cc:2243] T da26c83976df40328cf9b11734fcfa44 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.424795 10041 raft_consensus.cc:2272] T da26c83976df40328cf9b11734fcfa44 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.425861 10041 ts_tablet_manager.cc:1916] T da26c83976df40328cf9b11734fcfa44 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.432808 10041 ts_tablet_manager.cc:1929] T da26c83976df40328cf9b11734fcfa44 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.433293 10040 tablet_replica.cc:333] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:02.433467 10040 raft_consensus.cc:2243] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.433569 10829 raft_consensus.cc:493] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:02.433741 10829 raft_consensus.cc:515] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.434403 10829 leader_election.cc:290] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:02.434981 10829 raft_consensus.cc:493] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:02.435122 10829 raft_consensus.cc:515] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.435202 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet c1f99ca89f924b58b1416d2945a6ba40 with delete_type TABLET_DATA_DELETED (replaced by tablet d25b493ab8ca401bb1cbe01e355d7bcd) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.435539 10043 tablet_replica.cc:333] T c1f99ca89f924b58b1416d2945a6ba40 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:02.435765 10829 leader_election.cc:290] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:02.435923 10043 raft_consensus.cc:2243] T c1f99ca89f924b58b1416d2945a6ba40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.436096 10043 raft_consensus.cc:2272] T c1f99ca89f924b58b1416d2945a6ba40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:02.436702 9710 leader_election.cc:343] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 92a2393cff8044028c86c8cd2bccd930
W20251027 21:08:02.437127 9710 leader_election.cc:343] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 0a58cd07e1bb4ffdbf39e7e16a46022a
I20251027 21:08:02.437633 10029 tablet_bootstrap.cc:492] T adaa4f66bbab491f8030d439d85cc7d1 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
W20251027 21:08:02.438359 9711 leader_election.cc:343] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 92a2393cff8044028c86c8cd2bccd930
I20251027 21:08:02.438375 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 9882b234e79940088be5f604c41576b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.438468 9711 leader_election.cc:304] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
W20251027 21:08:02.438814 9711 leader_election.cc:343] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 0a58cd07e1bb4ffdbf39e7e16a46022a
I20251027 21:08:02.438894 9711 leader_election.cc:304] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:02.439158 10043 ts_tablet_manager.cc:1916] T c1f99ca89f924b58b1416d2945a6ba40 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.433616 10040 raft_consensus.cc:2272] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.439213 10718 raft_consensus.cc:2749] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:02.439297 10029 tablet_bootstrap.cc:654] T adaa4f66bbab491f8030d439d85cc7d1 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.439394 10718 raft_consensus.cc:2749] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:02.439659 10040 ts_tablet_manager.cc:1916] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.443567 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_707e838734704c27a8dd97d0d58e6a33" } new_table_name: "default.Table_8ebde3403af34f4589ee6222700b2283" modify_external_catalogs: true
I20251027 21:08:02.445693 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 999a737767824f03b857caa3d7cd9cdf (DEFAULT_TABLE table=default.Table_c5c0ff9082504f9f99ba0d43d21acbc2 [id=23e1c6d92bc84af09aa5a4c375773e44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.446041 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 999a737767824f03b857caa3d7cd9cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.447638 10029 tablet_bootstrap.cc:492] T adaa4f66bbab491f8030d439d85cc7d1 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.447758 10029 ts_tablet_manager.cc:1403] T adaa4f66bbab491f8030d439d85cc7d1 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
W20251027 21:08:02.449717 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b748f59a56c54c2b850f5336a1d61c26 consensus state has no leader
I20251027 21:08:02.449918 10029 raft_consensus.cc:359] T adaa4f66bbab491f8030d439d85cc7d1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.450052 10029 raft_consensus.cc:385] T adaa4f66bbab491f8030d439d85cc7d1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.450093 10029 raft_consensus.cc:740] T adaa4f66bbab491f8030d439d85cc7d1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.453430 9307 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet a9339c6b1154400eace53d765528fdf9 (table default.Table_5a6211c8465040dbad1cc86377a370ea [id=91fb9f95fe9343fca1d8cb0d0f781afb]). Expected version 2 got 0
I20251027 21:08:02.454095 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_35f4f0dcffb34a5e9b565ebd86d78445"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:02.454272 10029 consensus_queue.cc:260] T adaa4f66bbab491f8030d439d85cc7d1 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:02.454603 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35f4f0dcffb34a5e9b565ebd86d78445 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.439229 10031 tablet_bootstrap.cc:492] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:02.454875 10031 ts_tablet_manager.cc:1403] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.082s user 0.006s sys 0.002s
I20251027 21:08:02.455523 10031 raft_consensus.cc:359] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.455662 10031 raft_consensus.cc:385] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.455719 10031 raft_consensus.cc:740] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.455890 10031 consensus_queue.cc:260] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.457531 10681 raft_consensus.cc:493] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:02.457649 10681 raft_consensus.cc:515] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.458113 10031 ts_tablet_manager.cc:1434] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:08:02.458391 10031 tablet_bootstrap.cc:492] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:02.458428 10681 leader_election.cc:290] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:02.459444 10029 ts_tablet_manager.cc:1434] T adaa4f66bbab491f8030d439d85cc7d1 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251027 21:08:02.461074 9739 tablet_service.cc:1505] Processing CreateTablet for tablet fee5f2cf523f4500a6c898fc8de3ccf8 (DEFAULT_TABLE table=default.Table_b85311fce5904b9fa4e6bc7c318bbd47 [id=0f50da86a002427bbfd8bd5e850ce103]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.461392 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fee5f2cf523f4500a6c898fc8de3ccf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.418267 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 4ae59aace74b48199219ef2252d4c7eb (DEFAULT_TABLE table=default.Table_5f6e0aaf15fe439c87306828e9a8cb1b [id=933d2ef428b24d2eb9c18d89c15e7819]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.434121 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 32f80614e30d41128f9fb2e0ea165223 with delete_type TABLET_DATA_DELETED (replaced by tablet a7d8d060916941a5b7e944e8b08a2882) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:02.461587 9614 tablet_service.cc:1505] Processing CreateTablet for tablet fee5f2cf523f4500a6c898fc8de3ccf8 (DEFAULT_TABLE table=default.Table_b85311fce5904b9fa4e6bc7c318bbd47 [id=0f50da86a002427bbfd8bd5e850ce103]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.461980 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fee5f2cf523f4500a6c898fc8de3ccf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.462821 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ae59aace74b48199219ef2252d4c7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.463313 10029 tablet_bootstrap.cc:492] T 3103c8e5a7c744be90be097179904ba3 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:02.464771 10029 tablet_bootstrap.cc:654] T 3103c8e5a7c744be90be097179904ba3 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.417125 10032 tablet_bootstrap.cc:654] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.458806 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 999a737767824f03b857caa3d7cd9cdf (DEFAULT_TABLE table=default.Table_c5c0ff9082504f9f99ba0d43d21acbc2 [id=23e1c6d92bc84af09aa5a4c375773e44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.461182 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8856c5dc8f144358445165324bfb477" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:02.467659 9924 raft_consensus.cc:2468] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:02.467728 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 999a737767824f03b857caa3d7cd9cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.468948 10029 tablet_bootstrap.cc:492] T 3103c8e5a7c744be90be097179904ba3 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.469070 10029 ts_tablet_manager.cc:1403] T 3103c8e5a7c744be90be097179904ba3 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20251027 21:08:02.469699 10029 raft_consensus.cc:359] T 3103c8e5a7c744be90be097179904ba3 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.469853 10029 raft_consensus.cc:385] T 3103c8e5a7c744be90be097179904ba3 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.469913 10029 raft_consensus.cc:740] T 3103c8e5a7c744be90be097179904ba3 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.470311 10041 log.cc:1199] T da26c83976df40328cf9b11734fcfa44 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/da26c83976df40328cf9b11734fcfa44
I20251027 21:08:02.470410 9568 leader_election.cc:304] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:02.470705 10681 raft_consensus.cc:2804] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:02.470737 10041 ts_tablet_manager.cc:1950] T da26c83976df40328cf9b11734fcfa44 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:02.470791 10681 raft_consensus.cc:493] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:02.470873 10681 raft_consensus.cc:3060] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:02.471825 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet da26c83976df40328cf9b11734fcfa44 (table default.Table_ac66ea1435fe49518c073dfc9b2368c8 [id=985d97a0b33f4ef79a9b0df362a75c32]) successfully deleted
W20251027 21:08:02.472059 10856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 21:08:02.473153 10681 raft_consensus.cc:515] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.473554 10031 tablet_bootstrap.cc:654] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.474012 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8856c5dc8f144358445165324bfb477" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:02.474215 9924 raft_consensus.cc:3060] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:02.474604 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8856c5dc8f144358445165324bfb477" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:02.475071 10029 consensus_queue.cc:260] T 3103c8e5a7c744be90be097179904ba3 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.476933 9924 raft_consensus.cc:2468] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:02.477010 9303 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet a9339c6b1154400eace53d765528fdf9 (table default.Table_5a6211c8465040dbad1cc86377a370ea [id=91fb9f95fe9343fca1d8cb0d0f781afb]). Expected version 2 got 0
I20251027 21:08:02.478099 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet 43611e33512b49d58485cf15de788be7 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:02.478428 10029 ts_tablet_manager.cc:1434] T 3103c8e5a7c744be90be097179904ba3 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
W20251027 21:08:02.478966 9566 leader_election.cc:343] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:02.479302 9568 leader_election.cc:304] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:02.479564 10755 raft_consensus.cc:2804] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
W20251027 21:08:02.480849 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 217fbfb1f58f420f8f258b4178901370 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 217fbfb1f58f420f8f258b4178901370 already in progress: creating tablet
I20251027 21:08:02.481776 10041 tablet_replica.cc:333] T 2a4aa1fe05114c1287ca580c1f411d3f P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:02.481902 10029 tablet_bootstrap.cc:492] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:02.482034 10041 raft_consensus.cc:2243] T 2a4aa1fe05114c1287ca580c1f411d3f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.482204 10041 raft_consensus.cc:2272] T 2a4aa1fe05114c1287ca580c1f411d3f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.483054 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet adaa4f66bbab491f8030d439d85cc7d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.483158 9895 tablet_service.cc:1505] Processing CreateTablet for tablet fee5f2cf523f4500a6c898fc8de3ccf8 (DEFAULT_TABLE table=default.Table_b85311fce5904b9fa4e6bc7c318bbd47 [id=0f50da86a002427bbfd8bd5e850ce103]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.483479 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fee5f2cf523f4500a6c898fc8de3ccf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.484368 10829 raft_consensus.cc:493] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:02.484480 10829 raft_consensus.cc:515] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.485008 10829 leader_election.cc:290] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:02.485644 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 43b5cef5fdcd4d2fa89875aa6d0efcd0 (DEFAULT_TABLE table=default.Table_c5e35fa342f64432b4654f177b365624 [id=387596c8ecd74edc9ad877e8ed9fb72d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.485977 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43b5cef5fdcd4d2fa89875aa6d0efcd0. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:02.485918 9711 leader_election.cc:343] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: c1f99ca89f924b58b1416d2945a6ba40
I20251027 21:08:02.487018 9739 tablet_service.cc:1505] Processing CreateTablet for tablet 43b5cef5fdcd4d2fa89875aa6d0efcd0 (DEFAULT_TABLE table=default.Table_c5e35fa342f64432b4654f177b365624 [id=387596c8ecd74edc9ad877e8ed9fb72d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.487427 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43b5cef5fdcd4d2fa89875aa6d0efcd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.491305 10029 tablet_bootstrap.cc:654] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.474355 10681 leader_election.cc:290] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:02.492204 10041 ts_tablet_manager.cc:1916] T 2a4aa1fe05114c1287ca580c1f411d3f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.493252 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 0f90e09e74b6495db1aa2df85de21181
I20251027 21:08:02.493674 9311 catalog_manager.cc:6719] ReplaceTablet: tablet 0f90e09e74b6495db1aa2df85de21181 deleted and replaced by tablet 45d67d0ef7294b3ea63529db2cdaafaf
I20251027 21:08:02.495368 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f90e09e74b6495db1aa2df85de21181 with delete_type TABLET_DATA_DELETED (replaced by tablet 45d67d0ef7294b3ea63529db2cdaafaf) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.495615 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f90e09e74b6495db1aa2df85de21181 with delete_type TABLET_DATA_DELETED (replaced by tablet 45d67d0ef7294b3ea63529db2cdaafaf) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:02.496552 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f90e09e74b6495db1aa2df85de21181 with delete_type TABLET_DATA_DELETED (replaced by tablet 45d67d0ef7294b3ea63529db2cdaafaf) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.497166 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8856c5dc8f144358445165324bfb477" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:02.497932 9566 leader_election.cc:343] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:02.503435 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 9882b234e79940088be5f604c41576b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.503839 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_ae5235e10b154ddd95f2aa9bb7d34a6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:02.505143 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae5235e10b154ddd95f2aa9bb7d34a6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.507370 10639 server_base.cc:1047] running on GCE node
I20251027 21:08:02.508455 10639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 21:08:02.509434 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 43b5cef5fdcd4d2fa89875aa6d0efcd0 (DEFAULT_TABLE table=default.Table_c5e35fa342f64432b4654f177b365624 [id=387596c8ecd74edc9ad877e8ed9fb72d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.509850 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43b5cef5fdcd4d2fa89875aa6d0efcd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.510257 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 3103c8e5a7c744be90be097179904ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.516211 10040 ts_tablet_manager.cc:1929] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.516320 10040 log.cc:1199] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/d52d71f4976143e4a514350621ebe309
I20251027 21:08:02.516620 10040 ts_tablet_manager.cc:1950] T d52d71f4976143e4a514350621ebe309 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:02.517551 10040 tablet_replica.cc:333] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:02.517784 10040 raft_consensus.cc:2243] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.517921 10040 raft_consensus.cc:2272] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.518716 10755 raft_consensus.cc:697] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:02.519011 10755 consensus_queue.cc:237] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.520438 9890 tablet_service.cc:1505] Processing CreateTablet for tablet c2992034f072482382a4a593fb7f7aaa (DEFAULT_TABLE table=default.Table_b94af6879cd940829a7f531277d5544c [id=72afefacacf048cf87cb4ff1706051dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.520493 9614 tablet_service.cc:1505] Processing CreateTablet for tablet c2992034f072482382a4a593fb7f7aaa (DEFAULT_TABLE table=default.Table_b94af6879cd940829a7f531277d5544c [id=72afefacacf048cf87cb4ff1706051dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.520758 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2992034f072482382a4a593fb7f7aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.520957 9615 tablet_service.cc:1505] Processing CreateTablet for tablet ce9bad29c7b24f4cb729c8ef2e163c72 (DEFAULT_TABLE table=default.Table_48f4510204084408aef690097faeaca1 [id=17026cd878fb45519af322bb5185654e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.521253 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce9bad29c7b24f4cb729c8ef2e163c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.525593 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet d52d71f4976143e4a514350621ebe309 (table default.Table_c305a5155cce4754bcbc792f3900b690 [id=82b0b6e6d2e145a7870a13ed5b407b47]) successfully deleted
W20251027 21:08:02.525883 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 85ab6ac53e604ca0bfe051373606fe71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85ab6ac53e604ca0bfe051373606fe71 already in progress: creating tablet
I20251027 21:08:02.526300 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2992034f072482382a4a593fb7f7aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.528077 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d3a044fd69ff47428850c4d429855c80" } modify_external_catalogs: true
I20251027 21:08:02.528242 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d3a044fd69ff47428850c4d429855c80" } modify_external_catalogs: true
I20251027 21:08:02.529989 10639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 21:08:02.530588 9739 tablet_service.cc:1505] Processing CreateTablet for tablet c2992034f072482382a4a593fb7f7aaa (DEFAULT_TABLE table=default.Table_b94af6879cd940829a7f531277d5544c [id=72afefacacf048cf87cb4ff1706051dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.531427 10029 tablet_bootstrap.cc:492] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.531551 10029 ts_tablet_manager.cc:1403] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.001s
I20251027 21:08:02.532308 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2992034f072482382a4a593fb7f7aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.532245 10029 raft_consensus.cc:359] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.532384 10029 raft_consensus.cc:385] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.532428 10029 raft_consensus.cc:740] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.532589 10029 consensus_queue.cc:260] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.533020 10029 ts_tablet_manager.cc:1434] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251027 21:08:02.533262 10029 tablet_bootstrap.cc:492] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:02.533354 10639 hybrid_clock.cc:648] HybridClock initialized: now 1761599282533207 us; error 142 us; skew 500 ppm
I20251027 21:08:02.533773 10639 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 21:08:02.537175 10029 tablet_bootstrap.cc:654] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.537340 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1f99ca89f924b58b1416d2945a6ba40" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:02.537998 9710 leader_election.cc:343] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: Tablet not RUNNING: STOPPED
I20251027 21:08:02.538127 9710 leader_election.cc:304] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:02.538396 10829 raft_consensus.cc:2749] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:02.541493 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c5e35fa342f64432b4654f177b365624" } modify_external_catalogs: true
I20251027 21:08:02.541685 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c5e35fa342f64432b4654f177b365624" } modify_external_catalogs: true
I20251027 21:08:02.545212 9755 tablet_service.cc:1505] Processing CreateTablet for tablet ce9bad29c7b24f4cb729c8ef2e163c72 (DEFAULT_TABLE table=default.Table_48f4510204084408aef690097faeaca1 [id=17026cd878fb45519af322bb5185654e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.545392 10041 ts_tablet_manager.cc:1929] T 2a4aa1fe05114c1287ca580c1f411d3f P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.545496 10041 log.cc:1199] T 2a4aa1fe05114c1287ca580c1f411d3f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/2a4aa1fe05114c1287ca580c1f411d3f
I20251027 21:08:02.545581 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce9bad29c7b24f4cb729c8ef2e163c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.545769 10041 ts_tablet_manager.cc:1950] T 2a4aa1fe05114c1287ca580c1f411d3f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:02.546099 9890 tablet_service.cc:1505] Processing CreateTablet for tablet ce9bad29c7b24f4cb729c8ef2e163c72 (DEFAULT_TABLE table=default.Table_48f4510204084408aef690097faeaca1 [id=17026cd878fb45519af322bb5185654e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.546471 10032 tablet_bootstrap.cc:492] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:02.546597 10032 ts_tablet_manager.cc:1403] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.151s user 0.002s sys 0.001s
I20251027 21:08:02.546471 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce9bad29c7b24f4cb729c8ef2e163c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.546698 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 2a4aa1fe05114c1287ca580c1f411d3f (table default.Table_f08509866aae44609f26ce5e30963f6c [id=63bc2d46b97e4054b557025436c371d9]) successfully deleted
I20251027 21:08:02.547392 10032 raft_consensus.cc:359] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.547531 10032 raft_consensus.cc:385] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.547572 10032 raft_consensus.cc:740] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.547780 10032 consensus_queue.cc:260] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.550745 10032 ts_tablet_manager.cc:1434] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251027 21:08:02.551015 10032 tablet_bootstrap.cc:492] T c3b9a461567a48fcb4af514ecdf618b4 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:02.552507 10032 tablet_bootstrap.cc:654] T c3b9a461567a48fcb4af514ecdf618b4 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.556530 10043 ts_tablet_manager.cc:1929] T c1f99ca89f924b58b1416d2945a6ba40 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.556659 10043 log.cc:1199] T c1f99ca89f924b58b1416d2945a6ba40 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/c1f99ca89f924b58b1416d2945a6ba40
I20251027 21:08:02.564575 9305 catalog_manager.cc:5649] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:02.564985 9305 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet b8856c5dc8f144358445165324bfb477 (table default.Table_203191e192be4e3a9a7477b4313ef4f4 [id=5da6bd4463e24c03accd9985f72b6ebb]). Expected version 3 got 0
I20251027 21:08:02.566686 10639 webserver.cc:492] Webserver started at http://127.9.14.124:41907/ using document root <none> and password file <none>
I20251027 21:08:02.568365 10639 fs_manager.cc:362] Metadata directory not provided
I20251027 21:08:02.569170 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 82f102a02d104d32a48a790240cb86c7 (DEFAULT_TABLE table=default.Table_ba2ceb40c6f745c099ab46df479c912c [id=dd8154a52dd34729adfe3cd9d56c0754]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.569527 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82f102a02d104d32a48a790240cb86c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.575423 9306 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet ca477937278f45b4b115fec715f32a92 (table default.Table_69627cab65c2407b9226a05106b7bdf4 [id=fc5a0b5a96e6400da438307d26f73614]). Expected version 3 got 0
W20251027 21:08:02.576061 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet e419e588b31d48c8bc19d7948081cc81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e419e588b31d48c8bc19d7948081cc81 already in progress: creating tablet
I20251027 21:08:02.576179 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_cfef185a988e4d61ae6575c228abc88d" } modify_external_catalogs: true
I20251027 21:08:02.576421 10639 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 21:08:02.576390 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_cfef185a988e4d61ae6575c228abc88d" } modify_external_catalogs: true
I20251027 21:08:02.579242 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 82f102a02d104d32a48a790240cb86c7 (DEFAULT_TABLE table=default.Table_ba2ceb40c6f745c099ab46df479c912c [id=dd8154a52dd34729adfe3cd9d56c0754]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.579630 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82f102a02d104d32a48a790240cb86c7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:02.586104 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet fb7fdc91156e41de80872b5c941c5d95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb7fdc91156e41de80872b5c941c5d95 already in progress: creating tablet
I20251027 21:08:02.586448 10041 tablet_replica.cc:333] T 32f80614e30d41128f9fb2e0ea165223 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:02.586668 10041 raft_consensus.cc:2243] T 32f80614e30d41128f9fb2e0ea165223 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.586843 10041 raft_consensus.cc:2272] T 32f80614e30d41128f9fb2e0ea165223 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.586887 10031 tablet_bootstrap.cc:492] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:02.587026 10031 ts_tablet_manager.cc:1403] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.129s user 0.001s sys 0.009s
I20251027 21:08:02.587707 9304 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet ca477937278f45b4b115fec715f32a92 (table default.Table_69627cab65c2407b9226a05106b7bdf4 [id=fc5a0b5a96e6400da438307d26f73614]). Expected version 3 got 0
I20251027 21:08:02.587752 10031 raft_consensus.cc:359] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.587886 10031 raft_consensus.cc:385] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.587935 10031 raft_consensus.cc:740] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.588135 10031 consensus_queue.cc:260] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.588398 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c1eb65a428444379825e1d29557bbc23" } modify_external_catalogs: true
I20251027 21:08:02.588551 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c1eb65a428444379825e1d29557bbc23" } modify_external_catalogs: true
I20251027 21:08:02.590788 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 1ba47e34f5464b47bc32b2776f37d97d (DEFAULT_TABLE table=default.Table_8c2f4fc52c37435fb5a25817b1545503 [id=a42eb3665bfc426a8d24f698c1276893]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.591107 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ba47e34f5464b47bc32b2776f37d97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.591924 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 1ba47e34f5464b47bc32b2776f37d97d (DEFAULT_TABLE table=default.Table_8c2f4fc52c37435fb5a25817b1545503 [id=a42eb3665bfc426a8d24f698c1276893]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.592234 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ba47e34f5464b47bc32b2776f37d97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.592236 10031 ts_tablet_manager.cc:1434] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251027 21:08:02.593199 10031 tablet_bootstrap.cc:492] T fe24af9838954a9bb680474800642e62 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:02.593446 10043 ts_tablet_manager.cc:1950] T c1f99ca89f924b58b1416d2945a6ba40 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:02.595057 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet c1f99ca89f924b58b1416d2945a6ba40 (table default.Table_90157e3841104d92aafc5d82ffea13a9 [id=bc676ad23e1f4c4bae88752f31108ee2]) successfully deleted
I20251027 21:08:02.567787 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 82f102a02d104d32a48a790240cb86c7 (DEFAULT_TABLE table=default.Table_ba2ceb40c6f745c099ab46df479c912c [id=dd8154a52dd34729adfe3cd9d56c0754]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.595573 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82f102a02d104d32a48a790240cb86c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.598469 10041 ts_tablet_manager.cc:1916] T 32f80614e30d41128f9fb2e0ea165223 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.592561 9615 tablet_service.cc:1505] Processing CreateTablet for tablet eaa869a2e41b4f83978b7e51c4717edd (DEFAULT_TABLE table=default.Table_066671f8fa8b403386935bcda7a8d095 [id=24bdea549f024941b08739d57ab2b231]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.561969 10029 tablet_bootstrap.cc:492] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.600464 10029 ts_tablet_manager.cc:1403] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.067s user 0.006s sys 0.002s
I20251027 21:08:02.601161 10029 raft_consensus.cc:359] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.601302 10029 raft_consensus.cc:385] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.601351 10029 raft_consensus.cc:740] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.601539 10029 consensus_queue.cc:260] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.601974 10029 ts_tablet_manager.cc:1434] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251027 21:08:02.603120 10029 tablet_bootstrap.cc:492] T b719dc40d7e348608cdfc78401c19824 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:02.604562 10029 tablet_bootstrap.cc:654] T b719dc40d7e348608cdfc78401c19824 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.606232 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaa869a2e41b4f83978b7e51c4717edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.609421 10041 ts_tablet_manager.cc:1929] T 32f80614e30d41128f9fb2e0ea165223 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.609513 10041 log.cc:1199] T 32f80614e30d41128f9fb2e0ea165223 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/32f80614e30d41128f9fb2e0ea165223
I20251027 21:08:02.609750 10041 ts_tablet_manager.cc:1950] T 32f80614e30d41128f9fb2e0ea165223 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:02.610987 10031 tablet_bootstrap.cc:654] T fe24af9838954a9bb680474800642e62 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.612365 10032 tablet_bootstrap.cc:492] T c3b9a461567a48fcb4af514ecdf618b4 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:02.612483 10032 ts_tablet_manager.cc:1403] T c3b9a461567a48fcb4af514ecdf618b4 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.062s user 0.002s sys 0.001s
I20251027 21:08:02.613119 10032 raft_consensus.cc:359] T c3b9a461567a48fcb4af514ecdf618b4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.613258 10032 raft_consensus.cc:385] T c3b9a461567a48fcb4af514ecdf618b4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.613301 10032 raft_consensus.cc:740] T c3b9a461567a48fcb4af514ecdf618b4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.613452 10032 consensus_queue.cc:260] T c3b9a461567a48fcb4af514ecdf618b4 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.614004 10032 ts_tablet_manager.cc:1434] T c3b9a461567a48fcb4af514ecdf618b4 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:02.617041 10032 tablet_bootstrap.cc:492] T 4debc3f13f3a42939b6b83f6523aab7c P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:02.620831 10829 raft_consensus.cc:493] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:02.620944 10829 raft_consensus.cc:515] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.621136 10029 tablet_bootstrap.cc:492] T b719dc40d7e348608cdfc78401c19824 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.621236 10029 ts_tablet_manager.cc:1403] T b719dc40d7e348608cdfc78401c19824 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20251027 21:08:02.621500 10829 leader_election.cc:290] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:02.621750 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f4c9a8d0ca5f4277b9ff989e1b832985" } modify_external_catalogs: true
I20251027 21:08:02.621919 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f4c9a8d0ca5f4277b9ff989e1b832985" } modify_external_catalogs: true
I20251027 21:08:02.621905 10029 raft_consensus.cc:359] T b719dc40d7e348608cdfc78401c19824 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.622066 10029 raft_consensus.cc:385] T b719dc40d7e348608cdfc78401c19824 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.622119 10029 raft_consensus.cc:740] T b719dc40d7e348608cdfc78401c19824 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.622553 10029 consensus_queue.cc:260] T b719dc40d7e348608cdfc78401c19824 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:08:02.622617 9711 leader_election.cc:343] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: e0968c2f25b64f6685f38a999ee9d60e
I20251027 21:08:02.623148 10029 ts_tablet_manager.cc:1434] T b719dc40d7e348608cdfc78401c19824 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:02.623425 10029 tablet_bootstrap.cc:492] T c4996d8fae094db8a96599b9057f1649 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:02.624127 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet c3b9a461567a48fcb4af514ecdf618b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:08:02.624411 9710 leader_election.cc:343] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: e0968c2f25b64f6685f38a999ee9d60e
I20251027 21:08:02.624545 9710 leader_election.cc:304] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:02.625118 10029 tablet_bootstrap.cc:654] T c4996d8fae094db8a96599b9057f1649 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.626108 9742 tablet_service.cc:1505] Processing CreateTablet for tablet eaa869a2e41b4f83978b7e51c4717edd (DEFAULT_TABLE table=default.Table_066671f8fa8b403386935bcda7a8d095 [id=24bdea549f024941b08739d57ab2b231]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.626226 9890 tablet_service.cc:1505] Processing CreateTablet for tablet eaa869a2e41b4f83978b7e51c4717edd (DEFAULT_TABLE table=default.Table_066671f8fa8b403386935bcda7a8d095 [id=24bdea549f024941b08739d57ab2b231]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.626547 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaa869a2e41b4f83978b7e51c4717edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.626591 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaa869a2e41b4f83978b7e51c4717edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.626776 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 1ba47e34f5464b47bc32b2776f37d97d (DEFAULT_TABLE table=default.Table_8c2f4fc52c37435fb5a25817b1545503 [id=a42eb3665bfc426a8d24f698c1276893]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.627138 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ba47e34f5464b47bc32b2776f37d97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.630652 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 32f80614e30d41128f9fb2e0ea165223 (table default.Table_a4ae12c512d645e58eadc2daa1bb7b0a [id=e1f4a77dfc0a497080c3c0ad4fe2e370]) successfully deleted
I20251027 21:08:02.630816 10040 ts_tablet_manager.cc:1916] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.630908 10829 raft_consensus.cc:2749] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:02.631728 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 0f90e09e74b6495db1aa2df85de21181 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f90e09e74b6495db1aa2df85de21181 already in progress: creating tablet
I20251027 21:08:02.632704 10041 tablet_replica.cc:333] T c3b9a461567a48fcb4af514ecdf618b4 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:02.633108 10041 raft_consensus.cc:2243] T c3b9a461567a48fcb4af514ecdf618b4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.633940 10041 raft_consensus.cc:2272] T c3b9a461567a48fcb4af514ecdf618b4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.634511 10043 tablet_replica.cc:333] T 9882b234e79940088be5f604c41576b7 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:02.634671 10029 tablet_bootstrap.cc:492] T c4996d8fae094db8a96599b9057f1649 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.634960 10029 ts_tablet_manager.cc:1403] T c4996d8fae094db8a96599b9057f1649 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
I20251027 21:08:02.635442 10041 ts_tablet_manager.cc:1916] T c3b9a461567a48fcb4af514ecdf618b4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.636437 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet 533f34d4872444278477abcce89da9ec with delete_type TABLET_DATA_DELETED (replaced by tablet 5054bbc3e27243298b4df6b0a87af8be) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.634738 10043 raft_consensus.cc:2243] T 9882b234e79940088be5f604c41576b7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.636730 10043 raft_consensus.cc:2272] T 9882b234e79940088be5f604c41576b7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.637377 10043 ts_tablet_manager.cc:1916] T 9882b234e79940088be5f604c41576b7 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:02.638548 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0e64d21abef342de86555e876f020787 consensus state has no leader
I20251027 21:08:02.637156 10032 tablet_bootstrap.cc:654] T 4debc3f13f3a42939b6b83f6523aab7c P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.642122 10800 raft_consensus.cc:493] T 6600774e4c7b48479a2fdb02614ba505 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:02.642280 10800 raft_consensus.cc:515] T 6600774e4c7b48479a2fdb02614ba505 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.642977 10800 leader_election.cc:290] T 6600774e4c7b48479a2fdb02614ba505 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:02.645287 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6600774e4c7b48479a2fdb02614ba505" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:02.645711 9860 leader_election.cc:343] T 6600774e4c7b48479a2fdb02614ba505 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:02.646167 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6600774e4c7b48479a2fdb02614ba505" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:02.646545 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_bae8db615e424af398f1f9baec11cb76" } new_table_name: "default.Table_0a6d8856e582400da9228702189fc7a9" modify_external_catalogs: true
W20251027 21:08:02.646713 9859 leader_election.cc:343] T 6600774e4c7b48479a2fdb02614ba505 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:02.646821 9859 leader_election.cc:304] T 6600774e4c7b48479a2fdb02614ba505 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:02.647395 10800 raft_consensus.cc:2749] T 6600774e4c7b48479a2fdb02614ba505 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:02.647522 10800 raft_consensus.cc:493] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:02.647636 10800 raft_consensus.cc:515] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.648139 10800 leader_election.cc:290] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:02.648761 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67bb12de9f044c73ae9ab26d10498c90" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:02.648865 9739 tablet_service.cc:1505] Processing CreateTablet for tablet 49f5e255ff574ef39476e57080320a11 (DEFAULT_TABLE table=default.Table_e86d94359267418f85f5fda8a5ff79d7 [id=2f3ca82c248f440b9a5e4c3df729812d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.649230 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49f5e255ff574ef39476e57080320a11. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:02.651327 9859 leader_election.cc:343] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:02.651682 10639 fs_manager.cc:714] Time spent opening directory manager: real 0.063s user 0.004s sys 0.002s
I20251027 21:08:02.655198 10029 raft_consensus.cc:359] T c4996d8fae094db8a96599b9057f1649 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.655463 10029 raft_consensus.cc:385] T c4996d8fae094db8a96599b9057f1649 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.655582 10029 raft_consensus.cc:740] T c4996d8fae094db8a96599b9057f1649 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.655808 10031 tablet_bootstrap.cc:492] T fe24af9838954a9bb680474800642e62 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:02.655936 10031 ts_tablet_manager.cc:1403] T fe24af9838954a9bb680474800642e62 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.000s
I20251027 21:08:02.656075 10029 consensus_queue.cc:260] T c4996d8fae094db8a96599b9057f1649 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.658833 10029 ts_tablet_manager.cc:1434] T c4996d8fae094db8a96599b9057f1649 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.023s user 0.001s sys 0.002s
I20251027 21:08:02.659229 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 49f5e255ff574ef39476e57080320a11 (DEFAULT_TABLE table=default.Table_e86d94359267418f85f5fda8a5ff79d7 [id=2f3ca82c248f440b9a5e4c3df729812d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.659303 10029 tablet_bootstrap.cc:492] T c5f2e60caa824b5db2330b423221873c P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:02.659646 10829 raft_consensus.cc:493] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:02.659756 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49f5e255ff574ef39476e57080320a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.659761 10829 raft_consensus.cc:515] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.660255 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67bb12de9f044c73ae9ab26d10498c90" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:02.660326 10829 leader_election.cc:290] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:02.660655 9860 leader_election.cc:343] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:02.660751 9860 leader_election.cc:304] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:02.661078 10873 raft_consensus.cc:2749] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:02.661711 10029 tablet_bootstrap.cc:654] T c5f2e60caa824b5db2330b423221873c P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:02.661701 9711 leader_election.cc:343] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: abc7458fcfb94985acf76d77c0292b08
W20251027 21:08:02.662076 9710 leader_election.cc:343] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: abc7458fcfb94985acf76d77c0292b08
I20251027 21:08:02.662182 9710 leader_election.cc:304] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:02.662603 10829 raft_consensus.cc:2749] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:02.667006 10032 tablet_bootstrap.cc:492] T 4debc3f13f3a42939b6b83f6523aab7c P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:02.667138 10032 ts_tablet_manager.cc:1403] T 4debc3f13f3a42939b6b83f6523aab7c P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.003s
I20251027 21:08:02.667431 10040 ts_tablet_manager.cc:1929] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.667548 10040 log.cc:1199] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/6f1129eecfb54448965222269d9c32b7
I20251027 21:08:02.667836 10040 ts_tablet_manager.cc:1950] T 6f1129eecfb54448965222269d9c32b7 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:02.667835 10032 raft_consensus.cc:359] T 4debc3f13f3a42939b6b83f6523aab7c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.667980 10032 raft_consensus.cc:385] T 4debc3f13f3a42939b6b83f6523aab7c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.668033 10032 raft_consensus.cc:740] T 4debc3f13f3a42939b6b83f6523aab7c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.668099 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 8885465e81444b468b96955f0e44e9d2 (DEFAULT_TABLE table=default.Table_0f287ac2ad254338bd5151726cc40821 [id=c78c21c0d3fe49cfaf9b8c9c9eff3926]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.668232 10032 consensus_queue.cc:260] T 4debc3f13f3a42939b6b83f6523aab7c P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.668431 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8885465e81444b468b96955f0e44e9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.668335 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 6f1129eecfb54448965222269d9c32b7 (table default.Table_f16bc07fcb4c48b7969d13b8341aefdc [id=b5f4e2be1a5e42f58d98b64b8700ef1a]) successfully deleted
I20251027 21:08:02.668655 10032 ts_tablet_manager.cc:1434] T 4debc3f13f3a42939b6b83f6523aab7c P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:02.670289 10829 raft_consensus.cc:493] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:02.670396 10829 raft_consensus.cc:515] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.670904 10829 leader_election.cc:290] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:02.671195 10031 raft_consensus.cc:359] T fe24af9838954a9bb680474800642e62 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.671296 10043 ts_tablet_manager.cc:1929] T 9882b234e79940088be5f604c41576b7 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.671331 10031 raft_consensus.cc:385] T fe24af9838954a9bb680474800642e62 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.671377 10043 log.cc:1199] T 9882b234e79940088be5f604c41576b7 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/9882b234e79940088be5f604c41576b7
I20251027 21:08:02.671394 10031 raft_consensus.cc:740] T fe24af9838954a9bb680474800642e62 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.671607 10043 ts_tablet_manager.cc:1950] T 9882b234e79940088be5f604c41576b7 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:02.671574 10031 consensus_queue.cc:260] T fe24af9838954a9bb680474800642e62 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.671931 10032 tablet_bootstrap.cc:492] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:08:02.671968 9710 leader_election.cc:343] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 4e8c83b501d8426fa0cad6a89f9a0b9b
I20251027 21:08:02.673434 10032 tablet_bootstrap.cc:654] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:02.676311 9711 leader_election.cc:343] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 4e8c83b501d8426fa0cad6a89f9a0b9b
I20251027 21:08:02.676425 9711 leader_election.cc:304] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:02.676743 10829 raft_consensus.cc:2749] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:02.677135 10031 ts_tablet_manager.cc:1434] T fe24af9838954a9bb680474800642e62 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20251027 21:08:02.678021 10041 ts_tablet_manager.cc:1929] T c3b9a461567a48fcb4af514ecdf618b4 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.678335 10041 log.cc:1199] T c3b9a461567a48fcb4af514ecdf618b4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/c3b9a461567a48fcb4af514ecdf618b4
I20251027 21:08:02.678807 10041 ts_tablet_manager.cc:1950] T c3b9a461567a48fcb4af514ecdf618b4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:02.679021 10031 tablet_bootstrap.cc:492] T c5f2e60caa824b5db2330b423221873c P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:02.680678 10031 tablet_bootstrap.cc:654] T c5f2e60caa824b5db2330b423221873c P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.683091 10043 tablet_replica.cc:333] T adaa4f66bbab491f8030d439d85cc7d1 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:02.683344 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 9882b234e79940088be5f604c41576b7 (table default.Table_e0dee9209ca14dd98065b0384855d953 [id=92995507b7d2462f91b1a16434d3ac06]) successfully deleted
I20251027 21:08:02.683357 10043 raft_consensus.cc:2243] T adaa4f66bbab491f8030d439d85cc7d1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.683602 10043 raft_consensus.cc:2272] T adaa4f66bbab491f8030d439d85cc7d1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.685786 10829 raft_consensus.cc:493] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:02.685909 10829 raft_consensus.cc:515] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.686509 10829 leader_election.cc:290] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:02.687295 10043 ts_tablet_manager.cc:1916] T adaa4f66bbab491f8030d439d85cc7d1 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.687465 10029 tablet_bootstrap.cc:492] T c5f2e60caa824b5db2330b423221873c P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
W20251027 21:08:02.687510 9710 leader_election.cc:343] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 9882b234e79940088be5f604c41576b7
I20251027 21:08:02.687664 10029 ts_tablet_manager.cc:1403] T c5f2e60caa824b5db2330b423221873c P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.001s
I20251027 21:08:02.688648 10029 raft_consensus.cc:359] T c5f2e60caa824b5db2330b423221873c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.684509 10032 tablet_bootstrap.cc:492] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:02.689123 10032 ts_tablet_manager.cc:1403] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.004s
I20251027 21:08:02.689859 10032 raft_consensus.cc:359] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.690091 10032 raft_consensus.cc:385] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.690227 10032 raft_consensus.cc:740] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.690574 10032 consensus_queue.cc:260] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.691407 10032 ts_tablet_manager.cc:1434] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 21:08:02.691927 10032 tablet_bootstrap.cc:492] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:02.692756 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet fe24af9838954a9bb680474800642e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.693511 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:02.693718 10032 tablet_bootstrap.cc:654] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:02.694396 9711 leader_election.cc:343] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 9882b234e79940088be5f604c41576b7
I20251027 21:08:02.694509 9711 leader_election.cc:304] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:02.695549 10032 tablet_bootstrap.cc:492] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:02.695665 10032 ts_tablet_manager.cc:1403] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251027 21:08:02.695964 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_df7c27fe46e74ecda6bd9e58bc24bf72" } new_table_name: "default.Table_d52fc4f0f55b4bab9edbfff9297f8728" modify_external_catalogs: true
I20251027 21:08:02.696233 10829 raft_consensus.cc:2749] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:02.696367 10032 raft_consensus.cc:359] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.696521 10032 raft_consensus.cc:385] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.696563 10032 raft_consensus.cc:740] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.697036 10029 raft_consensus.cc:385] T c5f2e60caa824b5db2330b423221873c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.698177 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet c3b9a461567a48fcb4af514ecdf618b4 (table default.Table_fc2da109d1934a3fbe14ae51b20e771d [id=56f8db49ad8c4670a73a29bf4ed613e9]) successfully deleted
I20251027 21:08:02.698227 10029 raft_consensus.cc:740] T c5f2e60caa824b5db2330b423221873c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.698451 10029 consensus_queue.cc:260] T c5f2e60caa824b5db2330b423221873c P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.701395 10029 ts_tablet_manager.cc:1434] T c5f2e60caa824b5db2330b423221873c P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20251027 21:08:02.703510 10873 raft_consensus.cc:493] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:02.703677 10873 raft_consensus.cc:515] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.704578 10873 leader_election.cc:290] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:02.704820 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 4debc3f13f3a42939b6b83f6523aab7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:02.705085 10041 tablet_replica.cc:333] T 4debc3f13f3a42939b6b83f6523aab7c P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:02.705400 10041 raft_consensus.cc:2243] T 4debc3f13f3a42939b6b83f6523aab7c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.705338 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 8885465e81444b468b96955f0e44e9d2 (DEFAULT_TABLE table=default.Table_0f287ac2ad254338bd5151726cc40821 [id=c78c21c0d3fe49cfaf9b8c9c9eff3926]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.705801 10041 raft_consensus.cc:2272] T 4debc3f13f3a42939b6b83f6523aab7c P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.706419 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca477937278f45b4b115fec715f32a92" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:02.706902 9859 leader_election.cc:343] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:02.707454 10043 ts_tablet_manager.cc:1929] T adaa4f66bbab491f8030d439d85cc7d1 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.707603 10043 log.cc:1199] T adaa4f66bbab491f8030d439d85cc7d1 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/adaa4f66bbab491f8030d439d85cc7d1
I20251027 21:08:02.707604 10040 tablet_replica.cc:333] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:02.707746 10041 ts_tablet_manager.cc:1916] T 4debc3f13f3a42939b6b83f6523aab7c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.707780 10040 raft_consensus.cc:2243] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.707926 10040 raft_consensus.cc:2272] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.708119 10043 ts_tablet_manager.cc:1950] T adaa4f66bbab491f8030d439d85cc7d1 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:02.708395 10040 ts_tablet_manager.cc:1916] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.708420 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca477937278f45b4b115fec715f32a92" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:02.709100 9860 leader_election.cc:343] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:02.709192 9860 leader_election.cc:304] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:02.709549 10800 raft_consensus.cc:2749] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:02.711124 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8885465e81444b468b96955f0e44e9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.696703 10032 consensus_queue.cc:260] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.715343 10032 ts_tablet_manager.cc:1434] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20251027 21:08:02.715606 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet adaa4f66bbab491f8030d439d85cc7d1 (table default.Table_3f5e27698f4c4da88d9bacc9dd719304 [id=d174e821ea8641fc8b6658d5c61c7e8a]) successfully deleted
I20251027 21:08:02.716274 10032 tablet_bootstrap.cc:492] T 04c579bb008448e9baa841135b6dbc61 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:02.717515 10029 tablet_bootstrap.cc:492] T 648f506ec87c4fb8a3f6d4591873dbb8 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:02.720078 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 43611e33512b49d58485cf15de788be7
I20251027 21:08:02.720242 10032 tablet_bootstrap.cc:654] T 04c579bb008448e9baa841135b6dbc61 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.720626 9303 catalog_manager.cc:6719] ReplaceTablet: tablet 43611e33512b49d58485cf15de788be7 deleted and replaced by tablet 5e372257733148199d2dbe70d0887cbb
I20251027 21:08:02.721597 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 43611e33512b49d58485cf15de788be7 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e372257733148199d2dbe70d0887cbb) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:02.722384 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 43611e33512b49d58485cf15de788be7 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e372257733148199d2dbe70d0887cbb) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.723177 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 43611e33512b49d58485cf15de788be7 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e372257733148199d2dbe70d0887cbb) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:02.723939 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 0f90e09e74b6495db1aa2df85de21181 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f90e09e74b6495db1aa2df85de21181 already in progress: creating tablet
I20251027 21:08:02.724280 10043 tablet_replica.cc:333] T 3103c8e5a7c744be90be097179904ba3 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:02.724474 10043 raft_consensus.cc:2243] T 3103c8e5a7c744be90be097179904ba3 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.724622 10043 raft_consensus.cc:2272] T 3103c8e5a7c744be90be097179904ba3 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.725858 9739 tablet_service.cc:1505] Processing CreateTablet for tablet 8885465e81444b468b96955f0e44e9d2 (DEFAULT_TABLE table=default.Table_0f287ac2ad254338bd5151726cc40821 [id=c78c21c0d3fe49cfaf9b8c9c9eff3926]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.726284 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8885465e81444b468b96955f0e44e9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.727351 10875 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 21:08:02.727948 10029 tablet_bootstrap.cc:654] T 648f506ec87c4fb8a3f6d4591873dbb8 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.728746 10043 ts_tablet_manager.cc:1916] T 3103c8e5a7c744be90be097179904ba3 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.731278 10032 tablet_bootstrap.cc:492] T 04c579bb008448e9baa841135b6dbc61 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:02.731417 10032 ts_tablet_manager.cc:1403] T 04c579bb008448e9baa841135b6dbc61 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.001s
I20251027 21:08:02.732622 10032 raft_consensus.cc:359] T 04c579bb008448e9baa841135b6dbc61 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.732782 10032 raft_consensus.cc:385] T 04c579bb008448e9baa841135b6dbc61 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.732831 10032 raft_consensus.cc:740] T 04c579bb008448e9baa841135b6dbc61 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.733026 10032 consensus_queue.cc:260] T 04c579bb008448e9baa841135b6dbc61 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.733366 9303 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 9fde4fd91943498d8888d1783005e492 (table default.Table_87ac731a20bf4f6bbce8d82c1ad3bd97 [id=3a0c8b19df934d1580349b061fe0e8f5]). Expected version 1 got 0
I20251027 21:08:02.736084 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_563c35f040434ab5bd6eee6981ddb400" } new_table_name: "default.Table_da6055c495724b4783a105c4fac610cf" modify_external_catalogs: true
I20251027 21:08:02.741102 10029 tablet_bootstrap.cc:492] T 648f506ec87c4fb8a3f6d4591873dbb8 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.741192 10032 ts_tablet_manager.cc:1434] T 04c579bb008448e9baa841135b6dbc61 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251027 21:08:02.741451 10032 tablet_bootstrap.cc:492] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:02.741665 10029 ts_tablet_manager.cc:1403] T 648f506ec87c4fb8a3f6d4591873dbb8 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20251027 21:08:02.742285 10041 ts_tablet_manager.cc:1929] T 4debc3f13f3a42939b6b83f6523aab7c P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.742388 10041 log.cc:1199] T 4debc3f13f3a42939b6b83f6523aab7c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/4debc3f13f3a42939b6b83f6523aab7c
I20251027 21:08:02.742648 10041 ts_tablet_manager.cc:1950] T 4debc3f13f3a42939b6b83f6523aab7c P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:02.743158 10029 raft_consensus.cc:359] T 648f506ec87c4fb8a3f6d4591873dbb8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.743613 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 4debc3f13f3a42939b6b83f6523aab7c (table default.Table_4bd57498972a413ca2bb7de2cd588e8b [id=5ebbc228b61046f8aef2d6fe13f12e57]) successfully deleted
I20251027 21:08:02.743863 10029 raft_consensus.cc:385] T 648f506ec87c4fb8a3f6d4591873dbb8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.743929 10029 raft_consensus.cc:740] T 648f506ec87c4fb8a3f6d4591873dbb8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
W20251027 21:08:02.744033 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 43611e33512b49d58485cf15de788be7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43611e33512b49d58485cf15de788be7 already in progress: creating tablet
I20251027 21:08:02.744148 10029 consensus_queue.cc:260] T 648f506ec87c4fb8a3f6d4591873dbb8 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.748138 10029 ts_tablet_manager.cc:1434] T 648f506ec87c4fb8a3f6d4591873dbb8 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251027 21:08:02.751282 10029 tablet_bootstrap.cc:492] T a9bf3a093be8498c9dd5511b53436b16 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:02.751784 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 312dc1e8caea4627b268536a6ce6344d (DEFAULT_TABLE table=default.Table_fdca4795c6d042ec80cf84a378acb979 [id=a92ec852c916461eb59fba87bd1c81eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.752221 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 312dc1e8caea4627b268536a6ce6344d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.754233 10639 fs_manager.cc:730] Time spent opening block manager: real 0.100s user 0.006s sys 0.002s
I20251027 21:08:02.755138 10029 tablet_bootstrap.cc:654] T a9bf3a093be8498c9dd5511b53436b16 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.754463 10639 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data,/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/wal
uuid: "6e86ca5e83dd4df5b72f045e7b787e81"
format_stamp: "Formatted at 2025-10-27 21:07:44 on dist-test-slave-r942"
I20251027 21:08:02.755570 10639 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 21:08:02.759123 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 312dc1e8caea4627b268536a6ce6344d (DEFAULT_TABLE table=default.Table_fdca4795c6d042ec80cf84a378acb979 [id=a92ec852c916461eb59fba87bd1c81eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.759492 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 312dc1e8caea4627b268536a6ce6344d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.762423 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 49f5e255ff574ef39476e57080320a11 (DEFAULT_TABLE table=default.Table_e86d94359267418f85f5fda8a5ff79d7 [id=2f3ca82c248f440b9a5e4c3df729812d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.762761 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49f5e255ff574ef39476e57080320a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.764302 10043 ts_tablet_manager.cc:1929] T 3103c8e5a7c744be90be097179904ba3 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.764407 10043 log.cc:1199] T 3103c8e5a7c744be90be097179904ba3 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/3103c8e5a7c744be90be097179904ba3
I20251027 21:08:02.764668 10043 ts_tablet_manager.cc:1950] T 3103c8e5a7c744be90be097179904ba3 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:02.765657 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 3103c8e5a7c744be90be097179904ba3 (table default.Table_c04ff0f25ba04d25958346fc631b2802 [id=a6a32b47c2354a2d9535ff2b48d506de]) successfully deleted
W20251027 21:08:02.765951 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 43611e33512b49d58485cf15de788be7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43611e33512b49d58485cf15de788be7 already in progress: creating tablet
I20251027 21:08:02.767267 10029 tablet_bootstrap.cc:492] T a9bf3a093be8498c9dd5511b53436b16 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.767382 10029 ts_tablet_manager.cc:1403] T a9bf3a093be8498c9dd5511b53436b16 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.005s
I20251027 21:08:02.767903 9311 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet aaa8910d9fdb4146945621ec4ebf969e (table default.Table_d8da07f0725347338012d992737c5d04 [id=bb676562b97f48088767941239cfbf33]). Expected version 2 got 0
I20251027 21:08:02.768805 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7b39d3d222b7499fa27b4230bc87dc5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:02.768882 10029 raft_consensus.cc:359] T a9bf3a093be8498c9dd5511b53436b16 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.769019 10029 raft_consensus.cc:385] T a9bf3a093be8498c9dd5511b53436b16 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.769071 10029 raft_consensus.cc:740] T a9bf3a093be8498c9dd5511b53436b16 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.769258 10029 consensus_queue.cc:260] T a9bf3a093be8498c9dd5511b53436b16 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:02.769429 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b39d3d222b7499fa27b4230bc87dc5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.769780 10032 tablet_bootstrap.cc:654] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.773509 10029 ts_tablet_manager.cc:1434] T a9bf3a093be8498c9dd5511b53436b16 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251027 21:08:02.773710 10040 ts_tablet_manager.cc:1929] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.774212 10040 log.cc:1199] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/92a2393cff8044028c86c8cd2bccd930
I20251027 21:08:02.774649 10040 ts_tablet_manager.cc:1950] T 92a2393cff8044028c86c8cd2bccd930 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:02.775928 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet eec11e8fcb60425fb7fd742b6051fc2d
I20251027 21:08:02.776994 10029 tablet_bootstrap.cc:492] T 84b9de82d55447bba2a897edb683e5c5 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:02.778587 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet eec11e8fcb60425fb7fd742b6051fc2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.778846 10029 tablet_bootstrap.cc:654] T 84b9de82d55447bba2a897edb683e5c5 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.780462 9888 tablet_service.cc:1552] Processing DeleteTablet for tablet eec11e8fcb60425fb7fd742b6051fc2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:08:02.782529 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet eec11e8fcb60425fb7fd742b6051fc2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eec11e8fcb60425fb7fd742b6051fc2d already in progress: creating tablet
I20251027 21:08:02.784523 10032 tablet_bootstrap.cc:492] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:02.784628 10032 ts_tablet_manager.cc:1403] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.004s
I20251027 21:08:02.785264 10032 raft_consensus.cc:359] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.785424 10032 raft_consensus.cc:385] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.780308 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet eec11e8fcb60425fb7fd742b6051fc2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.785467 10032 raft_consensus.cc:740] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.785619 10032 consensus_queue.cc:260] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:02.785655 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet eec11e8fcb60425fb7fd742b6051fc2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eec11e8fcb60425fb7fd742b6051fc2d already in progress: creating tablet
I20251027 21:08:02.786393 10032 ts_tablet_manager.cc:1434] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:02.789101 10032 tablet_bootstrap.cc:492] T 59b6818d29624da5a49dd1a7d20994c6 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:02.790903 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 92a2393cff8044028c86c8cd2bccd930 (table default.Table_1d766c40686340d59cc4680fbf8a2dbb [id=e5fba763a46b488cb19c738cee468ad1]) successfully deleted
I20251027 21:08:02.790997 10032 tablet_bootstrap.cc:654] T 59b6818d29624da5a49dd1a7d20994c6 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.791317 10040 tablet_replica.cc:333] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:02.791498 10040 raft_consensus.cc:2243] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251027 21:08:02.791519 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 431313b4c80940239536aea40721b635 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 431313b4c80940239536aea40721b635 already in progress: creating tablet
I20251027 21:08:02.791638 10040 raft_consensus.cc:2272] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:02.791731 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 33c5bc33130a424fb63e3f92a2cc49fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33c5bc33130a424fb63e3f92a2cc49fc already in progress: creating tablet
W20251027 21:08:02.791946 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 0b4bf2b20e7c416aad123c6f96355cac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b4bf2b20e7c416aad123c6f96355cac already in progress: creating tablet
I20251027 21:08:02.792151 10040 ts_tablet_manager.cc:1916] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:02.792168 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 1ea10027bc7246e98092b894241d9d61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ea10027bc7246e98092b894241d9d61 already in progress: creating tablet
I20251027 21:08:02.792923 10029 tablet_bootstrap.cc:492] T 84b9de82d55447bba2a897edb683e5c5 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.793042 10029 ts_tablet_manager.cc:1403] T 84b9de82d55447bba2a897edb683e5c5 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.003s
I20251027 21:08:02.793674 10029 raft_consensus.cc:359] T 84b9de82d55447bba2a897edb683e5c5 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.793833 10029 raft_consensus.cc:385] T 84b9de82d55447bba2a897edb683e5c5 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.793892 10029 raft_consensus.cc:740] T 84b9de82d55447bba2a897edb683e5c5 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.794083 10029 consensus_queue.cc:260] T 84b9de82d55447bba2a897edb683e5c5 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.797837 10029 ts_tablet_manager.cc:1434] T 84b9de82d55447bba2a897edb683e5c5 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251027 21:08:02.799741 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 784df834432244e8812d1802f9e1a1bc (DEFAULT_TABLE table=default.Table_f5602dbcd14b4fb5ae7190b6a0bd65bc [id=078add1b91ad4cbfbe2215fe89b2334e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.800056 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 784df834432244e8812d1802f9e1a1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.800365 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 784df834432244e8812d1802f9e1a1bc (DEFAULT_TABLE table=default.Table_f5602dbcd14b4fb5ae7190b6a0bd65bc [id=078add1b91ad4cbfbe2215fe89b2334e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.800817 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 784df834432244e8812d1802f9e1a1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.804097 10032 tablet_bootstrap.cc:492] T 59b6818d29624da5a49dd1a7d20994c6 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:02.804215 10032 ts_tablet_manager.cc:1403] T 59b6818d29624da5a49dd1a7d20994c6 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
I20251027 21:08:02.804867 10032 raft_consensus.cc:359] T 59b6818d29624da5a49dd1a7d20994c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.805032 10032 raft_consensus.cc:385] T 59b6818d29624da5a49dd1a7d20994c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.805083 10032 raft_consensus.cc:740] T 59b6818d29624da5a49dd1a7d20994c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.805305 10032 consensus_queue.cc:260] T 59b6818d29624da5a49dd1a7d20994c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.807927 10032 ts_tablet_manager.cc:1434] T 59b6818d29624da5a49dd1a7d20994c6 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251027 21:08:02.808646 10032 tablet_bootstrap.cc:492] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:02.808974 10859 raft_consensus.cc:493] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:02.809106 10859 raft_consensus.cc:515] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.809665 10859 leader_election.cc:290] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:02.809772 10031 tablet_bootstrap.cc:492] T c5f2e60caa824b5db2330b423221873c P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:02.809882 10031 ts_tablet_manager.cc:1403] T c5f2e60caa824b5db2330b423221873c P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.131s user 0.001s sys 0.009s
I20251027 21:08:02.809983 10032 tablet_bootstrap.cc:654] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.810277 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45568792d7954b498581ce69fb611948" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:02.810693 9566 leader_election.cc:343] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:02.811245 10031 raft_consensus.cc:359] T c5f2e60caa824b5db2330b423221873c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.811401 10031 raft_consensus.cc:385] T c5f2e60caa824b5db2330b423221873c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.811453 10031 raft_consensus.cc:740] T c5f2e60caa824b5db2330b423221873c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.811383 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8c2f4fc52c37435fb5a25817b1545503" } new_table_name: "default.Table_0d7161154d094df4a95cdf85beffb72f" modify_external_catalogs: true
I20251027 21:08:02.811692 10031 consensus_queue.cc:260] T c5f2e60caa824b5db2330b423221873c P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.811942 10029 tablet_bootstrap.cc:492] T d872cf3ef39042ff9a615b55e7195afa P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:02.816174 10029 tablet_bootstrap.cc:654] T d872cf3ef39042ff9a615b55e7195afa P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.817600 10031 ts_tablet_manager.cc:1434] T c5f2e60caa824b5db2330b423221873c P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20251027 21:08:02.820437 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 9d19e9f63acf4c8c894af38a02225fc6 (DEFAULT_TABLE table=default.Table_9cd91730e4c4415e9f65d748f4592366 [id=abfccf9280c64ed19771c6e9c678673a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.820760 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d19e9f63acf4c8c894af38a02225fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.821626 9748 tablet_service.cc:1505] Processing CreateTablet for tablet b748f59a56c54c2b850f5336a1d61c26 (DEFAULT_TABLE table=default.Table_937452027956409d907e8d44579e9d66 [id=983e5d91d7994da8ad3e4231d799e80a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.821974 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b748f59a56c54c2b850f5336a1d61c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.824075 9745 tablet_service.cc:1505] Processing CreateTablet for tablet ada34d5086484ef18e2050ac05b71725 (DEFAULT_TABLE table=default.Table_9e349c15adbf444f96f4d83d3acb5990 [id=04c44da04ebc4b828c688eeb3bbd2e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.824350 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ada34d5086484ef18e2050ac05b71725. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.826356 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 6bbee5f1fb704709b6f166a153e69808 (DEFAULT_TABLE table=default.Table_b0bf4b9e330b4ec48becdc293dcef72a [id=be5aa962c8004df9aa16483c6eeab83f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.826651 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bbee5f1fb704709b6f166a153e69808. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.833194 10031 tablet_bootstrap.cc:492] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:02.834649 10031 tablet_bootstrap.cc:654] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.838996 10029 tablet_bootstrap.cc:492] T d872cf3ef39042ff9a615b55e7195afa P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.839211 10029 ts_tablet_manager.cc:1403] T d872cf3ef39042ff9a615b55e7195afa P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.005s
I20251027 21:08:02.840382 10040 ts_tablet_manager.cc:1929] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.840485 10040 log.cc:1199] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/4e8c83b501d8426fa0cad6a89f9a0b9b
I20251027 21:08:02.840755 10040 ts_tablet_manager.cc:1950] T 4e8c83b501d8426fa0cad6a89f9a0b9b P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:02.840876 10029 raft_consensus.cc:359] T d872cf3ef39042ff9a615b55e7195afa P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.841060 10029 raft_consensus.cc:385] T d872cf3ef39042ff9a615b55e7195afa P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.841140 10029 raft_consensus.cc:740] T d872cf3ef39042ff9a615b55e7195afa P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.841279 10040 tablet_replica.cc:333] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:02.841238 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 4e8c83b501d8426fa0cad6a89f9a0b9b (table default.Table_4f6664b7160e4549a50589454ff10313 [id=67b71acdadc64445bb2492e5d424cb2d]) successfully deleted
I20251027 21:08:02.841451 10040 raft_consensus.cc:2243] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.841427 10029 consensus_queue.cc:260] T d872cf3ef39042ff9a615b55e7195afa P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.841594 10040 raft_consensus.cc:2272] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:02.841610 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet d8691aa22fe84cf29893ed4e79d156cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8691aa22fe84cf29893ed4e79d156cc already in progress: creating tablet
I20251027 21:08:02.842049 10040 ts_tablet_manager.cc:1916] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.843900 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 43b5cef5fdcd4d2fa89875aa6d0efcd0
I20251027 21:08:02.845193 9600 tablet_service.cc:1505] Processing CreateTablet for tablet b748f59a56c54c2b850f5336a1d61c26 (DEFAULT_TABLE table=default.Table_937452027956409d907e8d44579e9d66 [id=983e5d91d7994da8ad3e4231d799e80a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.845359 9609 tablet_service.cc:1505] Processing CreateTablet for tablet ada34d5086484ef18e2050ac05b71725 (DEFAULT_TABLE table=default.Table_9e349c15adbf444f96f4d83d3acb5990 [id=04c44da04ebc4b828c688eeb3bbd2e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.845572 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b748f59a56c54c2b850f5336a1d61c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.845191 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 9d19e9f63acf4c8c894af38a02225fc6 (DEFAULT_TABLE table=default.Table_9cd91730e4c4415e9f65d748f4592366 [id=abfccf9280c64ed19771c6e9c678673a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.845887 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d19e9f63acf4c8c894af38a02225fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.846338 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 6bbee5f1fb704709b6f166a153e69808 (DEFAULT_TABLE table=default.Table_b0bf4b9e330b4ec48becdc293dcef72a [id=be5aa962c8004df9aa16483c6eeab83f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.846614 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bbee5f1fb704709b6f166a153e69808. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.850386 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ada34d5086484ef18e2050ac05b71725. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.850634 9611 tablet_service.cc:1552] Processing DeleteTablet for tablet 43b5cef5fdcd4d2fa89875aa6d0efcd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:02.851281 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 43b5cef5fdcd4d2fa89875aa6d0efcd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43b5cef5fdcd4d2fa89875aa6d0efcd0 already in progress: creating tablet
I20251027 21:08:02.854475 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 43b5cef5fdcd4d2fa89875aa6d0efcd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.860244 9309 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 94f2819dd0394530ae93754bdd1e482f (table default.Table_090fa7c40bdb40d09777fd8458acd6b2 [id=63147f61013c4d6dbc96fd25769b6136]). Expected version 2 got 0
I20251027 21:08:02.860688 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 2f641bc4dbed4c0abd0a55b253352507
I20251027 21:08:02.861320 10029 ts_tablet_manager.cc:1434] T d872cf3ef39042ff9a615b55e7195afa P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.022s user 0.000s sys 0.001s
I20251027 21:08:02.863193 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f641bc4dbed4c0abd0a55b253352507 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.865757 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f641bc4dbed4c0abd0a55b253352507 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.866057 10029 tablet_bootstrap.cc:492] T 630ae4d66f99474382814cc32cc60ea0 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
W20251027 21:08:02.866364 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 2f641bc4dbed4c0abd0a55b253352507 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f641bc4dbed4c0abd0a55b253352507 already in progress: creating tablet
I20251027 21:08:02.868096 10029 tablet_bootstrap.cc:654] T 630ae4d66f99474382814cc32cc60ea0 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.878151 9748 tablet_service.cc:1505] Processing CreateTablet for tablet dbf46e4ce7714398b2c2019314e44578 (DEFAULT_TABLE table=default.Table_56d94ef7571b42b4ba4e66f6ce476ead [id=aa53ec98c45e4f1db9d9abf1993ff200]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.879706 10032 tablet_bootstrap.cc:492] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:02.879844 10032 ts_tablet_manager.cc:1403] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.071s user 0.012s sys 0.007s
I20251027 21:08:02.880503 10032 raft_consensus.cc:359] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.880656 10032 raft_consensus.cc:385] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.880700 10032 raft_consensus.cc:740] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.880873 10032 consensus_queue.cc:260] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.881294 10032 ts_tablet_manager.cc:1434] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251027 21:08:02.881567 10032 tablet_bootstrap.cc:492] T cef2bfee222f45e6b6f20c6d63ff44d4 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:02.883013 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 016e1aa5266d44b5b35a3f47e8f11a46 (DEFAULT_TABLE table=default.Table_db03a0faacfc4ecfad89dbf48def9140 [id=409b668e9fed4a748a4cb43c5c69551a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.883347 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 016e1aa5266d44b5b35a3f47e8f11a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.883843 10031 tablet_bootstrap.cc:492] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:02.883982 10031 ts_tablet_manager.cc:1403] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.051s user 0.000s sys 0.004s
I20251027 21:08:02.884106 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 016e1aa5266d44b5b35a3f47e8f11a46 (DEFAULT_TABLE table=default.Table_db03a0faacfc4ecfad89dbf48def9140 [id=409b668e9fed4a748a4cb43c5c69551a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.884761 10031 raft_consensus.cc:359] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.884918 10031 raft_consensus.cc:385] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.884989 10031 raft_consensus.cc:740] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.887229 9599 tablet_service.cc:1505] Processing CreateTablet for tablet dbf46e4ce7714398b2c2019314e44578 (DEFAULT_TABLE table=default.Table_56d94ef7571b42b4ba4e66f6ce476ead [id=aa53ec98c45e4f1db9d9abf1993ff200]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.888638 10032 tablet_bootstrap.cc:654] T cef2bfee222f45e6b6f20c6d63ff44d4 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.890002 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbf46e4ce7714398b2c2019314e44578. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.893114 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 070167be7d8d4ddf8da561df14448360 (DEFAULT_TABLE table=default.Table_92cfd00027b045f3b151deec3558ece6 [id=479548b373ef42b4ad8683a9dcfb3ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.893450 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 070167be7d8d4ddf8da561df14448360. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.893981 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_df2d8d09ba3147afb5c9271da5e23d45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:02.894563 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df2d8d09ba3147afb5c9271da5e23d45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.896854 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b29f906d3a774523839fd36ba1f5d2e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:02.897341 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b29f906d3a774523839fd36ba1f5d2e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:02.900058 10032 tablet_bootstrap.cc:492] T cef2bfee222f45e6b6f20c6d63ff44d4 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:02.900200 10032 ts_tablet_manager.cc:1403] T cef2bfee222f45e6b6f20c6d63ff44d4 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20251027 21:08:02.900915 10032 raft_consensus.cc:359] T cef2bfee222f45e6b6f20c6d63ff44d4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:02.901084 10032 raft_consensus.cc:385] T cef2bfee222f45e6b6f20c6d63ff44d4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.901139 10032 raft_consensus.cc:740] T cef2bfee222f45e6b6f20c6d63ff44d4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.901309 10032 consensus_queue.cc:260] T cef2bfee222f45e6b6f20c6d63ff44d4 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:02.901774 10032 ts_tablet_manager.cc:1434] T cef2bfee222f45e6b6f20c6d63ff44d4 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 21:08:02.902004 10032 tablet_bootstrap.cc:492] T f775a7e42bdc4910bdb604b26a28a7f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:02.904722 10032 tablet_bootstrap.cc:654] T f775a7e42bdc4910bdb604b26a28a7f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.905419 9309 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet abd3b8e71d3041b4b9e5f6663841bfcf (table default.Table_fd331f3cfc07425082d58ab5af3c1094 [id=f3e3147411e5488bae105ed755f5b50f]). Expected version 2 got 0
I20251027 21:08:02.905702 9309 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 45568792d7954b498581ce69fb611948 (table default.Table_3e3e0c17a3bf452fbb217c6e3ee8bdcb [id=2eb093cd58e643a58b14e564d5914fbc]). Expected version 1 got 0
I20251027 21:08:02.909111 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet c5f2e60caa824b5db2330b423221873c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:02.909559 10031 consensus_queue.cc:260] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:02.911211 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet a9bf3a093be8498c9dd5511b53436b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.911351 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 84b9de82d55447bba2a897edb683e5c5 with delete_type TABLET_DATA_DELETED (replaced by tablet befdd3a485a1426b8a3adf340d4eae4f) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.911535 10043 tablet_replica.cc:333] T 84b9de82d55447bba2a897edb683e5c5 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:02.911525 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 648f506ec87c4fb8a3f6d4591873dbb8 with delete_type TABLET_DATA_DELETED (replaced by tablet 96a680e8f16343b1908829a67e960b60) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.911707 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet c4996d8fae094db8a96599b9057f1649 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.911720 10043 raft_consensus.cc:2243] T 84b9de82d55447bba2a897edb683e5c5 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.911789 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet d872cf3ef39042ff9a615b55e7195afa with delete_type TABLET_DATA_DELETED (replaced by tablet 6f901626832f4adb948ccb65f8573e86) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.911895 10043 raft_consensus.cc:2272] T 84b9de82d55447bba2a897edb683e5c5 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.911746 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet c5f2e60caa824b5db2330b423221873c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.912743 10043 ts_tablet_manager.cc:1916] T 84b9de82d55447bba2a897edb683e5c5 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.914760 10031 ts_tablet_manager.cc:1434] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.031s user 0.001s sys 0.002s
I20251027 21:08:02.918354 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet b719dc40d7e348608cdfc78401c19824 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.919584 10031 tablet_bootstrap.cc:492] T abd3b8e71d3041b4b9e5f6663841bfcf P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:02.919870 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbf46e4ce7714398b2c2019314e44578. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.920190 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 016e1aa5266d44b5b35a3f47e8f11a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.924886 10043 ts_tablet_manager.cc:1929] T 84b9de82d55447bba2a897edb683e5c5 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.930931 9739 tablet_service.cc:1505] Processing CreateTablet for tablet 070167be7d8d4ddf8da561df14448360 (DEFAULT_TABLE table=default.Table_92cfd00027b045f3b151deec3558ece6 [id=479548b373ef42b4ad8683a9dcfb3ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.931159 10043 log.cc:1199] T 84b9de82d55447bba2a897edb683e5c5 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/84b9de82d55447bba2a897edb683e5c5
I20251027 21:08:02.931347 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 070167be7d8d4ddf8da561df14448360. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.931558 10043 ts_tablet_manager.cc:1950] T 84b9de82d55447bba2a897edb683e5c5 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:02.932482 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 84b9de82d55447bba2a897edb683e5c5 (table default.Table_4755cbfcd1464ec1b6f0b909b99f0c3f [id=6b70bbbf477d4b788b3ffb10c9c7d7c9]) successfully deleted
I20251027 21:08:02.932798 10043 tablet_replica.cc:333] T 648f506ec87c4fb8a3f6d4591873dbb8 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:02.933010 10043 raft_consensus.cc:2243] T 648f506ec87c4fb8a3f6d4591873dbb8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.933166 10043 raft_consensus.cc:2272] T 648f506ec87c4fb8a3f6d4591873dbb8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.934883 10043 ts_tablet_manager.cc:1916] T 648f506ec87c4fb8a3f6d4591873dbb8 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.943928 10029 tablet_bootstrap.cc:492] T 630ae4d66f99474382814cc32cc60ea0 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.944044 10029 ts_tablet_manager.cc:1403] T 630ae4d66f99474382814cc32cc60ea0 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.078s user 0.004s sys 0.005s
I20251027 21:08:02.944670 10029 raft_consensus.cc:359] T 630ae4d66f99474382814cc32cc60ea0 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.944839 10029 raft_consensus.cc:385] T 630ae4d66f99474382814cc32cc60ea0 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.944886 10029 raft_consensus.cc:740] T 630ae4d66f99474382814cc32cc60ea0 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.945062 10029 consensus_queue.cc:260] T 630ae4d66f99474382814cc32cc60ea0 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.945529 10029 ts_tablet_manager.cc:1434] T 630ae4d66f99474382814cc32cc60ea0 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251027 21:08:02.945828 10029 tablet_bootstrap.cc:492] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:02.947842 10029 tablet_bootstrap.cc:654] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.952811 10043 ts_tablet_manager.cc:1929] T 648f506ec87c4fb8a3f6d4591873dbb8 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.959450 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 34f7313f60c440a1aba37b61dbbabc02 (DEFAULT_TABLE table=default.Table_2592614ec6a74410b7050171ee4d6d1b [id=7168e50c0645436b8c706086190b1736]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.959796 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34f7313f60c440a1aba37b61dbbabc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.960801 10639 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 21:08:02.961804 10639 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 21:08:02.962078 10639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 21:08:02.962255 10043 log.cc:1199] T 648f506ec87c4fb8a3f6d4591873dbb8 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/648f506ec87c4fb8a3f6d4591873dbb8
I20251027 21:08:02.962600 10043 ts_tablet_manager.cc:1950] T 648f506ec87c4fb8a3f6d4591873dbb8 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:02.962734 10031 tablet_bootstrap.cc:654] T abd3b8e71d3041b4b9e5f6663841bfcf P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.963413 10043 tablet_replica.cc:333] T a9bf3a093be8498c9dd5511b53436b16 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:02.963618 10043 raft_consensus.cc:2243] T a9bf3a093be8498c9dd5511b53436b16 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.963583 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 34f7313f60c440a1aba37b61dbbabc02 (DEFAULT_TABLE table=default.Table_2592614ec6a74410b7050171ee4d6d1b [id=7168e50c0645436b8c706086190b1736]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.963795 10043 raft_consensus.cc:2272] T a9bf3a093be8498c9dd5511b53436b16 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.964798 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45568792d7954b498581ce69fb611948" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:02.964990 9924 raft_consensus.cc:2468] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:02.965026 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 312dc1e8caea4627b268536a6ce6344d (DEFAULT_TABLE table=default.Table_fdca4795c6d042ec80cf84a378acb979 [id=a92ec852c916461eb59fba87bd1c81eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.965497 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 312dc1e8caea4627b268536a6ce6344d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.966811 10873 raft_consensus.cc:493] T d56d323ef0e74344a479940b93fea11a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:02.966940 10873 raft_consensus.cc:515] T d56d323ef0e74344a479940b93fea11a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.967483 10873 leader_election.cc:290] T d56d323ef0e74344a479940b93fea11a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:02.968354 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d56d323ef0e74344a479940b93fea11a" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:02.969022 9860 leader_election.cc:343] T d56d323ef0e74344a479940b93fea11a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:02.968012 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d56d323ef0e74344a479940b93fea11a" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:02.971917 9859 leader_election.cc:343] T d56d323ef0e74344a479940b93fea11a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:02.972021 9859 leader_election.cc:304] T d56d323ef0e74344a479940b93fea11a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:02.972216 10873 raft_consensus.cc:2749] T d56d323ef0e74344a479940b93fea11a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:02.974342 10040 ts_tablet_manager.cc:1929] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.974442 10040 log.cc:1199] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/c1f99ca89f924b58b1416d2945a6ba40
I20251027 21:08:02.974710 10040 ts_tablet_manager.cc:1950] T c1f99ca89f924b58b1416d2945a6ba40 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:02.975267 10043 ts_tablet_manager.cc:1916] T a9bf3a093be8498c9dd5511b53436b16 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.975302 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet c1f99ca89f924b58b1416d2945a6ba40 (table default.Table_90157e3841104d92aafc5d82ffea13a9 [id=bc676ad23e1f4c4bae88752f31108ee2]) successfully deleted
I20251027 21:08:02.975742 10040 tablet_replica.cc:333] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:02.975903 10040 raft_consensus.cc:2243] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.964030 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34f7313f60c440a1aba37b61dbbabc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:02.964918 10029 tablet_bootstrap.cc:492] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:02.978264 10029 ts_tablet_manager.cc:1403] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.005s
I20251027 21:08:02.978957 10029 raft_consensus.cc:359] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.979103 10029 raft_consensus.cc:385] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:02.979158 10029 raft_consensus.cc:740] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:02.979337 10029 consensus_queue.cc:260] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.979763 10029 ts_tablet_manager.cc:1434] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251027 21:08:02.980031 10029 tablet_bootstrap.cc:492] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:02.981513 10029 tablet_bootstrap.cc:654] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:02.982637 10040 raft_consensus.cc:2272] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.983110 10040 ts_tablet_manager.cc:1916] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:02.987260 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 04c579bb008448e9baa841135b6dbc61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:02.987979 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 784df834432244e8812d1802f9e1a1bc (DEFAULT_TABLE table=default.Table_f5602dbcd14b4fb5ae7190b6a0bd65bc [id=078add1b91ad4cbfbe2215fe89b2334e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.988371 10873 raft_consensus.cc:493] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:02.988476 10873 raft_consensus.cc:515] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:02.989012 10873 leader_election.cc:290] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:02.989722 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bbb4116d1edd492893adfbf5fb3ddeef" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:02.990219 9859 leader_election.cc:343] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:02.990628 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bbb4116d1edd492893adfbf5fb3ddeef" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:02.991145 9860 leader_election.cc:343] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:02.991253 9860 leader_election.cc:304] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:02.991641 10873 raft_consensus.cc:2749] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:02.992323 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 630ae4d66f99474382814cc32cc60ea0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2f641bc4dbed4c0abd0a55b253352507) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:02.993269 10043 ts_tablet_manager.cc:1929] T a9bf3a093be8498c9dd5511b53436b16 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:02.993409 10043 log.cc:1199] T a9bf3a093be8498c9dd5511b53436b16 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/a9bf3a093be8498c9dd5511b53436b16
I20251027 21:08:02.993767 10043 ts_tablet_manager.cc:1950] T a9bf3a093be8498c9dd5511b53436b16 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:02.994547 10043 tablet_replica.cc:333] T c4996d8fae094db8a96599b9057f1649 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:02.994758 10043 raft_consensus.cc:2243] T c4996d8fae094db8a96599b9057f1649 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:02.994913 10043 raft_consensus.cc:2272] T c4996d8fae094db8a96599b9057f1649 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:02.996181 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 9d19e9f63acf4c8c894af38a02225fc6 (DEFAULT_TABLE table=default.Table_9cd91730e4c4415e9f65d748f4592366 [id=abfccf9280c64ed19771c6e9c678673a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.996714 9899 tablet_service.cc:1505] Processing CreateTablet for tablet b748f59a56c54c2b850f5336a1d61c26 (DEFAULT_TABLE table=default.Table_937452027956409d907e8d44579e9d66 [id=983e5d91d7994da8ad3e4231d799e80a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.996781 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 6bbee5f1fb704709b6f166a153e69808 (DEFAULT_TABLE table=default.Table_b0bf4b9e330b4ec48becdc293dcef72a [id=be5aa962c8004df9aa16483c6eeab83f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.997124 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 43b5cef5fdcd4d2fa89875aa6d0efcd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:02.997473 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 59b6818d29624da5a49dd1a7d20994c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:02.997833 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f641bc4dbed4c0abd0a55b253352507 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:02.998414 9905 tablet_service.cc:1505] Processing CreateTablet for tablet ada34d5086484ef18e2050ac05b71725 (DEFAULT_TABLE table=default.Table_9e349c15adbf444f96f4d83d3acb5990 [id=04c44da04ebc4b828c688eeb3bbd2e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.998996 9898 tablet_service.cc:1505] Processing CreateTablet for tablet dbf46e4ce7714398b2c2019314e44578 (DEFAULT_TABLE table=default.Table_56d94ef7571b42b4ba4e66f6ce476ead [id=aa53ec98c45e4f1db9d9abf1993ff200]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:02.999583 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 016e1aa5266d44b5b35a3f47e8f11a46 (DEFAULT_TABLE table=default.Table_db03a0faacfc4ecfad89dbf48def9140 [id=409b668e9fed4a748a4cb43c5c69551a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:03.000183 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 070167be7d8d4ddf8da561df14448360 (DEFAULT_TABLE table=default.Table_92cfd00027b045f3b151deec3558ece6 [id=479548b373ef42b4ad8683a9dcfb3ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:03.000602 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet bbb4116d1edd492893adfbf5fb3ddeef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.001008 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet cef2bfee222f45e6b6f20c6d63ff44d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.001364 9568 leader_election.cc:304] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.001784 10859 raft_consensus.cc:2804] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:03.001860 10859 raft_consensus.cc:493] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:03.001928 10859 raft_consensus.cc:3060] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:03.002665 10043 ts_tablet_manager.cc:1916] T c4996d8fae094db8a96599b9057f1649 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.004480 10859 raft_consensus.cc:515] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.005354 10859 leader_election.cc:290] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:03.005885 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45568792d7954b498581ce69fb611948" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:03.006296 9566 leader_election.cc:343] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.009224 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 1d1d4031a14e448ea5a60327bd37d003 (DEFAULT_TABLE table=default.Table_456ae12ae5d04554b252e71a72c3d9c1 [id=f9b30813deaa450d8b8ce2748950fcf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:03.009534 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d1d4031a14e448ea5a60327bd37d003. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:03.019234 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f0f82afba21846a89a2fa83897fde34f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:03.020026 10043 ts_tablet_manager.cc:1929] T c4996d8fae094db8a96599b9057f1649 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.020179 10043 log.cc:1199] T c4996d8fae094db8a96599b9057f1649 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/c4996d8fae094db8a96599b9057f1649
I20251027 21:08:03.020526 10043 ts_tablet_manager.cc:1950] T c4996d8fae094db8a96599b9057f1649 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:03.025139 10043 tablet_replica.cc:333] T d872cf3ef39042ff9a615b55e7195afa P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:03.025343 10043 raft_consensus.cc:2243] T d872cf3ef39042ff9a615b55e7195afa P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.025478 10043 raft_consensus.cc:2272] T d872cf3ef39042ff9a615b55e7195afa P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.026043 10873 raft_consensus.cc:493] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.026113 10043 ts_tablet_manager.cc:1916] T d872cf3ef39042ff9a615b55e7195afa P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.026211 10873 raft_consensus.cc:515] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.019614 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 1391ad8a22944ab7a8f45aceec19008b
I20251027 21:08:03.026839 10873 leader_election.cc:290] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:03.027153 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_27e2b6a6c7684a7baefcca99d2f93e87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:03.027659 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27e2b6a6c7684a7baefcca99d2f93e87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:03.028719 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "980eb6755aee4bcfb2ea44d84009021f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:03.029476 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0f82afba21846a89a2fa83897fde34f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:03.030365 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c5bc4ee5ec5f4a039356ac8841f5b836
I20251027 21:08:03.030983 9310 master_service.cc:954] ReplaceTablet: received request to replace tablet 87612b8bd8a04fd9b32f2df3f9e0d997 from {username='slave'} at 127.0.0.1:41710
W20251027 21:08:03.031401 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 050e9442041d495496bfa7ce25ac6a88 consensus state has no leader
W20251027 21:08:03.032522 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93d2f7df14a74731aea84927f0b7ed45 consensus state has no leader
W20251027 21:08:03.033972 9859 leader_election.cc:343] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.034017 10043 ts_tablet_manager.cc:1929] T d872cf3ef39042ff9a615b55e7195afa P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.034121 10043 log.cc:1199] T d872cf3ef39042ff9a615b55e7195afa P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/d872cf3ef39042ff9a615b55e7195afa
I20251027 21:08:03.034415 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet c5bc4ee5ec5f4a039356ac8841f5b836 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:03.034579 10043 ts_tablet_manager.cc:1950] T d872cf3ef39042ff9a615b55e7195afa P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:03.035605 10043 tablet_replica.cc:333] T c5f2e60caa824b5db2330b423221873c P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:03.035904 10043 raft_consensus.cc:2243] T c5f2e60caa824b5db2330b423221873c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.036113 10043 raft_consensus.cc:2272] T c5f2e60caa824b5db2330b423221873c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.036986 10043 ts_tablet_manager.cc:1916] T c5f2e60caa824b5db2330b423221873c P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.037406 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "980eb6755aee4bcfb2ea44d84009021f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:03.037906 9860 leader_election.cc:343] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.038087 9860 leader_election.cc:304] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.038833 10873 raft_consensus.cc:2749] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:03.045174 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet c5bc4ee5ec5f4a039356ac8841f5b836 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.049243 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 1391ad8a22944ab7a8f45aceec19008b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:03.050077 10040 ts_tablet_manager.cc:1929] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.061411 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 784df834432244e8812d1802f9e1a1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:03.062369 10043 ts_tablet_manager.cc:1929] T c5f2e60caa824b5db2330b423221873c P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.062464 10043 log.cc:1199] T c5f2e60caa824b5db2330b423221873c P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/c5f2e60caa824b5db2330b423221873c
I20251027 21:08:03.062727 10043 ts_tablet_manager.cc:1950] T c5f2e60caa824b5db2330b423221873c P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:03.063515 10043 tablet_replica.cc:333] T b719dc40d7e348608cdfc78401c19824 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:03.063676 10043 raft_consensus.cc:2243] T b719dc40d7e348608cdfc78401c19824 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.063799 10043 raft_consensus.cc:2272] T b719dc40d7e348608cdfc78401c19824 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.063804 10041 tablet_replica.cc:333] T 04c579bb008448e9baa841135b6dbc61 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:03.064024 10041 raft_consensus.cc:2243] T 04c579bb008448e9baa841135b6dbc61 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.064164 10041 raft_consensus.cc:2272] T 04c579bb008448e9baa841135b6dbc61 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.064816 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b748f59a56c54c2b850f5336a1d61c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:03.069243 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d19e9f63acf4c8c894af38a02225fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:03.070849 10043 ts_tablet_manager.cc:1916] T b719dc40d7e348608cdfc78401c19824 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.074038 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bbee5f1fb704709b6f166a153e69808. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:03.077096 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbf46e4ce7714398b2c2019314e44578. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:03.078467 10043 ts_tablet_manager.cc:1929] T b719dc40d7e348608cdfc78401c19824 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.078573 10043 log.cc:1199] T b719dc40d7e348608cdfc78401c19824 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/b719dc40d7e348608cdfc78401c19824
I20251027 21:08:03.078874 10043 ts_tablet_manager.cc:1950] T b719dc40d7e348608cdfc78401c19824 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:03.079772 10043 tablet_replica.cc:333] T 630ae4d66f99474382814cc32cc60ea0 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:03.080006 10043 raft_consensus.cc:2243] T 630ae4d66f99474382814cc32cc60ea0 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.080145 10043 raft_consensus.cc:2272] T 630ae4d66f99474382814cc32cc60ea0 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.082249 10029 tablet_bootstrap.cc:492] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:03.082338 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 016e1aa5266d44b5b35a3f47e8f11a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:03.082363 10029 ts_tablet_manager.cc:1403] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.102s user 0.006s sys 0.005s
I20251027 21:08:03.082996 10029 raft_consensus.cc:359] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.083150 10029 raft_consensus.cc:385] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.083190 10029 raft_consensus.cc:740] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.083340 10029 consensus_queue.cc:260] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.083729 10029 ts_tablet_manager.cc:1434] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:03.085122 10029 tablet_bootstrap.cc:492] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:03.086345 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 070167be7d8d4ddf8da561df14448360. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:03.087467 10029 tablet_bootstrap.cc:654] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.092245 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ada34d5086484ef18e2050ac05b71725. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:03.098660 10032 tablet_bootstrap.cc:492] T f775a7e42bdc4910bdb604b26a28a7f4 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:03.098775 10032 ts_tablet_manager.cc:1403] T f775a7e42bdc4910bdb604b26a28a7f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.197s user 0.039s sys 0.057s
I20251027 21:08:03.099390 10032 raft_consensus.cc:359] T f775a7e42bdc4910bdb604b26a28a7f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.099531 10032 raft_consensus.cc:385] T f775a7e42bdc4910bdb604b26a28a7f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.099571 10032 raft_consensus.cc:740] T f775a7e42bdc4910bdb604b26a28a7f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.099754 10032 consensus_queue.cc:260] T f775a7e42bdc4910bdb604b26a28a7f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.100150 10041 ts_tablet_manager.cc:1916] T 04c579bb008448e9baa841135b6dbc61 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.101610 10032 ts_tablet_manager.cc:1434] T f775a7e42bdc4910bdb604b26a28a7f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:08:03.101958 10032 tablet_bootstrap.cc:492] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:03.104991 10032 tablet_bootstrap.cc:654] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.111078 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 648f506ec87c4fb8a3f6d4591873dbb8 (table default.Table_a46825cc888a4a0898d6db70b02e5ab6 [id=8579d3ff46d64c75b1da213fdee9cc1b]) successfully deleted
I20251027 21:08:03.111265 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet a9bf3a093be8498c9dd5511b53436b16 (table default.Table_627b1e3af2d34bc9b4508237abf020f5 [id=5c9fd870185a4211a5a956171f491cc0]) successfully deleted
I20251027 21:08:03.111433 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet c4996d8fae094db8a96599b9057f1649 (table default.Table_b462e4bf95dc4fc4ad3fd23b236fffa7 [id=68da6285e1d247f1950a0b4929c885a4]) successfully deleted
I20251027 21:08:03.111543 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet d872cf3ef39042ff9a615b55e7195afa (table default.Table_2c2ec3f9d23142c9b8c2d66d1c979525 [id=59de715802f547bca16f181917fcc418]) successfully deleted
I20251027 21:08:03.111665 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet c5f2e60caa824b5db2330b423221873c (table default.Table_a90c80308cb345d7abc6c44d3d4c70ff [id=0e7e4ea125894e78a2162e9e8063ee33]) successfully deleted
I20251027 21:08:03.111795 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet b719dc40d7e348608cdfc78401c19824 (table default.Table_41078339d0834d069db2e8ae485bcc89 [id=f641ffdf5fd7497096643ea3278c49d0]) successfully deleted
I20251027 21:08:03.113077 10040 log.cc:1199] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/288b88e13fdf4427aae53f64b87733bf
I20251027 21:08:03.113376 10040 ts_tablet_manager.cc:1950] T 288b88e13fdf4427aae53f64b87733bf P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:03.113889 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 288b88e13fdf4427aae53f64b87733bf (table default.Table_a061b11818fc4a2db61e51a5972d69db [id=92ff9768853d403fb15db41e79fe77bc]) successfully deleted
I20251027 21:08:03.114239 10040 tablet_replica.cc:333] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:03.114442 10040 raft_consensus.cc:2243] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.114662 10040 raft_consensus.cc:2272] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.115177 10040 ts_tablet_manager.cc:1916] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.121475 10040 ts_tablet_manager.cc:1929] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.121616 10040 log.cc:1199] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/e0968c2f25b64f6685f38a999ee9d60e
I20251027 21:08:03.121920 10040 ts_tablet_manager.cc:1950] T e0968c2f25b64f6685f38a999ee9d60e P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:03.122438 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet e0968c2f25b64f6685f38a999ee9d60e (table default.Table_68f478946b1543a68842ffad9e14864f [id=830444c79e80486aa3e7668a0ec91023]) successfully deleted
I20251027 21:08:03.122740 10040 tablet_replica.cc:333] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:03.122912 10040 raft_consensus.cc:2243] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.123096 10040 raft_consensus.cc:2272] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.123544 10040 ts_tablet_manager.cc:1916] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.125398 10041 ts_tablet_manager.cc:1929] T 04c579bb008448e9baa841135b6dbc61 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.126205 10041 log.cc:1199] T 04c579bb008448e9baa841135b6dbc61 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/04c579bb008448e9baa841135b6dbc61
I20251027 21:08:03.128192 10041 ts_tablet_manager.cc:1950] T 04c579bb008448e9baa841135b6dbc61 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:03.129163 10040 ts_tablet_manager.cc:1929] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.129262 10040 log.cc:1199] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/0a58cd07e1bb4ffdbf39e7e16a46022a
I20251027 21:08:03.129560 10040 ts_tablet_manager.cc:1950] T 0a58cd07e1bb4ffdbf39e7e16a46022a P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:03.130265 10041 tablet_replica.cc:333] T 59b6818d29624da5a49dd1a7d20994c6 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:03.130802 10043 ts_tablet_manager.cc:1916] T 630ae4d66f99474382814cc32cc60ea0 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.138346 10043 ts_tablet_manager.cc:1929] T 630ae4d66f99474382814cc32cc60ea0 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.138494 10043 log.cc:1199] T 630ae4d66f99474382814cc32cc60ea0 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/630ae4d66f99474382814cc32cc60ea0
I20251027 21:08:03.138818 10043 ts_tablet_manager.cc:1950] T 630ae4d66f99474382814cc32cc60ea0 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:03.139739 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 630ae4d66f99474382814cc32cc60ea0 (table default.Table_cfef185a988e4d61ae6575c228abc88d [id=8a26e744abae4c5eb1d18dd994cd6bd9]) successfully deleted
I20251027 21:08:03.140630 10041 raft_consensus.cc:2243] T 59b6818d29624da5a49dd1a7d20994c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.141003 10041 raft_consensus.cc:2272] T 59b6818d29624da5a49dd1a7d20994c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:03.141960 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet c5bc4ee5ec5f4a039356ac8841f5b836 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5bc4ee5ec5f4a039356ac8841f5b836 already in progress: creating tablet
I20251027 21:08:03.142724 10041 ts_tablet_manager.cc:1916] T 59b6818d29624da5a49dd1a7d20994c6 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:03.143358 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 1391ad8a22944ab7a8f45aceec19008b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1391ad8a22944ab7a8f45aceec19008b already in progress: creating tablet
W20251027 21:08:03.148079 9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.124:45071, user_credentials={real_user=slave}} blocked reactor thread for 204726us
I20251027 21:08:03.149466 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 1391ad8a22944ab7a8f45aceec19008b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.153753 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 0a58cd07e1bb4ffdbf39e7e16a46022a (table default.Table_bc5ec745ebe54017b6bc9635581c813d [id=31172964ef544c5f99cade30a42b340d]) successfully deleted
I20251027 21:08:03.154215 10040 tablet_replica.cc:333] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:03.154752 10040 raft_consensus.cc:2243] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.155368 10040 raft_consensus.cc:2272] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.156455 10040 ts_tablet_manager.cc:1916] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:03.159634 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c3c89b7036b438cad79d0472619257e consensus state has no leader
W20251027 21:08:03.161465 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1ba47e34f5464b47bc32b2776f37d97d consensus state has no leader
I20251027 21:08:03.156831 10031 tablet_bootstrap.cc:492] T abd3b8e71d3041b4b9e5f6663841bfcf P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:03.166847 10031 ts_tablet_manager.cc:1403] T abd3b8e71d3041b4b9e5f6663841bfcf P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.247s user 0.017s sys 0.020s
I20251027 21:08:03.166944 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 1d1d4031a14e448ea5a60327bd37d003 (DEFAULT_TABLE table=default.Table_456ae12ae5d04554b252e71a72c3d9c1 [id=f9b30813deaa450d8b8ce2748950fcf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:03.167270 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d1d4031a14e448ea5a60327bd37d003. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:03.167742 10031 raft_consensus.cc:359] T abd3b8e71d3041b4b9e5f6663841bfcf P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:03.167932 10031 raft_consensus.cc:385] T abd3b8e71d3041b4b9e5f6663841bfcf P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.167994 10031 raft_consensus.cc:740] T abd3b8e71d3041b4b9e5f6663841bfcf P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.168211 10031 consensus_queue.cc:260] T abd3b8e71d3041b4b9e5f6663841bfcf P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:03.168797 10031 ts_tablet_manager.cc:1434] T abd3b8e71d3041b4b9e5f6663841bfcf P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:03.171200 9305 master_service.cc:954] ReplaceTablet: received request to replace tablet a9339c6b1154400eace53d765528fdf9 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:03.172214 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0a1b25591a094198b8123b3ac2af4715"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:03.172777 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a1b25591a094198b8123b3ac2af4715 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:03.174043 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4755cbfcd1464ec1b6f0b909b99f0c3f" } new_table_name: "default.Table_8dce34af37324a659bd37f54cd0c379b" modify_external_catalogs: true
I20251027 21:08:03.175411 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_088cee7e137e418599e9dc3d73b42640"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:03.176045 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_088cee7e137e418599e9dc3d73b42640 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:03.177935 9311 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 345d3795a37b4a40b40967cfacf0f893 (table default.Table_ef79e4d09fb741429811f81e1efd90cb [id=93d6307da3a34f87a67018a06825cd15]). Expected version 2 got 0
I20251027 21:08:03.178798 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0a195bfbfcbc4c1484188d25d0399353"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:03.179235 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a195bfbfcbc4c1484188d25d0399353 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:03.182058 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5b5950cb52f34415b3a217c9c772bb3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:03.182464 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b5950cb52f34415b3a217c9c772bb3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:03.183524 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7057cb41215c437391bce90bc612d1df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:03.183961 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7057cb41215c437391bce90bc612d1df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:03.186189 10040 ts_tablet_manager.cc:1929] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.187994 10040 log.cc:1199] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/f34512059f0c47fc9af7db2645f86b40
I20251027 21:08:03.188398 10031 tablet_bootstrap.cc:492] T b8856c5dc8f144358445165324bfb477 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:03.190397 10031 tablet_bootstrap.cc:654] T b8856c5dc8f144358445165324bfb477 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.194566 10040 ts_tablet_manager.cc:1950] T f34512059f0c47fc9af7db2645f86b40 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:03.195013 10031 tablet_bootstrap.cc:492] T b8856c5dc8f144358445165324bfb477 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:03.196018 10031 ts_tablet_manager.cc:1403] T b8856c5dc8f144358445165324bfb477 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20251027 21:08:03.196735 10031 raft_consensus.cc:359] T b8856c5dc8f144358445165324bfb477 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.196959 10031 raft_consensus.cc:385] T b8856c5dc8f144358445165324bfb477 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.197062 10031 raft_consensus.cc:740] T b8856c5dc8f144358445165324bfb477 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.197316 10031 consensus_queue.cc:260] T b8856c5dc8f144358445165324bfb477 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:03.198213 9296 consensus_peers.cc:597] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 -> Peer 6e86ca5e83dd4df5b72f045e7b787e81 (127.9.14.124:45071): Couldn't send request to peer 6e86ca5e83dd4df5b72f045e7b787e81. Status: Network error: Client connection negotiation failed: client connection to 127.9.14.124:45071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20251027 21:08:03.198843 10031 ts_tablet_manager.cc:1434] T b8856c5dc8f144358445165324bfb477 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:08:03.198885 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 04c579bb008448e9baa841135b6dbc61 (table default.Table_fe308a2645c14e6c8efc7ce330b5add7 [id=73cba162a3af4c0fa5dfde81e1a81257]) successfully deleted
I20251027 21:08:03.199892 10031 tablet_bootstrap.cc:492] T 3103c8e5a7c744be90be097179904ba3 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:03.201117 10031 tablet_bootstrap.cc:654] T 3103c8e5a7c744be90be097179904ba3 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:03.206852 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 43b5cef5fdcd4d2fa89875aa6d0efcd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43b5cef5fdcd4d2fa89875aa6d0efcd0 already in progress: creating tablet
I20251027 21:08:03.209461 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet f34512059f0c47fc9af7db2645f86b40 (table default.Table_660eb77e69884fe5bd5e65b47ea91dde [id=14daa3e199fe4fa3ba2de06dd6be4330]) successfully deleted
W20251027 21:08:03.210714 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c9c86253ce914916bee6584675a82067 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9c86253ce914916bee6584675a82067 already in progress: creating tablet
W20251027 21:08:03.211736 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 4c023428437f45b4bae1374bd3dd36a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c023428437f45b4bae1374bd3dd36a7 already in progress: creating tablet
W20251027 21:08:03.212666 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 217fbfb1f58f420f8f258b4178901370 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 217fbfb1f58f420f8f258b4178901370 already in progress: creating tablet
I20251027 21:08:03.213300 10031 tablet_bootstrap.cc:492] T 3103c8e5a7c744be90be097179904ba3 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:03.213423 10031 ts_tablet_manager.cc:1403] T 3103c8e5a7c744be90be097179904ba3 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
W20251027 21:08:03.213527 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 0bee9d4601c94b138d68f7663769ccbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bee9d4601c94b138d68f7663769ccbe already in progress: creating tablet
I20251027 21:08:03.214167 10040 tablet_replica.cc:333] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:03.214700 10040 raft_consensus.cc:2243] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.214823 10873 raft_consensus.cc:493] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.214969 10873 raft_consensus.cc:515] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.215269 10040 raft_consensus.cc:2272] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.215586 10873 leader_election.cc:290] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:03.215731 10040 ts_tablet_manager.cc:1916] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.216214 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94f2819dd0394530ae93754bdd1e482f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:03.216621 9859 leader_election.cc:343] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.216900 9923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45568792d7954b498581ce69fb611948" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:03.217115 9923 raft_consensus.cc:3060] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:03.217480 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94f2819dd0394530ae93754bdd1e482f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:03.214298 10031 raft_consensus.cc:359] T 3103c8e5a7c744be90be097179904ba3 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.218092 10031 raft_consensus.cc:385] T 3103c8e5a7c744be90be097179904ba3 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251027 21:08:03.218070 9860 leader_election.cc:343] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.218165 10031 raft_consensus.cc:740] T 3103c8e5a7c744be90be097179904ba3 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.218187 9860 leader_election.cc:304] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.218597 10873 raft_consensus.cc:2749] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:03.219120 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 34f7313f60c440a1aba37b61dbbabc02 (DEFAULT_TABLE table=default.Table_2592614ec6a74410b7050171ee4d6d1b [id=7168e50c0645436b8c706086190b1736]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:03.219506 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34f7313f60c440a1aba37b61dbbabc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:03.220753 9923 raft_consensus.cc:2468] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:03.221005 10873 raft_consensus.cc:493] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.221108 10873 raft_consensus.cc:515] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.221621 10873 leader_election.cc:290] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:03.222101 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fde4fd91943498d8888d1783005e492" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:03.222467 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fde4fd91943498d8888d1783005e492" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:03.222853 9859 leader_election.cc:343] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.223013 10031 consensus_queue.cc:260] T 3103c8e5a7c744be90be097179904ba3 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:03.223161 9860 leader_election.cc:343] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.223239 9860 leader_election.cc:304] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.223438 10031 ts_tablet_manager.cc:1434] T 3103c8e5a7c744be90be097179904ba3 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251027 21:08:03.223707 10873 raft_consensus.cc:2749] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:03.226205 10907 raft_consensus.cc:493] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.226383 10907 raft_consensus.cc:515] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:03.226516 10031 tablet_bootstrap.cc:492] T 94f2dbf47a5c4458bee54b26f272156e P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:03.227483 10907 leader_election.cc:290] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:03.227484 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abd3b8e71d3041b4b9e5f6663841bfcf" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:03.227651 9769 raft_consensus.cc:2468] T abd3b8e71d3041b4b9e5f6663841bfcf P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:03.227890 10829 raft_consensus.cc:493] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.228015 10829 raft_consensus.cc:515] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:03.228540 10829 leader_election.cc:290] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:03.230386 9566 leader_election.cc:304] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
W20251027 21:08:03.230894 9711 leader_election.cc:343] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: b719dc40d7e348608cdfc78401c19824
I20251027 21:08:03.231384 10031 tablet_bootstrap.cc:654] T 94f2dbf47a5c4458bee54b26f272156e P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:03.233510 9710 leader_election.cc:343] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: b719dc40d7e348608cdfc78401c19824
I20251027 21:08:03.233629 9710 leader_election.cc:304] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:03.234108 10829 raft_consensus.cc:2749] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:03.234555 10859 raft_consensus.cc:2804] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:03.234638 10859 raft_consensus.cc:493] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:03.234721 10859 raft_consensus.cc:3060] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:03.237494 10859 raft_consensus.cc:515] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:03.238456 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abd3b8e71d3041b4b9e5f6663841bfcf" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:03.238603 9769 raft_consensus.cc:3060] T abd3b8e71d3041b4b9e5f6663841bfcf P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:03.238739 10859 leader_election.cc:290] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:03.241075 9769 raft_consensus.cc:2468] T abd3b8e71d3041b4b9e5f6663841bfcf P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:03.241477 9566 leader_election.cc:304] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:03.241799 10859 raft_consensus.cc:2804] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:03.241950 10859 raft_consensus.cc:697] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:03.242265 10859 consensus_queue.cc:237] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:03.246330 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 1391ad8a22944ab7a8f45aceec19008b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.246665 10040 ts_tablet_manager.cc:1929] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.246768 10040 log.cc:1199] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/abc7458fcfb94985acf76d77c0292b08
I20251027 21:08:03.246906 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet c5bc4ee5ec5f4a039356ac8841f5b836 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.247048 10040 ts_tablet_manager.cc:1950] T abc7458fcfb94985acf76d77c0292b08 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:03.247581 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet abc7458fcfb94985acf76d77c0292b08 (table default.Table_9c027ff744824da9b9cac5574017e7fd [id=40a6848701284864b34f727da289347b]) successfully deleted
I20251027 21:08:03.250447 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 1d1d4031a14e448ea5a60327bd37d003 (DEFAULT_TABLE table=default.Table_456ae12ae5d04554b252e71a72c3d9c1 [id=f9b30813deaa450d8b8ce2748950fcf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:03.250993 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d1d4031a14e448ea5a60327bd37d003. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:03.251708 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 4da0bb391aba49cbaeb72983553362b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4da0bb391aba49cbaeb72983553362b2 already in progress: creating tablet
W20251027 21:08:03.253332 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet e419e588b31d48c8bc19d7948081cc81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e419e588b31d48c8bc19d7948081cc81 already in progress: creating tablet
W20251027 21:08:03.254159 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet fb7fdc91156e41de80872b5c941c5d95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb7fdc91156e41de80872b5c941c5d95 already in progress: creating tablet
W20251027 21:08:03.255336 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 0f90e09e74b6495db1aa2df85de21181 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f90e09e74b6495db1aa2df85de21181 already in progress: creating tablet
I20251027 21:08:03.256191 10040 tablet_replica.cc:333] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:03.256824 10040 raft_consensus.cc:2243] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.257494 10040 raft_consensus.cc:2272] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.258497 10040 ts_tablet_manager.cc:1916] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.258667 9568 leader_election.cc:304] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.259505 10859 raft_consensus.cc:2804] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:03.259665 10859 raft_consensus.cc:697] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:03.259951 10859 consensus_queue.cc:237] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:03.268474 9299 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 0820648719a34f379507efced9cc434b (table default.Table_5a979a51f58942db97c3897dcd5feb08 [id=b0301c76f48f461c8ee32b925c91234b], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0820648719a34f379507efced9cc434b consensus state has no leader
I20251027 21:08:03.270339 9923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abd3b8e71d3041b4b9e5f6663841bfcf" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:03.270560 9923 raft_consensus.cc:2468] T abd3b8e71d3041b4b9e5f6663841bfcf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:03.270869 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abd3b8e71d3041b4b9e5f6663841bfcf" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:03.271028 9924 raft_consensus.cc:3060] T abd3b8e71d3041b4b9e5f6663841bfcf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:03.271255 10029 tablet_bootstrap.cc:492] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:03.271379 10029 ts_tablet_manager.cc:1403] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.186s user 0.003s sys 0.026s
I20251027 21:08:03.272028 10029 raft_consensus.cc:359] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.272159 10029 raft_consensus.cc:385] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.272202 10029 raft_consensus.cc:740] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.272364 10029 consensus_queue.cc:260] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.272763 10029 ts_tablet_manager.cc:1434] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:03.273034 10029 tablet_bootstrap.cc:492] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:03.275383 9924 raft_consensus.cc:2468] T abd3b8e71d3041b4b9e5f6663841bfcf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:03.282429 10040 ts_tablet_manager.cc:1929] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.282539 10040 log.cc:1199] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/9882b234e79940088be5f604c41576b7
I20251027 21:08:03.282708 10041 ts_tablet_manager.cc:1929] T 59b6818d29624da5a49dd1a7d20994c6 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.282816 10040 ts_tablet_manager.cc:1950] T 9882b234e79940088be5f604c41576b7 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:03.283329 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 9882b234e79940088be5f604c41576b7 (table default.Table_e0dee9209ca14dd98065b0384855d953 [id=92995507b7d2462f91b1a16434d3ac06]) successfully deleted
I20251027 21:08:03.283650 10829 raft_consensus.cc:493] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.283825 10829 raft_consensus.cc:515] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:03.284453 10829 leader_election.cc:290] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:03.284613 10029 tablet_bootstrap.cc:654] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:03.285347 9711 leader_election.cc:343] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 533f34d4872444278477abcce89da9ec
I20251027 21:08:03.287062 10040 tablet_replica.cc:333] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:03.287221 10040 raft_consensus.cc:2243] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.287358 10040 raft_consensus.cc:2272] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.287823 10040 ts_tablet_manager.cc:1916] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:03.289108 9710 leader_election.cc:343] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 533f34d4872444278477abcce89da9ec
I20251027 21:08:03.289198 9710 leader_election.cc:304] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
W20251027 21:08:03.289356 9710 raft_consensus.cc:2686] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251027 21:08:03.297894 10029 tablet_bootstrap.cc:492] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:03.298029 10029 ts_tablet_manager.cc:1403] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20251027 21:08:03.298687 10029 raft_consensus.cc:359] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:03.298852 10029 raft_consensus.cc:385] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.298902 10029 raft_consensus.cc:740] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.299081 10029 consensus_queue.cc:260] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:03.299531 10029 ts_tablet_manager.cc:1434] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20251027 21:08:03.299793 10029 tablet_bootstrap.cc:492] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:03.301469 10029 tablet_bootstrap.cc:654] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.307765 10040 ts_tablet_manager.cc:1929] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.307857 10040 log.cc:1199] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/533f34d4872444278477abcce89da9ec
I20251027 21:08:03.308120 10040 ts_tablet_manager.cc:1950] T 533f34d4872444278477abcce89da9ec P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:03.308744 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 533f34d4872444278477abcce89da9ec (table default.Table_61bbf6b5c1e740128b80465ef3b9a50e [id=439f48256cbd43f785f57857e83ae25f]) successfully deleted
I20251027 21:08:03.316812 10040 tablet_replica.cc:333] T fe24af9838954a9bb680474800642e62 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:03.316975 10040 raft_consensus.cc:2243] T fe24af9838954a9bb680474800642e62 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.317111 10040 raft_consensus.cc:2272] T fe24af9838954a9bb680474800642e62 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.317606 10040 ts_tablet_manager.cc:1916] T fe24af9838954a9bb680474800642e62 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.325663 10029 tablet_bootstrap.cc:492] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:03.325798 10029 ts_tablet_manager.cc:1403] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.005s
I20251027 21:08:03.326475 10029 raft_consensus.cc:359] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.326635 10029 raft_consensus.cc:385] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.326682 10029 raft_consensus.cc:740] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.326885 10029 consensus_queue.cc:260] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:03.327719 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet befdd3a485a1426b8a3adf340d4eae4f consensus state has no leader
I20251027 21:08:03.327814 10029 ts_tablet_manager.cc:1434] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251027 21:08:03.328050 10029 tablet_bootstrap.cc:492] T dd7029986a5246ee823e5fdb50683118 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:03.328238 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c4824ada6d0f433dbb4770562a66e1ad" } modify_external_catalogs: true
I20251027 21:08:03.328431 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c4824ada6d0f433dbb4770562a66e1ad" } modify_external_catalogs: true
I20251027 21:08:03.328579 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_782e39a9c1114d0590c00c48ee3946c6" } modify_external_catalogs: true
I20251027 21:08:03.328755 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_782e39a9c1114d0590c00c48ee3946c6" } modify_external_catalogs: true
I20251027 21:08:03.329581 10029 tablet_bootstrap.cc:654] T dd7029986a5246ee823e5fdb50683118 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.329989 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2c4482a14aa14f498e7c8ab78d9fe9ee" } modify_external_catalogs: true
I20251027 21:08:03.330181 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2c4482a14aa14f498e7c8ab78d9fe9ee" } modify_external_catalogs: true
I20251027 21:08:03.330772 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet b719dc40d7e348608cdfc78401c19824 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.330814 9308 master_service.cc:954] ReplaceTablet: received request to replace tablet b99232a6018a4b1a850da954c551d38d from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:03.331712 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet 67bb12de9f044c73ae9ab26d10498c90 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:03.337266 9312 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet abd3b8e71d3041b4b9e5f6663841bfcf (table default.Table_fd331f3cfc07425082d58ab5af3c1094 [id=f3e3147411e5488bae105ed755f5b50f]). Expected version 2 got 0
I20251027 21:08:03.337527 9312 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet b8856c5dc8f144358445165324bfb477 (table default.Table_203191e192be4e3a9a7477b4313ef4f4 [id=5da6bd4463e24c03accd9985f72b6ebb]). Expected version 3 got 0
I20251027 21:08:03.339099 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet 3103c8e5a7c744be90be097179904ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.343483 10029 tablet_bootstrap.cc:492] T dd7029986a5246ee823e5fdb50683118 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:03.343673 10029 ts_tablet_manager.cc:1403] T dd7029986a5246ee823e5fdb50683118 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.002s
I20251027 21:08:03.344285 10029 raft_consensus.cc:359] T dd7029986a5246ee823e5fdb50683118 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.345006 10029 raft_consensus.cc:385] T dd7029986a5246ee823e5fdb50683118 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.345227 10029 raft_consensus.cc:740] T dd7029986a5246ee823e5fdb50683118 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.345585 10029 consensus_queue.cc:260] T dd7029986a5246ee823e5fdb50683118 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.346935 10029 ts_tablet_manager.cc:1434] T dd7029986a5246ee823e5fdb50683118 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:08:03.348505 10029 tablet_bootstrap.cc:492] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:03.350909 10029 tablet_bootstrap.cc:654] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.351713 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_937452027956409d907e8d44579e9d66" } modify_external_catalogs: true
I20251027 21:08:03.352012 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_937452027956409d907e8d44579e9d66" } modify_external_catalogs: true
I20251027 21:08:03.356711 10029 tablet_bootstrap.cc:492] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:03.356827 10029 ts_tablet_manager.cc:1403] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20251027 21:08:03.357442 10029 raft_consensus.cc:359] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.357585 10029 raft_consensus.cc:385] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.357638 10029 raft_consensus.cc:740] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.359725 10029 consensus_queue.cc:260] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.360894 10029 ts_tablet_manager.cc:1434] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251027 21:08:03.361117 10029 tablet_bootstrap.cc:492] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:03.362273 10041 log.cc:1199] T 59b6818d29624da5a49dd1a7d20994c6 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/59b6818d29624da5a49dd1a7d20994c6
I20251027 21:08:03.362669 10041 ts_tablet_manager.cc:1950] T 59b6818d29624da5a49dd1a7d20994c6 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:03.363677 10029 tablet_bootstrap.cc:654] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.363694 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 59b6818d29624da5a49dd1a7d20994c6 (table default.Table_9904948d307e479ea184d6dc441b5e61 [id=500c0fe58264475087ef89652bf83263]) successfully deleted
I20251027 21:08:03.369645 10031 tablet_bootstrap.cc:492] T 94f2dbf47a5c4458bee54b26f272156e P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:03.369762 10031 ts_tablet_manager.cc:1403] T 94f2dbf47a5c4458bee54b26f272156e P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.143s user 0.005s sys 0.017s
I20251027 21:08:03.370435 10031 raft_consensus.cc:359] T 94f2dbf47a5c4458bee54b26f272156e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.370573 10031 raft_consensus.cc:385] T 94f2dbf47a5c4458bee54b26f272156e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.370613 10031 raft_consensus.cc:740] T 94f2dbf47a5c4458bee54b26f272156e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.370774 10031 consensus_queue.cc:260] T 94f2dbf47a5c4458bee54b26f272156e P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.371171 10031 ts_tablet_manager.cc:1434] T 94f2dbf47a5c4458bee54b26f272156e P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 21:08:03.371398 10031 tablet_bootstrap.cc:492] T 343eaeca24dc4816ae032c7e674e9412 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:03.372805 10031 tablet_bootstrap.cc:654] T 343eaeca24dc4816ae032c7e674e9412 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.375725 10859 raft_consensus.cc:493] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.375846 10859 raft_consensus.cc:515] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.376403 10859 leader_election.cc:290] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:03.376768 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53cac7817db440e79e7416d625a9b2c8" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:03.377127 9566 leader_election.cc:343] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:03.377583 9568 leader_election.cc:343] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 53cac7817db440e79e7416d625a9b2c8
I20251027 21:08:03.377687 9568 leader_election.cc:304] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.378101 10859 raft_consensus.cc:2749] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:03.380223 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet f775a7e42bdc4910bdb604b26a28a7f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 11627a84ff9b478c87c323617a09570b) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.382745 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b36a756b7ab54252aaa0c08e563b2ea3" } modify_external_catalogs: true
I20251027 21:08:03.382928 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b36a756b7ab54252aaa0c08e563b2ea3" } modify_external_catalogs: true
I20251027 21:08:03.385150 10031 tablet_bootstrap.cc:492] T 343eaeca24dc4816ae032c7e674e9412 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:03.385306 10031 ts_tablet_manager.cc:1403] T 343eaeca24dc4816ae032c7e674e9412 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20251027 21:08:03.386106 10031 raft_consensus.cc:359] T 343eaeca24dc4816ae032c7e674e9412 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.386282 10031 raft_consensus.cc:385] T 343eaeca24dc4816ae032c7e674e9412 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.386348 10031 raft_consensus.cc:740] T 343eaeca24dc4816ae032c7e674e9412 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.386574 10031 consensus_queue.cc:260] T 343eaeca24dc4816ae032c7e674e9412 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.387154 10031 ts_tablet_manager.cc:1434] T 343eaeca24dc4816ae032c7e674e9412 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:03.388034 10031 tablet_bootstrap.cc:492] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:03.388501 10029 tablet_bootstrap.cc:492] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:03.388621 10029 ts_tablet_manager.cc:1403] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.001s
I20251027 21:08:03.389664 10031 tablet_bootstrap.cc:654] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.391492 10029 raft_consensus.cc:359] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.391644 10029 raft_consensus.cc:385] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.391680 10029 raft_consensus.cc:740] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.391830 10029 consensus_queue.cc:260] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.392211 10029 ts_tablet_manager.cc:1434] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20251027 21:08:03.400812 10029 tablet_bootstrap.cc:492] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:03.402302 10029 tablet_bootstrap.cc:654] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.403812 10031 tablet_bootstrap.cc:492] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:03.403970 10031 ts_tablet_manager.cc:1403] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.003s
I20251027 21:08:03.404776 10031 raft_consensus.cc:359] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.404981 10031 raft_consensus.cc:385] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.405040 10031 raft_consensus.cc:740] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.405313 10031 consensus_queue.cc:260] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.405864 10031 ts_tablet_manager.cc:1434] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:03.406332 10031 tablet_bootstrap.cc:492] T 648f506ec87c4fb8a3f6d4591873dbb8 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:03.408002 10031 tablet_bootstrap.cc:654] T 648f506ec87c4fb8a3f6d4591873dbb8 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.408715 10029 tablet_bootstrap.cc:492] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:03.408818 10029 ts_tablet_manager.cc:1403] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20251027 21:08:03.409372 10029 raft_consensus.cc:359] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.409513 10029 raft_consensus.cc:385] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.409557 10029 raft_consensus.cc:740] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.409711 10029 consensus_queue.cc:260] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.416901 10029 ts_tablet_manager.cc:1434] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251027 21:08:03.420279 10029 tablet_bootstrap.cc:492] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:03.422590 10031 tablet_bootstrap.cc:492] T 648f506ec87c4fb8a3f6d4591873dbb8 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:03.422676 10029 tablet_bootstrap.cc:654] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.422741 10031 ts_tablet_manager.cc:1403] T 648f506ec87c4fb8a3f6d4591873dbb8 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.002s
I20251027 21:08:03.423408 10031 raft_consensus.cc:359] T 648f506ec87c4fb8a3f6d4591873dbb8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.423561 10031 raft_consensus.cc:385] T 648f506ec87c4fb8a3f6d4591873dbb8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.423607 10031 raft_consensus.cc:740] T 648f506ec87c4fb8a3f6d4591873dbb8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.423801 10031 consensus_queue.cc:260] T 648f506ec87c4fb8a3f6d4591873dbb8 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.424242 10031 ts_tablet_manager.cc:1434] T 648f506ec87c4fb8a3f6d4591873dbb8 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251027 21:08:03.424496 10031 tablet_bootstrap.cc:492] T adaa4f66bbab491f8030d439d85cc7d1 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:03.426162 10031 tablet_bootstrap.cc:654] T adaa4f66bbab491f8030d439d85cc7d1 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.428151 9769 raft_consensus.cc:3060] T b8856c5dc8f144358445165324bfb477 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:03.432929 10859 consensus_queue.cc:1048] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:03.437296 10029 tablet_bootstrap.cc:492] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:03.437758 10029 ts_tablet_manager.cc:1403] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.007s
I20251027 21:08:03.438550 10029 raft_consensus.cc:359] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.439425 10029 raft_consensus.cc:385] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.439569 10029 raft_consensus.cc:740] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.439828 10029 consensus_queue.cc:260] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.443255 10029 ts_tablet_manager.cc:1434] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251027 21:08:03.446787 10029 tablet_bootstrap.cc:492] T 2532bf7190064443a898e47e843fd9ac P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:03.447085 10041 tablet_replica.cc:333] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
W20251027 21:08:03.447468 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 2f641bc4dbed4c0abd0a55b253352507 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f641bc4dbed4c0abd0a55b253352507 already in progress: creating tablet
I20251027 21:08:03.448262 10029 tablet_bootstrap.cc:654] T 2532bf7190064443a898e47e843fd9ac P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.449632 10041 raft_consensus.cc:2243] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.449884 10041 raft_consensus.cc:2272] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.454106 10029 tablet_bootstrap.cc:492] T 2532bf7190064443a898e47e843fd9ac P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:03.454586 10029 ts_tablet_manager.cc:1403] T 2532bf7190064443a898e47e843fd9ac P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20251027 21:08:03.455415 10029 raft_consensus.cc:359] T 2532bf7190064443a898e47e843fd9ac P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.456348 10029 raft_consensus.cc:385] T 2532bf7190064443a898e47e843fd9ac P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.456413 10029 raft_consensus.cc:740] T 2532bf7190064443a898e47e843fd9ac P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.456604 10029 consensus_queue.cc:260] T 2532bf7190064443a898e47e843fd9ac P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.458532 10031 tablet_bootstrap.cc:492] T adaa4f66bbab491f8030d439d85cc7d1 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:03.458659 10031 ts_tablet_manager.cc:1403] T adaa4f66bbab491f8030d439d85cc7d1 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.002s
I20251027 21:08:03.458775 10041 ts_tablet_manager.cc:1916] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.459395 10031 raft_consensus.cc:359] T adaa4f66bbab491f8030d439d85cc7d1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.459538 10031 raft_consensus.cc:385] T adaa4f66bbab491f8030d439d85cc7d1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.459591 10031 raft_consensus.cc:740] T adaa4f66bbab491f8030d439d85cc7d1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.459817 10031 consensus_queue.cc:260] T adaa4f66bbab491f8030d439d85cc7d1 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.460779 10031 ts_tablet_manager.cc:1434] T adaa4f66bbab491f8030d439d85cc7d1 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:03.461058 10032 tablet_bootstrap.cc:492] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:03.461182 10032 ts_tablet_manager.cc:1403] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.359s user 0.073s sys 0.138s
I20251027 21:08:03.461342 10031 tablet_bootstrap.cc:492] T 45568792d7954b498581ce69fb611948 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:03.461850 10032 raft_consensus.cc:359] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.462020 10032 raft_consensus.cc:385] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.462082 10032 raft_consensus.cc:740] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.462325 10032 consensus_queue.cc:260] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.463099 10031 tablet_bootstrap.cc:654] T 45568792d7954b498581ce69fb611948 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.462751 10032 ts_tablet_manager.cc:1434] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20251027 21:08:03.465516 10032 tablet_bootstrap.cc:492] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:03.466933 10032 tablet_bootstrap.cc:654] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.467892 10029 ts_tablet_manager.cc:1434] T 2532bf7190064443a898e47e843fd9ac P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.013s user 0.000s sys 0.001s
I20251027 21:08:03.474954 10031 tablet_bootstrap.cc:492] T 45568792d7954b498581ce69fb611948 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:03.475068 10031 ts_tablet_manager.cc:1403] T 45568792d7954b498581ce69fb611948 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20251027 21:08:03.475715 10031 raft_consensus.cc:359] T 45568792d7954b498581ce69fb611948 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.475847 10031 raft_consensus.cc:385] T 45568792d7954b498581ce69fb611948 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.475894 10031 raft_consensus.cc:740] T 45568792d7954b498581ce69fb611948 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.476107 10031 consensus_queue.cc:260] T 45568792d7954b498581ce69fb611948 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.476835 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet d13d00353b6740b5bed5a153936d2059
I20251027 21:08:03.477077 10031 ts_tablet_manager.cc:1434] T 45568792d7954b498581ce69fb611948 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:03.477170 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet a9339c6b1154400eace53d765528fdf9
I20251027 21:08:03.477598 10031 tablet_bootstrap.cc:492] T a9bf3a093be8498c9dd5511b53436b16 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:03.477609 9305 catalog_manager.cc:6719] ReplaceTablet: tablet a9339c6b1154400eace53d765528fdf9 deleted and replaced by tablet a24dbceaa95e4f43a5ea416a20e7041e
I20251027 21:08:03.479135 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5cd332429fd147e1b99e2fed13e0a0e7" } new_table_name: "default.Table_836e232c1c5546ee84d694e0f866f646" modify_external_catalogs: true
I20251027 21:08:03.479372 10031 tablet_bootstrap.cc:654] T a9bf3a093be8498c9dd5511b53436b16 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.480317 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4b2294dc7f3745288d9f26232351ebd2" } new_table_name: "default.Table_1b066b58b2c649a2887ed91461f15d7c" modify_external_catalogs: true
I20251027 21:08:03.480795 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet a9339c6b1154400eace53d765528fdf9 with delete_type TABLET_DATA_DELETED (replaced by tablet a24dbceaa95e4f43a5ea416a20e7041e) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.480806 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 94f2dbf47a5c4458bee54b26f272156e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.481459 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 87612b8bd8a04fd9b32f2df3f9e0d997
I20251027 21:08:03.481820 9310 catalog_manager.cc:6719] ReplaceTablet: tablet 87612b8bd8a04fd9b32f2df3f9e0d997 deleted and replaced by tablet 0d75e6a4286e428eb85b24638382d23b
I20251027 21:08:03.482079 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8fee41d5402246f2b96068387e60cc49" } new_table_name: "default.Table_bd014e37535443cda5805af5baebc27d" modify_external_catalogs: true
I20251027 21:08:03.483697 9311 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 345d3795a37b4a40b40967cfacf0f893 (table default.Table_ef79e4d09fb741429811f81e1efd90cb [id=93d6307da3a34f87a67018a06825cd15]). Expected version 2 got 0
I20251027 21:08:03.484726 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_29369a432b8c4b108720c969d77f25b9" } new_table_name: "default.Table_f57888547f8c4fd984cc5066c90d9ae6" modify_external_catalogs: true
I20251027 21:08:03.487673 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet a9339c6b1154400eace53d765528fdf9 with delete_type TABLET_DATA_DELETED (replaced by tablet a24dbceaa95e4f43a5ea416a20e7041e) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:03.487731 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 87612b8bd8a04fd9b32f2df3f9e0d997 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d75e6a4286e428eb85b24638382d23b) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:03.487936 10043 tablet_replica.cc:333] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:03.488088 10043 raft_consensus.cc:2243] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:03.488361 10043 raft_consensus.cc:2272] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.487943 10029 tablet_bootstrap.cc:492] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:03.488641 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 87612b8bd8a04fd9b32f2df3f9e0d997 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d75e6a4286e428eb85b24638382d23b) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.490262 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 76e4fb2bea7f4a4696fe9f6bb892bccc with delete_type TABLET_DATA_DELETED (replaced by tablet 5cb8ca3a50c148e6b602e949650353f8) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.490628 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 87612b8bd8a04fd9b32f2df3f9e0d997 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d75e6a4286e428eb85b24638382d23b) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.490857 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet a9339c6b1154400eace53d765528fdf9 with delete_type TABLET_DATA_DELETED (replaced by tablet a24dbceaa95e4f43a5ea416a20e7041e) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.491079 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet d13d00353b6740b5bed5a153936d2059 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.492360 10029 tablet_bootstrap.cc:654] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.493932 10032 tablet_bootstrap.cc:492] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:03.494047 10032 ts_tablet_manager.cc:1403] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.001s
I20251027 21:08:03.494599 10859 raft_consensus.cc:493] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.494755 10859 raft_consensus.cc:515] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.495489 10859 leader_election.cc:290] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:03.496506 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22d861810ed846c3b77a452d05e93d72" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:03.496917 10032 raft_consensus.cc:359] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:08:03.497133 9566 leader_election.cc:343] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:03.497320 9568 leader_election.cc:343] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 22d861810ed846c3b77a452d05e93d72
I20251027 21:08:03.497430 9568 leader_election.cc:304] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.497540 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 8db666d9b7eb458b843ab5910cd7dff4
I20251027 21:08:03.497792 10859 raft_consensus.cc:2749] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:03.498001 10032 raft_consensus.cc:385] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.498060 10032 raft_consensus.cc:740] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.498253 10032 consensus_queue.cc:260] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:03.498662 10032 ts_tablet_manager.cc:1434] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251027 21:08:03.499699 10032 tablet_bootstrap.cc:492] T e6b58b1f4f63461aa96e416cae879be8 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:03.501580 10032 tablet_bootstrap.cc:654] T e6b58b1f4f63461aa96e416cae879be8 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.505223 10032 tablet_bootstrap.cc:492] T e6b58b1f4f63461aa96e416cae879be8 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:03.505345 10032 ts_tablet_manager.cc:1403] T e6b58b1f4f63461aa96e416cae879be8 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251027 21:08:03.505832 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 8db666d9b7eb458b843ab5910cd7dff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:03.505990 10032 raft_consensus.cc:359] T e6b58b1f4f63461aa96e416cae879be8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.506124 10032 raft_consensus.cc:385] T e6b58b1f4f63461aa96e416cae879be8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.506204 10032 raft_consensus.cc:740] T e6b58b1f4f63461aa96e416cae879be8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.506363 10032 consensus_queue.cc:260] T e6b58b1f4f63461aa96e416cae879be8 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.506798 10032 ts_tablet_manager.cc:1434] T e6b58b1f4f63461aa96e416cae879be8 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251027 21:08:03.507012 10032 tablet_bootstrap.cc:492] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:03.508773 10032 tablet_bootstrap.cc:654] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.511554 9309 catalog_manager.cc:5649] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:03.512111 9309 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet abd3b8e71d3041b4b9e5f6663841bfcf (table default.Table_fd331f3cfc07425082d58ab5af3c1094 [id=f3e3147411e5488bae105ed755f5b50f]). Expected version 2 got 0
I20251027 21:08:03.512454 9309 catalog_manager.cc:5649] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:03.512917 9309 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 45568792d7954b498581ce69fb611948 (table default.Table_3e3e0c17a3bf452fbb217c6e3ee8bdcb [id=2eb093cd58e643a58b14e564d5914fbc]). Expected version 1 got 0
I20251027 21:08:03.513389 9309 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 87d5af4607f04f92abcc7292b4176cbd (table default.Table_c78ef9ebe86b4da08443e37c83d4e80d [id=cdb023218cda4e9ea04a226b37a220d9]). Expected version 2 got 0
I20251027 21:08:03.514030 10032 tablet_bootstrap.cc:492] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:03.514156 10032 ts_tablet_manager.cc:1403] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20251027 21:08:03.514595 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 8db666d9b7eb458b843ab5910cd7dff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.514703 10032 raft_consensus.cc:359] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.514840 10032 raft_consensus.cc:385] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.514894 10032 raft_consensus.cc:740] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.515108 10032 consensus_queue.cc:260] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.515619 10032 ts_tablet_manager.cc:1434] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 21:08:03.515970 10032 tablet_bootstrap.cc:492] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:03.517489 10032 tablet_bootstrap.cc:654] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.520081 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 8db666d9b7eb458b843ab5910cd7dff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.521147 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet d13d00353b6740b5bed5a153936d2059 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.524350 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet d13d00353b6740b5bed5a153936d2059 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:03.524812 10939 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:59712: Illegal state: RaftConsensus is not running: State = Stopped
W20251027 21:08:03.525417 9298 catalog_manager.cc:5064] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): alter failed for tablet a9339c6b1154400eace53d765528fdf9 (table default.Table_5a6211c8465040dbad1cc86377a370ea [id=91fb9f95fe9343fca1d8cb0d0f781afb]): Illegal state: RaftConsensus is not running: State = Stopped
I20251027 21:08:03.528438 10032 tablet_bootstrap.cc:492] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:03.528543 10032 ts_tablet_manager.cc:1403] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.001s
I20251027 21:08:03.529122 10032 raft_consensus.cc:359] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:03.529254 10032 raft_consensus.cc:385] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.529307 10032 raft_consensus.cc:740] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.529454 10032 consensus_queue.cc:260] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:03.529826 10032 ts_tablet_manager.cc:1434] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:03.530067 10032 tablet_bootstrap.cc:492] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:03.532075 10032 tablet_bootstrap.cc:654] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.536815 10032 tablet_bootstrap.cc:492] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:03.536921 10032 ts_tablet_manager.cc:1403] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20251027 21:08:03.537475 10032 raft_consensus.cc:359] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.537590 10032 raft_consensus.cc:385] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.537628 10032 raft_consensus.cc:740] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.537868 10032 consensus_queue.cc:260] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.539285 10032 ts_tablet_manager.cc:1434] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:03.539525 10032 tablet_bootstrap.cc:492] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:03.540896 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c0857f89bc74495081d35ee55e785835
I20251027 21:08:03.541306 10032 tablet_bootstrap.cc:654] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.543844 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9e0da1160aff4c58887dbcf3ac217299"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:03.544426 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e0da1160aff4c58887dbcf3ac217299 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:03.549357 10032 tablet_bootstrap.cc:492] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:03.549460 10032 ts_tablet_manager.cc:1403] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20251027 21:08:03.550026 10032 raft_consensus.cc:359] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.550062 10873 raft_consensus.cc:493] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.550186 10032 raft_consensus.cc:385] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.550268 10032 raft_consensus.cc:740] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.550294 10873 raft_consensus.cc:515] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.550429 10032 consensus_queue.cc:260] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.550984 10873 leader_election.cc:290] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:03.551694 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaa8910d9fdb4146945621ec4ebf969e" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:03.552222 9860 leader_election.cc:343] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.552693 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaa8910d9fdb4146945621ec4ebf969e" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:03.553253 9859 leader_election.cc:343] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.553356 9859 leader_election.cc:304] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.553891 10873 raft_consensus.cc:2749] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:03.557410 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 01e9db9eaf8e41c6a2ee13b9ed127421 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.557861 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet c0857f89bc74495081d35ee55e785835 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.560336 10032 ts_tablet_manager.cc:1434] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.011s user 0.001s sys 0.009s
I20251027 21:08:03.560590 10032 tablet_bootstrap.cc:492] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:03.560505 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet c0857f89bc74495081d35ee55e785835 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.562273 10032 tablet_bootstrap.cc:654] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.562363 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet c0857f89bc74495081d35ee55e785835 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:03.574628 10907 consensus_queue.cc:1048] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:03.577463 10043 ts_tablet_manager.cc:1916] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.583575 10043 ts_tablet_manager.cc:1929] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:03.583787 10043 log.cc:1199] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/a9339c6b1154400eace53d765528fdf9
I20251027 21:08:03.584179 10043 ts_tablet_manager.cc:1950] T a9339c6b1154400eace53d765528fdf9 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:03.584388 10041 ts_tablet_manager.cc:1929] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.584497 10041 log.cc:1199] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/bbb4116d1edd492893adfbf5fb3ddeef
I20251027 21:08:03.584774 10041 ts_tablet_manager.cc:1950] T bbb4116d1edd492893adfbf5fb3ddeef P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:03.585464 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet a9339c6b1154400eace53d765528fdf9 (table default.Table_5a6211c8465040dbad1cc86377a370ea [id=91fb9f95fe9343fca1d8cb0d0f781afb]) successfully deleted
I20251027 21:08:03.585670 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet bbb4116d1edd492893adfbf5fb3ddeef (table default.Table_dfa3f51576fa4c689b2682b0e277df60 [id=088a35e69cc840c0b9386df033a885a8]) successfully deleted
W20251027 21:08:03.586232 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 87612b8bd8a04fd9b32f2df3f9e0d997 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87612b8bd8a04fd9b32f2df3f9e0d997 already in progress: creating tablet
W20251027 21:08:03.586887 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 8db666d9b7eb458b843ab5910cd7dff4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8db666d9b7eb458b843ab5910cd7dff4 already in progress: creating tablet
I20251027 21:08:03.587160 10031 tablet_bootstrap.cc:492] T a9bf3a093be8498c9dd5511b53436b16 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:03.587205 10043 tablet_replica.cc:333] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:03.587286 10031 ts_tablet_manager.cc:1403] T a9bf3a093be8498c9dd5511b53436b16 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.110s user 0.003s sys 0.015s
I20251027 21:08:03.587481 10043 raft_consensus.cc:2243] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 2 LEADER]: Raft consensus shutting down.
I20251027 21:08:03.587797 10043 raft_consensus.cc:2272] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.587924 10031 raft_consensus.cc:359] T a9bf3a093be8498c9dd5511b53436b16 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.588088 10031 raft_consensus.cc:385] T a9bf3a093be8498c9dd5511b53436b16 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.588135 10031 raft_consensus.cc:740] T a9bf3a093be8498c9dd5511b53436b16 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.588313 10031 consensus_queue.cc:260] T a9bf3a093be8498c9dd5511b53436b16 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.589051 10043 ts_tablet_manager.cc:1916] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.590612 10040 ts_tablet_manager.cc:1929] T fe24af9838954a9bb680474800642e62 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.590711 10040 log.cc:1199] T fe24af9838954a9bb680474800642e62 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/fe24af9838954a9bb680474800642e62
I20251027 21:08:03.591005 10040 ts_tablet_manager.cc:1950] T fe24af9838954a9bb680474800642e62 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:03.596370 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet b99232a6018a4b1a850da954c551d38d
I20251027 21:08:03.596866 9308 catalog_manager.cc:6719] ReplaceTablet: tablet b99232a6018a4b1a850da954c551d38d deleted and replaced by tablet 278d8f448ef144148cabfc3a57bd24fd
I20251027 21:08:03.597263 10031 ts_tablet_manager.cc:1434] T a9bf3a093be8498c9dd5511b53436b16 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251027 21:08:03.597489 10031 tablet_bootstrap.cc:492] T 26872be132fb46ee9e79a9c2e62374ef P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:03.598924 10031 tablet_bootstrap.cc:654] T 26872be132fb46ee9e79a9c2e62374ef P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.599705 9308 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet e6b58b1f4f63461aa96e416cae879be8 (table default.Table_f3c2de263e8849caaef87b570fc69459 [id=ef50ef2238804c98ad19c5b1c0b881c4]). Expected version 1 got 0
I20251027 21:08:03.602691 10040 tablet_replica.cc:333] T c5f2e60caa824b5db2330b423221873c P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:03.602663 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet fe24af9838954a9bb680474800642e62 (table default.Table_fc695aa989384c1ba355a09381428926 [id=dae1114f6b50443f8f6b79d18619d868]) successfully deleted
I20251027 21:08:03.602835 10040 raft_consensus.cc:2243] T c5f2e60caa824b5db2330b423221873c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251027 21:08:03.602880 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 43611e33512b49d58485cf15de788be7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43611e33512b49d58485cf15de788be7 already in progress: creating tablet
I20251027 21:08:03.602947 10040 raft_consensus.cc:2272] T c5f2e60caa824b5db2330b423221873c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.603044 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet a4d26f73b8534f08bcaa386d4de75c01 with delete_type TABLET_DATA_DELETED (replaced by tablet db2735604dc14039810df4ecd330c671) from {username='slave'} at 127.0.0.1:34644
W20251027 21:08:03.603071 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet eec11e8fcb60425fb7fd742b6051fc2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eec11e8fcb60425fb7fd742b6051fc2d already in progress: creating tablet
I20251027 21:08:03.603336 10040 ts_tablet_manager.cc:1916] T c5f2e60caa824b5db2330b423221873c P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.603601 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet a033551a591146039b017b4193b221ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.603932 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet b99232a6018a4b1a850da954c551d38d with delete_type TABLET_DATA_DELETED (replaced by tablet 278d8f448ef144148cabfc3a57bd24fd) from {username='slave'} at 127.0.0.1:40732
W20251027 21:08:03.604627 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 43b5cef5fdcd4d2fa89875aa6d0efcd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43b5cef5fdcd4d2fa89875aa6d0efcd0 already in progress: creating tablet
W20251027 21:08:03.604879 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 2f641bc4dbed4c0abd0a55b253352507 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f641bc4dbed4c0abd0a55b253352507 already in progress: creating tablet
I20251027 21:08:03.605034 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet b99232a6018a4b1a850da954c551d38d with delete_type TABLET_DATA_DELETED (replaced by tablet 278d8f448ef144148cabfc3a57bd24fd) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.606602 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet b99232a6018a4b1a850da954c551d38d with delete_type TABLET_DATA_DELETED (replaced by tablet 278d8f448ef144148cabfc3a57bd24fd) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:03.610764 10032 tablet_bootstrap.cc:492] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:03.610899 10032 ts_tablet_manager.cc:1403] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.006s
I20251027 21:08:03.611565 10032 raft_consensus.cc:359] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.611742 10032 raft_consensus.cc:385] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.611786 10032 raft_consensus.cc:740] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.612454 10043 ts_tablet_manager.cc:1929] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20251027 21:08:03.611948 10032 consensus_queue.cc:260] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.612543 10043 log.cc:1199] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/d13d00353b6740b5bed5a153936d2059
I20251027 21:08:03.612874 10043 ts_tablet_manager.cc:1950] T d13d00353b6740b5bed5a153936d2059 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:03.613229 10032 ts_tablet_manager.cc:1434] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:03.613664 10032 tablet_bootstrap.cc:492] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:03.613715 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 67bb12de9f044c73ae9ab26d10498c90
I20251027 21:08:03.614233 9303 catalog_manager.cc:6719] ReplaceTablet: tablet 67bb12de9f044c73ae9ab26d10498c90 deleted and replaced by tablet ea02ab2fed634dbcba3ca8b04808b694
I20251027 21:08:03.614530 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 67bb12de9f044c73ae9ab26d10498c90 with delete_type TABLET_DATA_DELETED (replaced by tablet ea02ab2fed634dbcba3ca8b04808b694) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.614569 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7ed5ee38ed7a4cf1a5a706eb6e233d41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:03.615100 10032 tablet_bootstrap.cc:654] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:03.615168 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ed5ee38ed7a4cf1a5a706eb6e233d41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:03.616590 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 00c90ff84d3f4a7f92fda235416eed1d
I20251027 21:08:03.616765 10040 ts_tablet_manager.cc:1929] T c5f2e60caa824b5db2330b423221873c P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.616837 10040 log.cc:1199] T c5f2e60caa824b5db2330b423221873c P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/c5f2e60caa824b5db2330b423221873c
I20251027 21:08:03.617056 10040 ts_tablet_manager.cc:1950] T c5f2e60caa824b5db2330b423221873c P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:03.617341 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 00c90ff84d3f4a7f92fda235416eed1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.617274 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_fb6cfb8df03347efbf6dd0002e05bf86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:03.617781 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb6cfb8df03347efbf6dd0002e05bf86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:03.617873 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet c5f2e60caa824b5db2330b423221873c (table default.Table_a90c80308cb345d7abc6c44d3d4c70ff [id=0e7e4ea125894e78a2162e9e8063ee33]) successfully deleted
I20251027 21:08:03.619524 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet b748f59a56c54c2b850f5336a1d61c26
I20251027 21:08:03.620716 9312 master_service.cc:954] ReplaceTablet: received request to replace tablet 19f3e1a9db434c519e26e864eb790706 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:03.622089 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet b748f59a56c54c2b850f5336a1d61c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.622606 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet d13d00353b6740b5bed5a153936d2059 (table default.Table_c4824ada6d0f433dbb4770562a66e1ad [id=f2edcf33b0b74923b4ddbbc2419e14ca]) successfully deleted
I20251027 21:08:03.622804 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 00c90ff84d3f4a7f92fda235416eed1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:03.622993 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet b748f59a56c54c2b850f5336a1d61c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:03.622455 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet b748f59a56c54c2b850f5336a1d61c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.623287 10032 tablet_bootstrap.cc:492] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:03.623394 10032 ts_tablet_manager.cc:1403] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20251027 21:08:03.624078 10032 raft_consensus.cc:359] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:03.624418 10032 raft_consensus.cc:385] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.624469 10032 raft_consensus.cc:740] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.624619 10032 consensus_queue.cc:260] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:03.624330 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 00c90ff84d3f4a7f92fda235416eed1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.625204 10032 ts_tablet_manager.cc:1434] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:03.625454 10032 tablet_bootstrap.cc:492] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:03.626575 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet 67bb12de9f044c73ae9ab26d10498c90 with delete_type TABLET_DATA_DELETED (replaced by tablet ea02ab2fed634dbcba3ca8b04808b694) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:03.627138 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 67bb12de9f044c73ae9ab26d10498c90 with delete_type TABLET_DATA_DELETED (replaced by tablet ea02ab2fed634dbcba3ca8b04808b694) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.627234 10032 tablet_bootstrap.cc:654] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.629081 10829 raft_consensus.cc:493] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.629226 10829 raft_consensus.cc:515] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:03.629525 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet c0857f89bc74495081d35ee55e785835 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0857f89bc74495081d35ee55e785835 already in progress: creating tablet
I20251027 21:08:03.629776 10829 leader_election.cc:290] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:03.630108 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet b99232a6018a4b1a850da954c551d38d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b99232a6018a4b1a850da954c551d38d already in progress: creating tablet
W20251027 21:08:03.630896 9710 leader_election.cc:343] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: c4996d8fae094db8a96599b9057f1649
W20251027 21:08:03.631058 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 00c90ff84d3f4a7f92fda235416eed1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00c90ff84d3f4a7f92fda235416eed1d already in progress: creating tablet
W20251027 21:08:03.633037 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet b748f59a56c54c2b850f5336a1d61c26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b748f59a56c54c2b850f5336a1d61c26 already in progress: creating tablet
W20251027 21:08:03.633085 9711 leader_election.cc:343] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: c4996d8fae094db8a96599b9057f1649
I20251027 21:08:03.633162 9711 leader_election.cc:304] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:03.633528 10829 raft_consensus.cc:2749] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:03.634006 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 67bb12de9f044c73ae9ab26d10498c90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67bb12de9f044c73ae9ab26d10498c90 already in progress: creating tablet
I20251027 21:08:03.635632 10907 raft_consensus.cc:493] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.635782 10907 raft_consensus.cc:515] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:03.636440 10907 leader_election.cc:290] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:03.637948 9568 leader_election.cc:343] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 26872be132fb46ee9e79a9c2e62374ef
W20251027 21:08:03.638396 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c5bc4ee5ec5f4a039356ac8841f5b836 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5bc4ee5ec5f4a039356ac8841f5b836 already in progress: creating tablet
I20251027 21:08:03.638832 10040 tablet_replica.cc:333] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:03.639015 10040 raft_consensus.cc:2243] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251027 21:08:03.639067 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 1391ad8a22944ab7a8f45aceec19008b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1391ad8a22944ab7a8f45aceec19008b already in progress: creating tablet
I20251027 21:08:03.639123 10040 raft_consensus.cc:2272] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.639447 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26872be132fb46ee9e79a9c2e62374ef" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:03.639945 9566 leader_election.cc:343] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.640054 9566 leader_election.cc:304] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.640442 10907 raft_consensus.cc:2749] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:03.639528 10040 ts_tablet_manager.cc:1916] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.641789 10907 raft_consensus.cc:493] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.641920 10907 raft_consensus.cc:515] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.642585 10907 leader_election.cc:290] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:03.643399 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71dc6b7a6ca146c6b0c58d9ad0ecbefa" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:03.643711 9568 leader_election.cc:343] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 71dc6b7a6ca146c6b0c58d9ad0ecbefa
W20251027 21:08:03.643910 9566 leader_election.cc:343] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.643999 9566 leader_election.cc:304] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.644367 10907 raft_consensus.cc:2749] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:03.651511 9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.124:45071, user_credentials={real_user=slave}} blocked reactor thread for 53020us
I20251027 21:08:03.659603 10032 tablet_bootstrap.cc:492] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:03.659835 10032 ts_tablet_manager.cc:1403] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.002s
I20251027 21:08:03.660785 10032 raft_consensus.cc:359] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.661051 10032 raft_consensus.cc:385] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.661154 10032 raft_consensus.cc:740] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.661450 10032 consensus_queue.cc:260] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.662531 10032 ts_tablet_manager.cc:1434] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:08:03.663504 10032 tablet_bootstrap.cc:492] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:08:03.675179 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d7955e31f10449718964a8ced3c18672 consensus state has no leader
I20251027 21:08:03.676432 10031 tablet_bootstrap.cc:492] T 26872be132fb46ee9e79a9c2e62374ef P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:03.676571 10031 ts_tablet_manager.cc:1403] T 26872be132fb46ee9e79a9c2e62374ef P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.079s user 0.009s sys 0.025s
I20251027 21:08:03.677361 10041 tablet_replica.cc:333] T cef2bfee222f45e6b6f20c6d63ff44d4 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:03.679941 10041 raft_consensus.cc:2243] T cef2bfee222f45e6b6f20c6d63ff44d4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.680110 10041 raft_consensus.cc:2272] T cef2bfee222f45e6b6f20c6d63ff44d4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.681439 10041 ts_tablet_manager.cc:1916] T cef2bfee222f45e6b6f20c6d63ff44d4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.683151 10031 raft_consensus.cc:359] T 26872be132fb46ee9e79a9c2e62374ef P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:03.683353 10031 raft_consensus.cc:385] T 26872be132fb46ee9e79a9c2e62374ef P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.683400 10031 raft_consensus.cc:740] T 26872be132fb46ee9e79a9c2e62374ef P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.684229 10040 ts_tablet_manager.cc:1929] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.684332 10040 log.cc:1199] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/b719dc40d7e348608cdfc78401c19824
I20251027 21:08:03.684640 10040 ts_tablet_manager.cc:1950] T b719dc40d7e348608cdfc78401c19824 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:03.685822 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet b719dc40d7e348608cdfc78401c19824 (table default.Table_41078339d0834d069db2e8ae485bcc89 [id=f641ffdf5fd7497096643ea3278c49d0]) successfully deleted
I20251027 21:08:03.686266 10031 consensus_queue.cc:260] T 26872be132fb46ee9e79a9c2e62374ef P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:03.687580 10031 ts_tablet_manager.cc:1434] T 26872be132fb46ee9e79a9c2e62374ef P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251027 21:08:03.687808 10031 tablet_bootstrap.cc:492] T 630ae4d66f99474382814cc32cc60ea0 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:03.689236 10032 tablet_bootstrap.cc:654] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.699231 10040 tablet_replica.cc:333] T 3103c8e5a7c744be90be097179904ba3 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:03.699405 10040 raft_consensus.cc:2243] T 3103c8e5a7c744be90be097179904ba3 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.699550 10040 raft_consensus.cc:2272] T 3103c8e5a7c744be90be097179904ba3 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.702453 10031 tablet_bootstrap.cc:654] T 630ae4d66f99474382814cc32cc60ea0 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.702553 10040 ts_tablet_manager.cc:1916] T 3103c8e5a7c744be90be097179904ba3 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.708636 10041 ts_tablet_manager.cc:1929] T cef2bfee222f45e6b6f20c6d63ff44d4 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.708741 10041 log.cc:1199] T cef2bfee222f45e6b6f20c6d63ff44d4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/cef2bfee222f45e6b6f20c6d63ff44d4
I20251027 21:08:03.709046 10041 ts_tablet_manager.cc:1950] T cef2bfee222f45e6b6f20c6d63ff44d4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:03.709311 10032 tablet_bootstrap.cc:492] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:03.709462 10032 ts_tablet_manager.cc:1403] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.002s
I20251027 21:08:03.710191 10032 raft_consensus.cc:359] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.710450 10032 raft_consensus.cc:385] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.710552 10032 raft_consensus.cc:740] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.710824 10032 consensus_queue.cc:260] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.712019 10032 ts_tablet_manager.cc:1434] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:03.714301 9306 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 29693cf06bc24930870dc60457126519 (table default.Table_c09cc86fc2a64450af56fc584302da69 [id=d46947a29ddf4d94b5a9da22d9e5383e]). Expected version 3 got 0
I20251027 21:08:03.714490 9306 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet b8856c5dc8f144358445165324bfb477 (table default.Table_203191e192be4e3a9a7477b4313ef4f4 [id=5da6bd4463e24c03accd9985f72b6ebb]). Expected version 3 got 0
I20251027 21:08:03.714922 10032 tablet_bootstrap.cc:492] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:03.717056 10032 tablet_bootstrap.cc:654] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.719046 10040 ts_tablet_manager.cc:1929] T 3103c8e5a7c744be90be097179904ba3 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.720533 9306 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet b8856c5dc8f144358445165324bfb477 (table default.Table_203191e192be4e3a9a7477b4313ef4f4 [id=5da6bd4463e24c03accd9985f72b6ebb]). Expected version 3 got 0
I20251027 21:08:03.721915 9306 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 9fa39697360c46138ef4cd21e8c7a950 (table default.Table_4ba4c823d97045959f6926a92a489794 [id=434ed8c436c149ef9c7eb86ad252291d]). Expected version 3 got 0
I20251027 21:08:03.724565 10040 log.cc:1199] T 3103c8e5a7c744be90be097179904ba3 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/3103c8e5a7c744be90be097179904ba3
I20251027 21:08:03.724859 10040 ts_tablet_manager.cc:1950] T 3103c8e5a7c744be90be097179904ba3 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:03.725232 10040 tablet_replica.cc:333] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:03.725405 10040 raft_consensus.cc:2243] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.725435 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 3103c8e5a7c744be90be097179904ba3 (table default.Table_c04ff0f25ba04d25958346fc631b2802 [id=a6a32b47c2354a2d9535ff2b48d506de]) successfully deleted
I20251027 21:08:03.725536 10040 raft_consensus.cc:2272] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.726001 10040 ts_tablet_manager.cc:1916] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.726661 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet aabaa34787cf4d179822269b7f323314 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.727123 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 631ff72613734913818a3cad24f20dbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.727741 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c34c8f1433b40d491ae05e7c6e60ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.728312 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4f5e79447c4b4bab8c8c12cf2adab418"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:03.729214 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet cef2bfee222f45e6b6f20c6d63ff44d4 (table default.Table_c3f0afa8e76249b69b9fbe4b3f67d020 [id=43518e3f60164df7a812afb41f6c4936]) successfully deleted
W20251027 21:08:03.729390 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f5e79447c4b4bab8c8c12cf2adab418 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:03.723059 10031 tablet_bootstrap.cc:492] T 630ae4d66f99474382814cc32cc60ea0 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:03.730254 10031 ts_tablet_manager.cc:1403] T 630ae4d66f99474382814cc32cc60ea0 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
W20251027 21:08:03.730850 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 1391ad8a22944ab7a8f45aceec19008b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1391ad8a22944ab7a8f45aceec19008b already in progress: creating tablet
I20251027 21:08:03.730870 10031 raft_consensus.cc:359] T 630ae4d66f99474382814cc32cc60ea0 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.731005 10031 raft_consensus.cc:385] T 630ae4d66f99474382814cc32cc60ea0 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.731055 10031 raft_consensus.cc:740] T 630ae4d66f99474382814cc32cc60ea0 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.731230 10031 consensus_queue.cc:260] T 630ae4d66f99474382814cc32cc60ea0 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.731621 10031 ts_tablet_manager.cc:1434] T 630ae4d66f99474382814cc32cc60ea0 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:03.732086 10031 tablet_bootstrap.cc:492] T dd7029986a5246ee823e5fdb50683118 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:03.733541 10955 raft_consensus.cc:493] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251027 21:08:03.733716 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d672118ecd6f4094855adf106e5f7115 consensus state has no leader
I20251027 21:08:03.733659 10955 raft_consensus.cc:515] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.734261 10955 leader_election.cc:290] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:03.734551 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aabaa34787cf4d179822269b7f323314" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:03.734642 10031 tablet_bootstrap.cc:654] T dd7029986a5246ee823e5fdb50683118 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.735219 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aabaa34787cf4d179822269b7f323314" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:03.735816 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6c4fc648fa04d9aa4a10b5b6b07cd21 consensus state has no leader
W20251027 21:08:03.736398 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b98a836363414da489fb4090abe5d088 consensus state has no leader
W20251027 21:08:03.737218 9859 leader_election.cc:343] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.737547 10029 tablet_bootstrap.cc:492] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
W20251027 21:08:03.737618 9860 leader_election.cc:343] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.737723 10029 ts_tablet_manager.cc:1403] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.250s user 0.020s sys 0.042s
I20251027 21:08:03.737735 9860 leader_election.cc:304] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.738197 10955 raft_consensus.cc:2749] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:03.738428 10029 raft_consensus.cc:359] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.738831 10029 raft_consensus.cc:385] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.738881 10029 raft_consensus.cc:740] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.739794 10029 consensus_queue.cc:260] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.741727 10029 ts_tablet_manager.cc:1434] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251027 21:08:03.741958 10029 tablet_bootstrap.cc:492] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:03.743892 10032 tablet_bootstrap.cc:492] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:03.744010 10032 ts_tablet_manager.cc:1403] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.001s
I20251027 21:08:03.744612 10032 raft_consensus.cc:359] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.744758 10032 raft_consensus.cc:385] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.744799 10032 raft_consensus.cc:740] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.745229 10032 consensus_queue.cc:260] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.745793 10032 ts_tablet_manager.cc:1434] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:03.746102 10032 tablet_bootstrap.cc:492] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:03.747993 10032 tablet_bootstrap.cc:654] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.748270 10029 tablet_bootstrap.cc:654] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.750730 10032 tablet_bootstrap.cc:492] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:03.750895 10032 ts_tablet_manager.cc:1403] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20251027 21:08:03.751606 10032 raft_consensus.cc:359] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.751842 10032 raft_consensus.cc:385] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.751952 10032 raft_consensus.cc:740] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.751929 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_12e5e752397f4aadae74519cfdf8f4d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:03.752228 10032 consensus_queue.cc:260] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
W20251027 21:08:03.752533 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12e5e752397f4aadae74519cfdf8f4d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:03.753922 10032 ts_tablet_manager.cc:1434] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251027 21:08:03.754091 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_cc55a31e64194c35a8e42c5cdf11600d" } new_table_name: "default.Table_15aa76b39480416498545595ddf5cf1b" modify_external_catalogs: true
I20251027 21:08:03.755652 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3e61b43276ac41a2b626158c42642384"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:03.756250 10040 ts_tablet_manager.cc:1929] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251027 21:08:03.756268 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e61b43276ac41a2b626158c42642384 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:03.756335 10040 log.cc:1199] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/a9339c6b1154400eace53d765528fdf9
I20251027 21:08:03.756655 10040 ts_tablet_manager.cc:1950] T a9339c6b1154400eace53d765528fdf9 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:03.756891 10907 raft_consensus.cc:493] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.757004 10907 raft_consensus.cc:515] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.757474 10907 leader_election.cc:290] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:03.758562 9568 leader_election.cc:343] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 24ec2e0ba1994ab59c53977583bd2058
I20251027 21:08:03.758955 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24ec2e0ba1994ab59c53977583bd2058" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:03.759814 9566 leader_election.cc:343] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.759908 9566 leader_election.cc:304] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.760221 10859 raft_consensus.cc:493] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.760310 10859 raft_consensus.cc:515] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.760790 10859 leader_election.cc:290] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:03.760893 10859 raft_consensus.cc:2749] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:03.761334 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccf10ef559f74e2db28023fd520c8de9" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:03.761376 9304 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet f2710f265ae9485dafec389168328067 (table default.Table_eb44e5cb7bc24597a807d81c1207a162 [id=27d1cff8fc7940c9abb741c8c7d87e24]). Expected version 2 got 0
I20251027 21:08:03.761494 9924 raft_consensus.cc:2468] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:03.761886 9568 leader_election.cc:304] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:03.762101 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a8719fbbe9b6463bb1dd10f1b8f92cb8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:03.762180 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccf10ef559f74e2db28023fd520c8de9" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:03.762524 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6b527fc023dd45f4ab080eddeeb8bb76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:03.762661 10907 raft_consensus.cc:2804] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:03.762733 10907 raft_consensus.cc:493] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:03.762780 10907 raft_consensus.cc:3060] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:03.762954 9308 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 45568792d7954b498581ce69fb611948 (table default.Table_3e3e0c17a3bf452fbb217c6e3ee8bdcb [id=2eb093cd58e643a58b14e564d5914fbc]). Expected version 1 got 0
W20251027 21:08:03.763029 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b527fc023dd45f4ab080eddeeb8bb76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:03.763206 9308 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet b8856c5dc8f144358445165324bfb477 (table default.Table_203191e192be4e3a9a7477b4313ef4f4 [id=5da6bd4463e24c03accd9985f72b6ebb]). Expected version 3 got 0
I20251027 21:08:03.764896 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f2e9f61a51bd4e1796f6bda56f48a6a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:03.765285 10907 raft_consensus.cc:515] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:03.765358 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2e9f61a51bd4e1796f6bda56f48a6a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:03.766072 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccf10ef559f74e2db28023fd520c8de9" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:03.766255 9924 raft_consensus.cc:3060] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:03.766398 10907 leader_election.cc:290] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:03.766992 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccf10ef559f74e2db28023fd520c8de9" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:03.767316 9566 leader_election.cc:343] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:03.767732 9566 leader_election.cc:343] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.768565 10040 tablet_replica.cc:333] T 94f2dbf47a5c4458bee54b26f272156e P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:03.768905 9924 raft_consensus.cc:2468] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:03.769059 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet adaa4f66bbab491f8030d439d85cc7d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.769229 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 343eaeca24dc4816ae032c7e674e9412 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:50 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.769384 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet c4996d8fae094db8a96599b9057f1649 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.769527 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 648f506ec87c4fb8a3f6d4591873dbb8 with delete_type TABLET_DATA_DELETED (replaced by tablet 96a680e8f16343b1908829a67e960b60) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.769435 10040 raft_consensus.cc:2243] T 94f2dbf47a5c4458bee54b26f272156e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.769876 10040 raft_consensus.cc:2272] T 94f2dbf47a5c4458bee54b26f272156e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.769904 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet a9339c6b1154400eace53d765528fdf9 (table default.Table_5a6211c8465040dbad1cc86377a370ea [id=91fb9f95fe9343fca1d8cb0d0f781afb]) successfully deleted
I20251027 21:08:03.770290 9568 leader_election.cc:304] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.770507 10907 raft_consensus.cc:2804] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:03.770928 10907 raft_consensus.cc:697] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:03.771193 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ec2e0ba1994ab59c53977583bd2058 with delete_type TABLET_DATA_DELETED (replaced by tablet 815e12edcd3847ff8485ba768291a824) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:03.771281 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet dd7029986a5246ee823e5fdb50683118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:03.771281 9611 tablet_service.cc:1552] Processing DeleteTablet for tablet 26872be132fb46ee9e79a9c2e62374ef with delete_type TABLET_DATA_DELETED (replaced by tablet 6409f45a907b4389bac84093cdc3c30f) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:03.771447 10907 consensus_queue.cc:237] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.771557 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet 22d861810ed846c3b77a452d05e93d72 with delete_type TABLET_DATA_DELETED (replaced by tablet c0e2772d66404d74891e0893c49a66e0) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:03.771711 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 53cac7817db440e79e7416d625a9b2c8 with delete_type TABLET_DATA_DELETED (replaced by tablet ed66f9eeeb2b4b4bb77062c241d2140f) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:03.771843 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:03.771979 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 2532bf7190064443a898e47e843fd9ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:03.772154 10043 tablet_replica.cc:333] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:03.772312 10043 raft_consensus.cc:2243] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.772424 10043 raft_consensus.cc:2272] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.773020 10043 ts_tablet_manager.cc:1916] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.780040 10955 raft_consensus.cc:493] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.780161 10955 raft_consensus.cc:515] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.780333 10043 ts_tablet_manager.cc:1929] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.780411 10043 log.cc:1199] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/24ec2e0ba1994ab59c53977583bd2058
I20251027 21:08:03.780632 10043 ts_tablet_manager.cc:1950] T 24ec2e0ba1994ab59c53977583bd2058 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:03.780699 10955 leader_election.cc:290] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:03.781461 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 24ec2e0ba1994ab59c53977583bd2058 (table default.Table_2992b8e207024e06957ce7befb657acb [id=decffeb1481e4d798cffd8c9883be59b]) successfully deleted
I20251027 21:08:03.782435 10043 tablet_replica.cc:333] T dd7029986a5246ee823e5fdb50683118 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:03.782788 10043 raft_consensus.cc:2243] T dd7029986a5246ee823e5fdb50683118 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.783119 10043 raft_consensus.cc:2272] T dd7029986a5246ee823e5fdb50683118 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.783118 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76e4fb2bea7f4a4696fe9f6bb892bccc" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:03.783735 9859 leader_election.cc:343] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.784385 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76e4fb2bea7f4a4696fe9f6bb892bccc" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:03.784958 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8719fbbe9b6463bb1dd10f1b8f92cb8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:03.786831 9303 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet b8856c5dc8f144358445165324bfb477 (table default.Table_203191e192be4e3a9a7477b4313ef4f4 [id=5da6bd4463e24c03accd9985f72b6ebb]). Expected version 3 got 0
I20251027 21:08:03.788256 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b1f06adb64714ba5818da1b0c060fd16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:03.788748 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1f06adb64714ba5818da1b0c060fd16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:03.788805 9307 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet f2710f265ae9485dafec389168328067 (table default.Table_eb44e5cb7bc24597a807d81c1207a162 [id=27d1cff8fc7940c9abb741c8c7d87e24]). Expected version 2 got 0
I20251027 21:08:03.789605 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b0bf4b9e330b4ec48becdc293dcef72a" } modify_external_catalogs: true
I20251027 21:08:03.789767 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b0bf4b9e330b4ec48becdc293dcef72a" } modify_external_catalogs: true
I20251027 21:08:03.790364 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 19f3e1a9db434c519e26e864eb790706
I20251027 21:08:03.790889 9312 catalog_manager.cc:6719] ReplaceTablet: tablet 19f3e1a9db434c519e26e864eb790706 deleted and replaced by tablet 1778f4a7f81541bc8932201ead4189b4
I20251027 21:08:03.790984 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 19f3e1a9db434c519e26e864eb790706 with delete_type TABLET_DATA_DELETED (replaced by tablet 1778f4a7f81541bc8932201ead4189b4) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:03.791111 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 19f3e1a9db434c519e26e864eb790706 with delete_type TABLET_DATA_DELETED (replaced by tablet 1778f4a7f81541bc8932201ead4189b4) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.791203 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c564272cb5b445acbb5a98d619938f00" } modify_external_catalogs: true
I20251027 21:08:03.791357 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c564272cb5b445acbb5a98d619938f00" } modify_external_catalogs: true
I20251027 21:08:03.792343 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 26872be132fb46ee9e79a9c2e62374ef with delete_type TABLET_DATA_DELETED (replaced by tablet 6409f45a907b4389bac84093cdc3c30f) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.792534 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet a9bf3a093be8498c9dd5511b53436b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.792683 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 630ae4d66f99474382814cc32cc60ea0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2f641bc4dbed4c0abd0a55b253352507) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.792817 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 19f3e1a9db434c519e26e864eb790706 with delete_type TABLET_DATA_DELETED (replaced by tablet 1778f4a7f81541bc8932201ead4189b4) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.794704 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3478938a9d2b44e7875de2ae7a4d471b" } modify_external_catalogs: true
I20251027 21:08:03.794868 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3478938a9d2b44e7875de2ae7a4d471b" } modify_external_catalogs: true
I20251027 21:08:03.795387 10040 ts_tablet_manager.cc:1916] T 94f2dbf47a5c4458bee54b26f272156e P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:03.796000 9860 leader_election.cc:343] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.796106 9860 leader_election.cc:304] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.796437 10955 raft_consensus.cc:2749] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:03.798322 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eaed784d81e74f9cb3514d3b55fdc0d1 consensus state has no leader
I20251027 21:08:03.805090 10032 tablet_bootstrap.cc:492] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:03.806900 10043 ts_tablet_manager.cc:1916] T dd7029986a5246ee823e5fdb50683118 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.810230 10032 tablet_bootstrap.cc:654] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.816911 10041 tablet_replica.cc:333] T f775a7e42bdc4910bdb604b26a28a7f4 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
W20251027 21:08:03.817066 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet c5bc4ee5ec5f4a039356ac8841f5b836 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5bc4ee5ec5f4a039356ac8841f5b836 already in progress: creating tablet
I20251027 21:08:03.817092 10041 raft_consensus.cc:2243] T f775a7e42bdc4910bdb604b26a28a7f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.817296 10041 raft_consensus.cc:2272] T f775a7e42bdc4910bdb604b26a28a7f4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.817797 10041 ts_tablet_manager.cc:1916] T f775a7e42bdc4910bdb604b26a28a7f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.818082 10040 ts_tablet_manager.cc:1929] T 94f2dbf47a5c4458bee54b26f272156e P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.818208 10040 log.cc:1199] T 94f2dbf47a5c4458bee54b26f272156e P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/94f2dbf47a5c4458bee54b26f272156e
I20251027 21:08:03.818490 10040 ts_tablet_manager.cc:1950] T 94f2dbf47a5c4458bee54b26f272156e P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:03.819001 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 94f2dbf47a5c4458bee54b26f272156e (table default.Table_7954e74d079b416cac73f9a77979c8b3 [id=0103a75b0a714d3c9a533d94b6bda41f]) successfully deleted
I20251027 21:08:03.819231 10907 raft_consensus.cc:493] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.819342 10907 raft_consensus.cc:515] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.820091 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f1d1bd52556463fa306e51eb253af40" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:03.820241 10907 leader_election.cc:290] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:03.820255 9924 raft_consensus.cc:2468] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:03.820333 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f1d1bd52556463fa306e51eb253af40" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:03.820827 9566 leader_election.cc:343] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:03.820986 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 87612b8bd8a04fd9b32f2df3f9e0d997 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87612b8bd8a04fd9b32f2df3f9e0d997 already in progress: creating tablet
I20251027 21:08:03.821218 10040 tablet_replica.cc:333] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:03.821218 9568 leader_election.cc:304] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.821278 10031 tablet_bootstrap.cc:492] T dd7029986a5246ee823e5fdb50683118 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:03.821363 10040 raft_consensus.cc:2243] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.821377 10031 ts_tablet_manager.cc:1403] T dd7029986a5246ee823e5fdb50683118 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.089s user 0.005s sys 0.004s
I20251027 21:08:03.821496 10040 raft_consensus.cc:2272] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.821566 10907 raft_consensus.cc:2804] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:03.821630 10907 raft_consensus.cc:493] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:03.821693 10907 raft_consensus.cc:3060] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:03.821928 10040 ts_tablet_manager.cc:1916] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.822077 10031 raft_consensus.cc:359] T dd7029986a5246ee823e5fdb50683118 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.822234 10031 raft_consensus.cc:385] T dd7029986a5246ee823e5fdb50683118 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.822280 10031 raft_consensus.cc:740] T dd7029986a5246ee823e5fdb50683118 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.822438 10955 raft_consensus.cc:493] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.822458 10031 consensus_queue.cc:260] T dd7029986a5246ee823e5fdb50683118 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.822536 10955 raft_consensus.cc:515] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.823041 10955 leader_election.cc:290] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:03.825189 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b8156d9ab2e4dfba383ddc72137aab4" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:03.825313 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b8156d9ab2e4dfba383ddc72137aab4" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:03.825686 9859 leader_election.cc:343] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.826588 10907 raft_consensus.cc:515] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.826956 10031 ts_tablet_manager.cc:1434] T dd7029986a5246ee823e5fdb50683118 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251027 21:08:03.827121 10907 leader_election.cc:290] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:03.827397 10031 tablet_bootstrap.cc:492] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:03.827999 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f1d1bd52556463fa306e51eb253af40" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:03.828281 9860 leader_election.cc:343] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.828375 9860 leader_election.cc:304] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:03.828398 9566 leader_election.cc:343] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.828992 10040 ts_tablet_manager.cc:1929] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20251027 21:08:03.829016 10955 raft_consensus.cc:2749] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:03.829073 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f1d1bd52556463fa306e51eb253af40" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:03.829150 10040 log.cc:1199] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/d13d00353b6740b5bed5a153936d2059
I20251027 21:08:03.829216 9924 raft_consensus.cc:3060] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:03.829336 10031 tablet_bootstrap.cc:654] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.829653 10040 ts_tablet_manager.cc:1950] T d13d00353b6740b5bed5a153936d2059 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:03.830296 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet d13d00353b6740b5bed5a153936d2059 (table default.Table_c4824ada6d0f433dbb4770562a66e1ad [id=f2edcf33b0b74923b4ddbbc2419e14ca]) successfully deleted
I20251027 21:08:03.830840 10955 raft_consensus.cc:493] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251027 21:08:03.830912 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 8db666d9b7eb458b843ab5910cd7dff4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8db666d9b7eb458b843ab5910cd7dff4 already in progress: creating tablet
I20251027 21:08:03.830956 10955 raft_consensus.cc:515] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.831408 10955 leader_election.cc:290] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:03.831741 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a033551a591146039b017b4193b221ca" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:03.831785 9924 raft_consensus.cc:2468] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:03.832130 9568 leader_election.cc:304] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:03.832299 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c0857f89bc74495081d35ee55e785835 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0857f89bc74495081d35ee55e785835 already in progress: creating tablet
W20251027 21:08:03.832432 9859 leader_election.cc:343] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.832477 10907 raft_consensus.cc:2804] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:03.832572 10907 raft_consensus.cc:697] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:03.832631 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a033551a591146039b017b4193b221ca" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:03.833293 9860 leader_election.cc:343] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.833387 9860 leader_election.cc:304] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.833823 10955 raft_consensus.cc:2749] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:03.832777 10907 consensus_queue.cc:237] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:03.834466 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet b99232a6018a4b1a850da954c551d38d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b99232a6018a4b1a850da954c551d38d already in progress: creating tablet
I20251027 21:08:03.833735 10029 tablet_bootstrap.cc:492] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:03.835114 10029 ts_tablet_manager.cc:1403] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.000s
W20251027 21:08:03.835515 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 67bb12de9f044c73ae9ab26d10498c90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67bb12de9f044c73ae9ab26d10498c90 already in progress: creating tablet
I20251027 21:08:03.835842 10029 raft_consensus.cc:359] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.835987 10029 raft_consensus.cc:385] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.836009 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet dd7029986a5246ee823e5fdb50683118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.836036 10029 raft_consensus.cc:740] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.836552 10029 consensus_queue.cc:260] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:03.836759 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 00c90ff84d3f4a7f92fda235416eed1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00c90ff84d3f4a7f92fda235416eed1d already in progress: creating tablet
I20251027 21:08:03.838230 10029 ts_tablet_manager.cc:1434] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20251027 21:08:03.838335 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet b748f59a56c54c2b850f5336a1d61c26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b748f59a56c54c2b850f5336a1d61c26 already in progress: creating tablet
I20251027 21:08:03.838627 10040 tablet_replica.cc:333] T adaa4f66bbab491f8030d439d85cc7d1 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:03.839041 10040 raft_consensus.cc:2243] T adaa4f66bbab491f8030d439d85cc7d1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.839210 10040 raft_consensus.cc:2272] T adaa4f66bbab491f8030d439d85cc7d1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.839319 10029 tablet_bootstrap.cc:492] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:03.839781 10040 ts_tablet_manager.cc:1916] T adaa4f66bbab491f8030d439d85cc7d1 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.839884 9312 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet fa5d83d6d18c4c4cbfad2157559517f6
I20251027 21:08:03.840816 10029 tablet_bootstrap.cc:654] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.841468 10031 tablet_bootstrap.cc:492] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:03.841611 10031 ts_tablet_manager.cc:1403] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.001s
I20251027 21:08:03.842267 10031 raft_consensus.cc:359] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.842454 10031 raft_consensus.cc:385] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.842531 10031 raft_consensus.cc:740] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.842749 10031 consensus_queue.cc:260] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.843175 10031 ts_tablet_manager.cc:1434] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 21:08:03.843437 10031 tablet_bootstrap.cc:492] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:03.845024 10031 tablet_bootstrap.cc:654] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.850114 10032 tablet_bootstrap.cc:492] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:03.850315 10032 ts_tablet_manager.cc:1403] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.045s user 0.001s sys 0.005s
I20251027 21:08:03.851089 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 22d861810ed846c3b77a452d05e93d72 with delete_type TABLET_DATA_DELETED (replaced by tablet c0e2772d66404d74891e0893c49a66e0) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.851022 10032 raft_consensus.cc:359] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.852202 10043 ts_tablet_manager.cc:1929] T dd7029986a5246ee823e5fdb50683118 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.858741 10040 ts_tablet_manager.cc:1929] T adaa4f66bbab491f8030d439d85cc7d1 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.858834 10040 log.cc:1199] T adaa4f66bbab491f8030d439d85cc7d1 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/adaa4f66bbab491f8030d439d85cc7d1
I20251027 21:08:03.859104 10040 ts_tablet_manager.cc:1950] T adaa4f66bbab491f8030d439d85cc7d1 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:03.859122 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b64e1c03b47345e2adc8d87940f6280e" } modify_external_catalogs: true
I20251027 21:08:03.859289 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b64e1c03b47345e2adc8d87940f6280e" } modify_external_catalogs: true
I20251027 21:08:03.859586 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet adaa4f66bbab491f8030d439d85cc7d1 (table default.Table_3f5e27698f4c4da88d9bacc9dd719304 [id=d174e821ea8641fc8b6658d5c61c7e8a]) successfully deleted
I20251027 21:08:03.860821 10040 tablet_replica.cc:333] T 343eaeca24dc4816ae032c7e674e9412 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:03.860941 10955 raft_consensus.cc:493] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.860989 10040 raft_consensus.cc:2243] T 343eaeca24dc4816ae032c7e674e9412 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.861116 10040 raft_consensus.cc:2272] T 343eaeca24dc4816ae032c7e674e9412 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.861065 10955 raft_consensus.cc:515] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.861402 10031 tablet_bootstrap.cc:492] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:03.861507 10031 ts_tablet_manager.cc:1403] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.001s
I20251027 21:08:03.861676 10955 leader_election.cc:290] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:03.861776 10040 ts_tablet_manager.cc:1916] T 343eaeca24dc4816ae032c7e674e9412 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.862440 10031 raft_consensus.cc:359] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.862603 10031 raft_consensus.cc:385] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.862655 10031 raft_consensus.cc:740] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.862828 10031 consensus_queue.cc:260] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.863221 10031 ts_tablet_manager.cc:1434] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:03.863467 10031 tablet_bootstrap.cc:492] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:03.863886 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29693cf06bc24930870dc60457126519" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:03.864904 9859 leader_election.cc:343] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.864979 10031 tablet_bootstrap.cc:654] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.869985 10040 ts_tablet_manager.cc:1929] T 343eaeca24dc4816ae032c7e674e9412 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.870077 10040 log.cc:1199] T 343eaeca24dc4816ae032c7e674e9412 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/343eaeca24dc4816ae032c7e674e9412
I20251027 21:08:03.870376 10040 ts_tablet_manager.cc:1950] T 343eaeca24dc4816ae032c7e674e9412 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:03.871043 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 343eaeca24dc4816ae032c7e674e9412 (table default.Table_546a5293975148e1841c8fff9eeac42a [id=b4569b8a023945989439915acb6097e4]) successfully deleted
I20251027 21:08:03.871469 10040 tablet_replica.cc:333] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:03.871623 10040 raft_consensus.cc:2243] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.871742 10040 raft_consensus.cc:2272] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.872336 10040 ts_tablet_manager.cc:1916] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.875038 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e31d4ff2c48e4eea9fe1f13da4fdf218" } modify_external_catalogs: true
I20251027 21:08:03.875286 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e31d4ff2c48e4eea9fe1f13da4fdf218" } modify_external_catalogs: true
I20251027 21:08:03.878712 10031 tablet_bootstrap.cc:492] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:03.878839 10031 ts_tablet_manager.cc:1403] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20251027 21:08:03.863333 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29693cf06bc24930870dc60457126519" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:03.872519 10859 raft_consensus.cc:493] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.879729 10859 raft_consensus.cc:515] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.880316 10859 leader_election.cc:290] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:03.880870 9860 leader_election.cc:343] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.880955 10031 raft_consensus.cc:359] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.881004 9860 leader_election.cc:304] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.881088 10031 raft_consensus.cc:385] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.881127 10031 raft_consensus.cc:740] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.881274 10031 consensus_queue.cc:260] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.881642 10031 ts_tablet_manager.cc:1434] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20251027 21:08:03.881757 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "345d3795a37b4a40b40967cfacf0f893" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:03.881853 10031 tablet_bootstrap.cc:492] T 84b9de82d55447bba2a897edb683e5c5 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:03.881851 10955 raft_consensus.cc:2749] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:03.881904 9924 raft_consensus.cc:2468] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:03.882262 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "345d3795a37b4a40b40967cfacf0f893" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:03.883231 10031 tablet_bootstrap.cc:654] T 84b9de82d55447bba2a897edb683e5c5 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.883970 10040 ts_tablet_manager.cc:1929] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.884052 10040 log.cc:1199] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/c4996d8fae094db8a96599b9057f1649
I20251027 21:08:03.884342 10040 ts_tablet_manager.cc:1950] T c4996d8fae094db8a96599b9057f1649 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
W20251027 21:08:03.884403 9566 leader_election.cc:343] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.884903 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet c4996d8fae094db8a96599b9057f1649 (table default.Table_b462e4bf95dc4fc4ad3fd23b236fffa7 [id=68da6285e1d247f1950a0b4929c885a4]) successfully deleted
I20251027 21:08:03.884965 9568 leader_election.cc:304] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.885172 10040 tablet_replica.cc:333] T 648f506ec87c4fb8a3f6d4591873dbb8 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:03.885327 10040 raft_consensus.cc:2243] T 648f506ec87c4fb8a3f6d4591873dbb8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.885437 10040 raft_consensus.cc:2272] T 648f506ec87c4fb8a3f6d4591873dbb8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.885655 10859 raft_consensus.cc:2804] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:03.885735 10859 raft_consensus.cc:493] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:03.885787 10859 raft_consensus.cc:3060] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:03.885937 10040 ts_tablet_manager.cc:1916] T 648f506ec87c4fb8a3f6d4591873dbb8 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.888787 10859 raft_consensus.cc:515] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.889673 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "345d3795a37b4a40b40967cfacf0f893" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:03.889854 9924 raft_consensus.cc:3060] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:03.889876 10859 leader_election.cc:290] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:03.890077 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "345d3795a37b4a40b40967cfacf0f893" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:03.890398 10043 log.cc:1199] T dd7029986a5246ee823e5fdb50683118 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/dd7029986a5246ee823e5fdb50683118
W20251027 21:08:03.890538 9566 leader_election.cc:343] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.890695 10043 ts_tablet_manager.cc:1950] T dd7029986a5246ee823e5fdb50683118 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:03.890918 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_47c1b3ca9dc541a6895e918264fd993f" } new_table_name: "default.Table_303556148a4445a191ca828fb6582bd2" modify_external_catalogs: true
I20251027 21:08:03.892284 10029 tablet_bootstrap.cc:492] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:03.892388 10029 ts_tablet_manager.cc:1403] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.000s
I20251027 21:08:03.892889 9924 raft_consensus.cc:2468] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:03.893728 10040 ts_tablet_manager.cc:1929] T 648f506ec87c4fb8a3f6d4591873dbb8 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.893821 10040 log.cc:1199] T 648f506ec87c4fb8a3f6d4591873dbb8 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/648f506ec87c4fb8a3f6d4591873dbb8
I20251027 21:08:03.894116 10040 ts_tablet_manager.cc:1950] T 648f506ec87c4fb8a3f6d4591873dbb8 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:03.894286 9568 leader_election.cc:304] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.894614 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 648f506ec87c4fb8a3f6d4591873dbb8 (table default.Table_a46825cc888a4a0898d6db70b02e5ab6 [id=8579d3ff46d64c75b1da213fdee9cc1b]) successfully deleted
I20251027 21:08:03.894632 10859 raft_consensus.cc:2804] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:03.894757 10859 raft_consensus.cc:697] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:03.895030 10859 consensus_queue.cc:237] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.895838 10031 tablet_bootstrap.cc:492] T 84b9de82d55447bba2a897edb683e5c5 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:03.895965 10031 ts_tablet_manager.cc:1403] T 84b9de82d55447bba2a897edb683e5c5 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20251027 21:08:03.896875 10031 raft_consensus.cc:359] T 84b9de82d55447bba2a897edb683e5c5 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.897019 10031 raft_consensus.cc:385] T 84b9de82d55447bba2a897edb683e5c5 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.897074 10031 raft_consensus.cc:740] T 84b9de82d55447bba2a897edb683e5c5 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.897270 10031 consensus_queue.cc:260] T 84b9de82d55447bba2a897edb683e5c5 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.898046 10031 ts_tablet_manager.cc:1434] T 84b9de82d55447bba2a897edb683e5c5 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:03.898849 10031 tablet_bootstrap.cc:492] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:03.899089 10040 tablet_replica.cc:333] T 26872be132fb46ee9e79a9c2e62374ef P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:03.899264 10040 raft_consensus.cc:2243] T 26872be132fb46ee9e79a9c2e62374ef P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.899392 10040 raft_consensus.cc:2272] T 26872be132fb46ee9e79a9c2e62374ef P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.899432 10029 raft_consensus.cc:359] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.899583 10029 raft_consensus.cc:385] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.899644 10029 raft_consensus.cc:740] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.900218 10040 ts_tablet_manager.cc:1916] T 26872be132fb46ee9e79a9c2e62374ef P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.851168 10032 raft_consensus.cc:385] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.900743 10032 raft_consensus.cc:740] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.900982 10955 raft_consensus.cc:493] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.901094 10955 raft_consensus.cc:515] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.901361 10029 consensus_queue.cc:260] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.901638 10955 leader_election.cc:290] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:03.902022 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2710f265ae9485dafec389168328067" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:03.902411 9860 leader_election.cc:343] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.902750 10043 tablet_replica.cc:333] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:03.902916 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet dd7029986a5246ee823e5fdb50683118 (table default.Table_79211c9993c54e4ea0455d01fec68b8e [id=421a64f7fb6f499bbf6f20fb38f736d0]) successfully deleted
I20251027 21:08:03.902918 10032 consensus_queue.cc:260] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:03.903797 10043 raft_consensus.cc:2243] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.903803 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2710f265ae9485dafec389168328067" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:03.904199 10043 raft_consensus.cc:2272] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.904861 10032 ts_tablet_manager.cc:1434] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.054s user 0.000s sys 0.002s
I20251027 21:08:03.905099 10032 tablet_bootstrap.cc:492] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:03.905589 10043 ts_tablet_manager.cc:1916] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.904354 10029 ts_tablet_manager.cc:1434] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251027 21:08:03.906884 10029 tablet_bootstrap.cc:492] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:03.909556 10907 raft_consensus.cc:493] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.909678 10907 raft_consensus.cc:515] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.910362 10029 tablet_bootstrap.cc:654] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:03.910893 9568 leader_election.cc:343] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: a339c0dee0994788b8644912f74c1b82
I20251027 21:08:03.911082 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 84b9de82d55447bba2a897edb683e5c5 with delete_type TABLET_DATA_DELETED (replaced by tablet befdd3a485a1426b8a3adf340d4eae4f) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.911172 10907 leader_election.cc:290] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:03.911278 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a339c0dee0994788b8644912f74c1b82" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:03.911679 9566 leader_election.cc:343] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.911777 9566 leader_election.cc:304] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.912151 10859 raft_consensus.cc:2749] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:03.913023 10032 tablet_bootstrap.cc:654] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.914007 9304 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 83f5fbf5757a49df93ab949b7657c61f (table default.Table_50b757fa69504e57872e1ecaee13583c [id=2719caa93b0945c59e6ad4d6f1e9e717]). Expected version 4 got 0
I20251027 21:08:03.918057 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 6bbee5f1fb704709b6f166a153e69808
I20251027 21:08:03.918176 10029 tablet_bootstrap.cc:492] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:03.918274 10029 ts_tablet_manager.cc:1403] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20251027 21:08:03.918897 10029 raft_consensus.cc:359] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.919037 10029 raft_consensus.cc:385] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.919080 10029 raft_consensus.cc:740] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.919263 10029 consensus_queue.cc:260] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.919891 10029 ts_tablet_manager.cc:1434] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:03.920118 10029 tablet_bootstrap.cc:492] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:03.920184 10829 raft_consensus.cc:493] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.920326 10829 raft_consensus.cc:515] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.920897 10829 leader_election.cc:290] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:03.921674 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bbee5f1fb704709b6f166a153e69808 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:03.921702 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22d861810ed846c3b77a452d05e93d72" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:03.921828 10029 tablet_bootstrap.cc:654] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.921875 9631 raft_consensus.cc:2468] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:08:03.922255 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bbee5f1fb704709b6f166a153e69808 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:03.922734 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bbee5f1fb704709b6f166a153e69808 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:03.923578 9710 leader_election.cc:304] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:03.924085 10829 raft_consensus.cc:2804] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:03.924156 10829 raft_consensus.cc:493] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:03.924216 10829 raft_consensus.cc:3060] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:03.926733 10829 raft_consensus.cc:515] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.927596 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22d861810ed846c3b77a452d05e93d72" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:03.927763 9631 raft_consensus.cc:3060] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:03.933735 10829 leader_election.cc:290] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:03.934204 9711 leader_election.cc:343] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 22d861810ed846c3b77a452d05e93d72
W20251027 21:08:03.934731 9711 leader_election.cc:343] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 22d861810ed846c3b77a452d05e93d72
I20251027 21:08:03.937902 9631 raft_consensus.cc:2468] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:03.938308 9710 leader_election.cc:304] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:03.938671 10829 raft_consensus.cc:2804] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:03.942831 10955 raft_consensus.cc:493] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:03.942961 10955 raft_consensus.cc:515] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:03.943523 10955 leader_election.cc:290] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:03.943944 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96a680e8f16343b1908829a67e960b60" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:03.943964 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96a680e8f16343b1908829a67e960b60" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:03.944240 10031 tablet_bootstrap.cc:654] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:03.944360 9860 leader_election.cc:343] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:03.944546 9859 leader_election.cc:343] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.944621 9859 leader_election.cc:304] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.944814 10955 raft_consensus.cc:2749] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:03.953313 10040 ts_tablet_manager.cc:1929] T 26872be132fb46ee9e79a9c2e62374ef P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.953442 10040 log.cc:1199] T 26872be132fb46ee9e79a9c2e62374ef P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/26872be132fb46ee9e79a9c2e62374ef
I20251027 21:08:03.956243 10040 ts_tablet_manager.cc:1950] T 26872be132fb46ee9e79a9c2e62374ef P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:03.956907 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 26872be132fb46ee9e79a9c2e62374ef (table default.Table_b23313ec4d7b4e32a4283dd404578616 [id=6a545a959eb74d0eb589f1be92a0bc0a]) successfully deleted
I20251027 21:08:03.957928 10040 tablet_replica.cc:333] T a9bf3a093be8498c9dd5511b53436b16 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:03.958432 10040 raft_consensus.cc:2243] T a9bf3a093be8498c9dd5511b53436b16 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.962360 10040 raft_consensus.cc:2272] T a9bf3a093be8498c9dd5511b53436b16 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.962602 10043 ts_tablet_manager.cc:1929] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.962877 10043 log.cc:1199] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/26872be132fb46ee9e79a9c2e62374ef
W20251027 21:08:03.963572 9859 leader_election.cc:343] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:03.963702 9859 leader_election.cc:304] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:03.963918 10955 raft_consensus.cc:2749] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:03.964182 10043 ts_tablet_manager.cc:1950] T 26872be132fb46ee9e79a9c2e62374ef P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:03.964468 10040 ts_tablet_manager.cc:1916] T a9bf3a093be8498c9dd5511b53436b16 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.965775 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 26872be132fb46ee9e79a9c2e62374ef (table default.Table_b23313ec4d7b4e32a4283dd404578616 [id=6a545a959eb74d0eb589f1be92a0bc0a]) successfully deleted
I20251027 21:08:03.967866 10043 tablet_replica.cc:333] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:03.968379 10043 raft_consensus.cc:2243] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.968849 10043 raft_consensus.cc:2272] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.970446 10031 tablet_bootstrap.cc:492] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:03.970646 10031 ts_tablet_manager.cc:1403] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.011s
I20251027 21:08:03.971563 10040 ts_tablet_manager.cc:1929] T a9bf3a093be8498c9dd5511b53436b16 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.971661 10040 log.cc:1199] T a9bf3a093be8498c9dd5511b53436b16 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/a9bf3a093be8498c9dd5511b53436b16
I20251027 21:08:03.971942 10040 ts_tablet_manager.cc:1950] T a9bf3a093be8498c9dd5511b53436b16 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:03.972465 10040 tablet_replica.cc:333] T 630ae4d66f99474382814cc32cc60ea0 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:03.972649 10040 raft_consensus.cc:2243] T 630ae4d66f99474382814cc32cc60ea0 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:03.972594 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet a9bf3a093be8498c9dd5511b53436b16 (table default.Table_627b1e3af2d34bc9b4508237abf020f5 [id=5c9fd870185a4211a5a956171f491cc0]) successfully deleted
I20251027 21:08:03.972777 10040 raft_consensus.cc:2272] T 630ae4d66f99474382814cc32cc60ea0 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:03.972189 10031 raft_consensus.cc:359] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.973258 10031 raft_consensus.cc:385] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:03.973289 10040 ts_tablet_manager.cc:1916] T 630ae4d66f99474382814cc32cc60ea0 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.973593 10031 raft_consensus.cc:740] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:03.973889 10031 consensus_queue.cc:260] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:03.974771 10031 ts_tablet_manager.cc:1434] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251027 21:08:03.975150 10031 tablet_bootstrap.cc:492] T d872cf3ef39042ff9a615b55e7195afa P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:03.975538 10043 ts_tablet_manager.cc:1916] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.979286 10031 tablet_bootstrap.cc:654] T d872cf3ef39042ff9a615b55e7195afa P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:03.984897 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f57888547f8c4fd984cc5066c90d9ae6" } new_table_name: "default.Table_d1dcc40063f248f089f3d87e214e4c16" modify_external_catalogs: true
I20251027 21:08:03.988348 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9193c70ec7c84ce4829d242e054520a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:03.990566 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet 0e4d35f229bc46e1859ded7c8196ab9d from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:03.990928 10041 ts_tablet_manager.cc:1929] T f775a7e42bdc4910bdb604b26a28a7f4 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:03.991030 10041 log.cc:1199] T f775a7e42bdc4910bdb604b26a28a7f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/f775a7e42bdc4910bdb604b26a28a7f4
I20251027 21:08:03.991318 10041 ts_tablet_manager.cc:1950] T f775a7e42bdc4910bdb604b26a28a7f4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:03.992244 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet f775a7e42bdc4910bdb604b26a28a7f4 (table default.Table_7b970942803b42f29feb36d6d845167a [id=20a729dd21c440a882083323ddb2561e]) successfully deleted
I20251027 21:08:03.992729 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_201f11890a6243e3915632a430cc0ab4" } new_table_name: "default.Table_f78d839900634631b6963b2d63f0d084" modify_external_catalogs: true
I20251027 21:08:03.995290 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet d03918b19787449a88e0f17a925ce599
I20251027 21:08:03.996091 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 9a2432f5c24b4bbca744d5f4806f2b58
I20251027 21:08:04.000952 10041 tablet_replica.cc:333] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:04.001168 10041 raft_consensus.cc:2243] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:04.001323 10041 raft_consensus.cc:2272] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:04.002007 10041 ts_tablet_manager.cc:1916] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:03.979562 10040 ts_tablet_manager.cc:1929] T 630ae4d66f99474382814cc32cc60ea0 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:04.002347 10040 log.cc:1199] T 630ae4d66f99474382814cc32cc60ea0 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/630ae4d66f99474382814cc32cc60ea0
I20251027 21:08:03.982237 10829 raft_consensus.cc:697] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:04.002902 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet d03918b19787449a88e0f17a925ce599 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:04.003093 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a2432f5c24b4bbca744d5f4806f2b58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:04.003247 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet d03918b19787449a88e0f17a925ce599 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.003132 10829 consensus_queue.cc:237] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.003396 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a2432f5c24b4bbca744d5f4806f2b58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.003809 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_cf9164135a4b4ccc91f15ca37555a7c1" } new_table_name: "default.Table_352763845d1f4fb2a97474f8c82883b7" modify_external_catalogs: true
I20251027 21:08:04.004702 10040 ts_tablet_manager.cc:1950] T 630ae4d66f99474382814cc32cc60ea0 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:04.005416 10040 tablet_replica.cc:333] T dd7029986a5246ee823e5fdb50683118 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:04.005630 10040 raft_consensus.cc:2243] T dd7029986a5246ee823e5fdb50683118 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:04.005754 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 630ae4d66f99474382814cc32cc60ea0 (table default.Table_cfef185a988e4d61ae6575c228abc88d [id=8a26e744abae4c5eb1d18dd994cd6bd9]) successfully deleted
I20251027 21:08:04.005782 10040 raft_consensus.cc:2272] T dd7029986a5246ee823e5fdb50683118 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:04.006021 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 19f3e1a9db434c519e26e864eb790706 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19f3e1a9db434c519e26e864eb790706 already in progress: creating tablet
I20251027 21:08:04.006631 10040 ts_tablet_manager.cc:1916] T dd7029986a5246ee823e5fdb50683118 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:04.006824 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet d03918b19787449a88e0f17a925ce599 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.006887 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a2432f5c24b4bbca744d5f4806f2b58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.009447 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_904989a77e13498f8fa57464da33ecfe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.010016 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_904989a77e13498f8fa57464da33ecfe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.011603 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_55f3c5dc04064eb7b371b99b5e64287a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.012046 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55f3c5dc04064eb7b371b99b5e64287a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.013495 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4a3918c0e2224ae8b75c099bd4332ca0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.014005 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a3918c0e2224ae8b75c099bd4332ca0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.014492 10031 tablet_bootstrap.cc:492] T d872cf3ef39042ff9a615b55e7195afa P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:04.014492 10029 tablet_bootstrap.cc:492] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:04.014619 10029 ts_tablet_manager.cc:1403] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.095s user 0.024s sys 0.068s
I20251027 21:08:04.014619 10031 ts_tablet_manager.cc:1403] T d872cf3ef39042ff9a615b55e7195afa P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.002s
I20251027 21:08:04.015244 10031 raft_consensus.cc:359] T d872cf3ef39042ff9a615b55e7195afa P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.015400 10031 raft_consensus.cc:385] T d872cf3ef39042ff9a615b55e7195afa P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.015451 10031 raft_consensus.cc:740] T d872cf3ef39042ff9a615b55e7195afa P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.015514 10029 raft_consensus.cc:359] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.015658 10029 raft_consensus.cc:385] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.015702 10029 raft_consensus.cc:740] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.016213 10029 consensus_queue.cc:260] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.016682 10029 ts_tablet_manager.cc:1434] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:04.016963 10029 tablet_bootstrap.cc:492] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:04.020004 10029 tablet_bootstrap.cc:654] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.023777 10031 consensus_queue.cc:260] T d872cf3ef39042ff9a615b55e7195afa P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.024302 10031 ts_tablet_manager.cc:1434] T d872cf3ef39042ff9a615b55e7195afa P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20251027 21:08:04.024581 10031 tablet_bootstrap.cc:492] T 24ec2e0ba1994ab59c53977583bd2058 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:04.026243 10031 tablet_bootstrap.cc:654] T 24ec2e0ba1994ab59c53977583bd2058 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.029800 10043 ts_tablet_manager.cc:1929] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:04.029930 10043 log.cc:1199] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/22d861810ed846c3b77a452d05e93d72
I20251027 21:08:04.030562 10040 ts_tablet_manager.cc:1929] T dd7029986a5246ee823e5fdb50683118 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:04.030700 10040 log.cc:1199] T dd7029986a5246ee823e5fdb50683118 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/dd7029986a5246ee823e5fdb50683118
I20251027 21:08:04.031019 10040 ts_tablet_manager.cc:1950] T dd7029986a5246ee823e5fdb50683118 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:04.031611 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet dd7029986a5246ee823e5fdb50683118 (table default.Table_79211c9993c54e4ea0455d01fec68b8e [id=421a64f7fb6f499bbf6f20fb38f736d0]) successfully deleted
W20251027 21:08:04.032907 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bc505d437e774afcbd030a4e88375649 has no consensus state
I20251027 21:08:04.033538 10955 raft_consensus.cc:493] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.033687 10955 raft_consensus.cc:515] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:04.034332 10955 leader_election.cc:290] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:04.035019 10040 tablet_replica.cc:333] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:04.035085 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01e9db9eaf8e41c6a2ee13b9ed127421" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:04.035095 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01e9db9eaf8e41c6a2ee13b9ed127421" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:04.035219 10040 raft_consensus.cc:2243] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:04.035634 10040 pending_rounds.cc:70] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
I20251027 21:08:04.035702 10040 pending_rounds.cc:77] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510667280953344 op_type: NO_OP noop_request { }
W20251027 21:08:04.035761 9859 leader_election.cc:343] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.035797 10040 raft_consensus.cc:2889] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:04.035902 10040 raft_consensus.cc:2272] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:04.036103 9860 leader_election.cc:343] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.036197 9860 leader_election.cc:304] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:04.036252 10043 ts_tablet_manager.cc:1950] T 22d861810ed846c3b77a452d05e93d72 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:04.036406 10955 raft_consensus.cc:2749] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:04.037415 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 22d861810ed846c3b77a452d05e93d72 (table default.Table_e4b781970438453eb7f7f4b885331bfb [id=c92fda1282d7419f94b0004ab8b8403d]) successfully deleted
I20251027 21:08:04.038477 10040 ts_tablet_manager.cc:1916] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:04.038830 10043 tablet_replica.cc:333] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:04.040166 10859 raft_consensus.cc:493] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.042268 10859 raft_consensus.cc:515] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.042361 10043 raft_consensus.cc:2243] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:04.042564 10043 raft_consensus.cc:2272] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:04.042892 10859 leader_election.cc:290] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:04.043147 10043 ts_tablet_manager.cc:1916] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:04.043987 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6600774e4c7b48479a2fdb02614ba505" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:04.044729 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6600774e4c7b48479a2fdb02614ba505" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
W20251027 21:08:04.044813 9566 leader_election.cc:343] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.044952 9924 raft_consensus.cc:2468] T 6600774e4c7b48479a2fdb02614ba505 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:04.045156 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_685b53b410ef4403890d1494d6d77042"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.045646 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_685b53b410ef4403890d1494d6d77042 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.045962 10029 tablet_bootstrap.cc:492] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:04.046061 10029 ts_tablet_manager.cc:1403] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.012s
I20251027 21:08:04.046787 10031 tablet_bootstrap.cc:492] T 24ec2e0ba1994ab59c53977583bd2058 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
W20251027 21:08:04.046857 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b98a836363414da489fb4090abe5d088 consensus state has no leader
I20251027 21:08:04.046883 10031 ts_tablet_manager.cc:1403] T 24ec2e0ba1994ab59c53977583bd2058 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.002s
I20251027 21:08:04.046938 9568 leader_election.cc:304] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:04.046977 10029 raft_consensus.cc:359] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.047124 10029 raft_consensus.cc:385] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.047169 10029 raft_consensus.cc:740] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.047346 10029 consensus_queue.cc:260] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.047493 10955 raft_consensus.cc:493] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.047564 10907 raft_consensus.cc:2804] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:04.047502 10031 raft_consensus.cc:359] T 24ec2e0ba1994ab59c53977583bd2058 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.047621 10907 raft_consensus.cc:493] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:04.047636 10031 raft_consensus.cc:385] T 24ec2e0ba1994ab59c53977583bd2058 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.047680 10907 raft_consensus.cc:3060] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:04.047614 10955 raft_consensus.cc:515] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:04.047699 10031 raft_consensus.cc:740] T 24ec2e0ba1994ab59c53977583bd2058 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.047904 10031 consensus_queue.cc:260] T 24ec2e0ba1994ab59c53977583bd2058 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.048149 10955 leader_election.cc:290] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:04.048285 10031 ts_tablet_manager.cc:1434] T 24ec2e0ba1994ab59c53977583bd2058 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251027 21:08:04.048676 10031 tablet_bootstrap.cc:492] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:04.048802 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4d26f73b8534f08bcaa386d4de75c01" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:04.048802 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4d26f73b8534f08bcaa386d4de75c01" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:04.049258 9859 leader_election.cc:343] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:04.049809 9860 leader_election.cc:343] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.049897 9860 leader_election.cc:304] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:04.049974 10031 tablet_bootstrap.cc:654] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.050071 10907 raft_consensus.cc:515] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.050887 10907 leader_election.cc:290] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:04.051237 10955 raft_consensus.cc:2749] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:04.051406 10029 ts_tablet_manager.cc:1434] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251027 21:08:04.051611 10029 tablet_bootstrap.cc:492] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:04.052165 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6600774e4c7b48479a2fdb02614ba505" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:04.052301 9924 raft_consensus.cc:3060] T 6600774e4c7b48479a2fdb02614ba505 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:04.052718 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6600774e4c7b48479a2fdb02614ba505" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:04.053145 9566 leader_election.cc:343] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.053227 10029 tablet_bootstrap.cc:654] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.053325 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_090167e6602f4bff86b275f39d9a58cb" } new_table_name: "default.Table_c243a899ea1e40d1b751b637923eebd9" modify_external_catalogs: true
I20251027 21:08:04.055704 9924 raft_consensus.cc:2468] T 6600774e4c7b48479a2fdb02614ba505 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:04.056282 9568 leader_election.cc:304] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:04.056638 10907 raft_consensus.cc:2804] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:04.056757 10907 raft_consensus.cc:697] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:04.056970 10907 consensus_queue.cc:237] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:04.059906 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dd9e6656d8df43afbb7f70706bc14585 consensus state has no leader
I20251027 21:08:04.063063 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9f371b16693a415896291c6a3edf3d25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.063581 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f371b16693a415896291c6a3edf3d25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.068077 10040 ts_tablet_manager.cc:1929] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:04.068207 10040 log.cc:1199] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/22d861810ed846c3b77a452d05e93d72
I20251027 21:08:04.068607 10040 ts_tablet_manager.cc:1950] T 22d861810ed846c3b77a452d05e93d72 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:04.069211 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 22d861810ed846c3b77a452d05e93d72 (table default.Table_e4b781970438453eb7f7f4b885331bfb [id=c92fda1282d7419f94b0004ab8b8403d]) successfully deleted
W20251027 21:08:04.069774 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd6152f7ac8a4123872098955b8837ac consensus state has no leader
I20251027 21:08:04.070497 10031 tablet_bootstrap.cc:492] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:04.070649 10031 ts_tablet_manager.cc:1403] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20251027 21:08:04.071686 10031 raft_consensus.cc:359] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.071888 10031 raft_consensus.cc:385] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.071944 10031 raft_consensus.cc:740] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.072203 10031 consensus_queue.cc:260] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.072687 10031 ts_tablet_manager.cc:1434] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:04.078361 10031 tablet_bootstrap.cc:492] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:04.079927 10031 tablet_bootstrap.cc:654] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.080339 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.081147 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ec2e0ba1994ab59c53977583bd2058 with delete_type TABLET_DATA_DELETED (replaced by tablet 815e12edcd3847ff8485ba768291a824) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.082213 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet d872cf3ef39042ff9a615b55e7195afa with delete_type TABLET_DATA_DELETED (replaced by tablet 6f901626832f4adb948ccb65f8573e86) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.082808 10043 ts_tablet_manager.cc:1929] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:04.082727 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_cfb487c77526400b84830a2d722437a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:04.082922 10043 log.cc:1199] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/53cac7817db440e79e7416d625a9b2c8
I20251027 21:08:04.083313 10043 ts_tablet_manager.cc:1950] T 53cac7817db440e79e7416d625a9b2c8 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
W20251027 21:08:04.083460 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cfb487c77526400b84830a2d722437a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.084571 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 53cac7817db440e79e7416d625a9b2c8 (table default.Table_79a3420c8b6542868231c740293ab1fb [id=f3babbbf131543d2813ed9d5882e3d00]) successfully deleted
I20251027 21:08:04.084889 10040 tablet_replica.cc:333] T 84b9de82d55447bba2a897edb683e5c5 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:04.085084 10040 raft_consensus.cc:2243] T 84b9de82d55447bba2a897edb683e5c5 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:04.085258 10040 raft_consensus.cc:2272] T 84b9de82d55447bba2a897edb683e5c5 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:04.085610 10041 ts_tablet_manager.cc:1929] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:04.086688 10031 tablet_bootstrap.cc:492] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:04.086833 10031 ts_tablet_manager.cc:1403] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20251027 21:08:04.087714 10031 raft_consensus.cc:359] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.087916 10031 raft_consensus.cc:385] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.087990 10031 raft_consensus.cc:740] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.088598 10040 ts_tablet_manager.cc:1916] T 84b9de82d55447bba2a897edb683e5c5 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:04.088888 10829 raft_consensus.cc:493] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.088999 10829 raft_consensus.cc:515] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.089494 10829 leader_election.cc:290] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:04.090479 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71dc6b7a6ca146c6b0c58d9ad0ecbefa" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:04.090551 9711 leader_election.cc:343] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 71dc6b7a6ca146c6b0c58d9ad0ecbefa
I20251027 21:08:04.090446 10031 consensus_queue.cc:260] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.090667 9631 raft_consensus.cc:2468] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:08:04.091056 10031 ts_tablet_manager.cc:1434] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251027 21:08:04.091892 10031 tablet_bootstrap.cc:492] T 53cac7817db440e79e7416d625a9b2c8 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:04.092283 9710 leader_election.cc:304] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:04.092664 10829 raft_consensus.cc:2804] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:04.092746 10829 raft_consensus.cc:493] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:04.092823 10829 raft_consensus.cc:3060] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:04.092994 10043 tablet_replica.cc:333] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:04.094676 10031 tablet_bootstrap.cc:654] T 53cac7817db440e79e7416d625a9b2c8 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.095382 10829 raft_consensus.cc:515] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.096350 10829 leader_election.cc:290] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:04.096765 9711 leader_election.cc:343] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 71dc6b7a6ca146c6b0c58d9ad0ecbefa
I20251027 21:08:04.097355 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71dc6b7a6ca146c6b0c58d9ad0ecbefa" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:04.097503 9631 raft_consensus.cc:3060] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:04.100031 10040 ts_tablet_manager.cc:1929] T 84b9de82d55447bba2a897edb683e5c5 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:04.100118 10040 log.cc:1199] T 84b9de82d55447bba2a897edb683e5c5 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/84b9de82d55447bba2a897edb683e5c5
I20251027 21:08:04.100142 9631 raft_consensus.cc:2468] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:04.100364 10040 ts_tablet_manager.cc:1950] T 84b9de82d55447bba2a897edb683e5c5 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:04.100561 9710 leader_election.cc:304] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:04.100636 10859 raft_consensus.cc:493] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.100764 10859 raft_consensus.cc:515] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.100831 10829 raft_consensus.cc:2804] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:04.101027 10829 raft_consensus.cc:697] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:04.101244 10829 consensus_queue.cc:237] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.101758 10859 leader_election.cc:290] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:04.102180 9568 leader_election.cc:343] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: a8328318a4f14b38a4ce36d183e99dfb
I20251027 21:08:04.103060 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8328318a4f14b38a4ce36d183e99dfb" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:04.104089 9566 leader_election.cc:343] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.104207 9566 leader_election.cc:304] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:04.104707 10859 raft_consensus.cc:2749] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:04.108462 10043 raft_consensus.cc:2243] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:04.108889 10043 raft_consensus.cc:2272] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:04.109233 10029 tablet_bootstrap.cc:492] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:04.109385 10029 ts_tablet_manager.cc:1403] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.058s user 0.020s sys 0.017s
I20251027 21:08:04.109478 10043 ts_tablet_manager.cc:1916] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:04.110163 10029 raft_consensus.cc:359] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.110353 10029 raft_consensus.cc:385] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.110421 10029 raft_consensus.cc:740] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.111593 10029 consensus_queue.cc:260] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.112018 10029 ts_tablet_manager.cc:1434] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251027 21:08:04.112097 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_aca70af1903740b5b35bbc3554beff36" } modify_external_catalogs: true
I20251027 21:08:04.112247 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_aca70af1903740b5b35bbc3554beff36" } modify_external_catalogs: true
I20251027 21:08:04.112304 10029 tablet_bootstrap.cc:492] T 67bb12de9f044c73ae9ab26d10498c90 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:04.113709 10029 tablet_bootstrap.cc:654] T 67bb12de9f044c73ae9ab26d10498c90 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.114500 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_44f9303ecfd745f99df2ffa39d8a0499"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:04.117000 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 84b9de82d55447bba2a897edb683e5c5 (table default.Table_8dce34af37324a659bd37f54cd0c379b [id=6b70bbbf477d4b788b3ffb10c9c7d7c9]) successfully deleted
I20251027 21:08:04.119235 10041 log.cc:1199] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/76e4fb2bea7f4a4696fe9f6bb892bccc
I20251027 21:08:04.119925 10031 tablet_bootstrap.cc:492] T 53cac7817db440e79e7416d625a9b2c8 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:04.120059 10031 ts_tablet_manager.cc:1403] T 53cac7817db440e79e7416d625a9b2c8 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20251027 21:08:04.120514 10041 ts_tablet_manager.cc:1950] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:04.120751 10031 raft_consensus.cc:359] T 53cac7817db440e79e7416d625a9b2c8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.120213 10032 tablet_bootstrap.cc:492] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:04.120946 10031 raft_consensus.cc:385] T 53cac7817db440e79e7416d625a9b2c8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.121002 10031 raft_consensus.cc:740] T 53cac7817db440e79e7416d625a9b2c8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.121161 10032 ts_tablet_manager.cc:1403] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.216s user 0.006s sys 0.007s
I20251027 21:08:04.121232 10031 consensus_queue.cc:260] T 53cac7817db440e79e7416d625a9b2c8 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.121773 10031 ts_tablet_manager.cc:1434] T 53cac7817db440e79e7416d625a9b2c8 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:04.122069 10031 tablet_bootstrap.cc:492] T 87d5af4607f04f92abcc7292b4176cbd P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:04.124104 10031 tablet_bootstrap.cc:654] T 87d5af4607f04f92abcc7292b4176cbd P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:04.125417 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 6bbee5f1fb704709b6f166a153e69808 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bbee5f1fb704709b6f166a153e69808 already in progress: creating tablet
I20251027 21:08:04.125528 10032 raft_consensus.cc:359] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:04.125720 10032 raft_consensus.cc:385] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.125784 10032 raft_consensus.cc:740] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.125777 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 76e4fb2bea7f4a4696fe9f6bb892bccc (table default.Table_b7607fa1c70d431d97f9e3f8a7c3d667 [id=080bf50609c045bd9b05f4e9144feac5]) successfully deleted
I20251027 21:08:04.126010 10032 consensus_queue.cc:260] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
W20251027 21:08:04.126356 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet d03918b19787449a88e0f17a925ce599 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d03918b19787449a88e0f17a925ce599 already in progress: creating tablet
I20251027 21:08:04.128484 10043 ts_tablet_manager.cc:1929] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:04.128633 10043 log.cc:1199] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/71dc6b7a6ca146c6b0c58d9ad0ecbefa
I20251027 21:08:04.128978 10043 ts_tablet_manager.cc:1950] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:04.128934 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_92cfd00027b045f3b151deec3558ece6" } modify_external_catalogs: true
I20251027 21:08:04.129078 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_92cfd00027b045f3b151deec3558ece6" } modify_external_catalogs: true
I20251027 21:08:04.129884 10043 tablet_replica.cc:333] T 2532bf7190064443a898e47e843fd9ac P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:04.132604 10032 ts_tablet_manager.cc:1434] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
W20251027 21:08:04.133407 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 87612b8bd8a04fd9b32f2df3f9e0d997 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87612b8bd8a04fd9b32f2df3f9e0d997 already in progress: creating tablet
I20251027 21:08:04.134599 10041 tablet_replica.cc:333] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:04.130215 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa (table default.Table_8fa71eed37b040fcb5f63d829b4a6315 [id=de5974f2ae38479fa2d403d5ac6e1dc5]) successfully deleted
I20251027 21:08:04.137220 10040 tablet_replica.cc:333] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:04.135293 10032 tablet_bootstrap.cc:492] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:04.138585 10041 raft_consensus.cc:2243] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251027 21:08:04.137077 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 9a2432f5c24b4bbca744d5f4806f2b58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a2432f5c24b4bbca744d5f4806f2b58 already in progress: creating tablet
W20251027 21:08:04.141755 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44f9303ecfd745f99df2ffa39d8a0499 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.143460 10032 tablet_bootstrap.cc:654] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:04.143939 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9193c70ec7c84ce4829d242e054520a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.153838 10031 tablet_bootstrap.cc:492] T 87d5af4607f04f92abcc7292b4176cbd P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:04.153980 10031 ts_tablet_manager.cc:1403] T 87d5af4607f04f92abcc7292b4176cbd P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20251027 21:08:04.154822 10031 raft_consensus.cc:359] T 87d5af4607f04f92abcc7292b4176cbd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.155004 10031 raft_consensus.cc:385] T 87d5af4607f04f92abcc7292b4176cbd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.155058 10031 raft_consensus.cc:740] T 87d5af4607f04f92abcc7292b4176cbd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.130089 10043 raft_consensus.cc:2243] T 2532bf7190064443a898e47e843fd9ac P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:04.155391 10043 raft_consensus.cc:2272] T 2532bf7190064443a898e47e843fd9ac P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:04.155781 10907 raft_consensus.cc:670] T 2532bf7190064443a898e47e843fd9ac P 51902b6b61ff4d5584ff21896e847e14: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopped
I20251027 21:08:04.155952 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 9861a0efcba64141aae450663c5d83de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:04.156805 10040 raft_consensus.cc:2243] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:04.157500 10040 pending_rounds.cc:70] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
I20251027 21:08:04.157567 10040 pending_rounds.cc:77] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510667683348480 op_type: NO_OP noop_request { }
I20251027 21:08:04.157721 10040 raft_consensus.cc:2889] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:04.157816 10040 raft_consensus.cc:2272] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:04.158641 10043 ts_tablet_manager.cc:1916] T 2532bf7190064443a898e47e843fd9ac P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:04.162448 10031 consensus_queue.cc:260] T 87d5af4607f04f92abcc7292b4176cbd P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.163025 10031 ts_tablet_manager.cc:1434] T 87d5af4607f04f92abcc7292b4176cbd P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20251027 21:08:04.163873 10040 ts_tablet_manager.cc:1916] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:04.166781 10031 tablet_bootstrap.cc:492] T a8328318a4f14b38a4ce36d183e99dfb P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:04.171833 10032 tablet_bootstrap.cc:492] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
W20251027 21:08:04.150475 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42d5df2a47b744ffb0bf3a34201903d7 consensus state has no leader
I20251027 21:08:04.171963 10032 ts_tablet_manager.cc:1403] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20251027 21:08:04.172242 10907 raft_consensus.cc:493] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.172386 10907 raft_consensus.cc:515] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.168211 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet c25be263facd42c5bb6fa421160fbef0 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:04.173303 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5af4607f04f92abcc7292b4176cbd" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:04.173473 9924 raft_consensus.cc:2468] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:04.173745 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5af4607f04f92abcc7292b4176cbd" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:04.173910 9769 raft_consensus.cc:2468] T 87d5af4607f04f92abcc7292b4176cbd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:04.173964 9568 leader_election.cc:304] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:04.174240 10859 raft_consensus.cc:2804] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:04.174347 10859 raft_consensus.cc:493] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:04.174433 10859 raft_consensus.cc:3060] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:04.174489 10907 leader_election.cc:290] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:04.177088 10859 raft_consensus.cc:515] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.178090 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5af4607f04f92abcc7292b4176cbd" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:04.178274 9924 raft_consensus.cc:3060] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:04.178383 10032 raft_consensus.cc:359] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:04.178529 10032 raft_consensus.cc:385] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.178584 10032 raft_consensus.cc:740] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.179090 10031 tablet_bootstrap.cc:654] T a8328318a4f14b38a4ce36d183e99dfb P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.180830 9924 raft_consensus.cc:2468] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:04.181030 10032 consensus_queue.cc:260] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:04.181578 9568 leader_election.cc:304] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:04.182031 10859 leader_election.cc:290] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:04.182405 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5af4607f04f92abcc7292b4176cbd" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:04.182559 9769 raft_consensus.cc:3060] T 87d5af4607f04f92abcc7292b4176cbd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:04.182415 10859 raft_consensus.cc:2804] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:04.182811 10859 raft_consensus.cc:697] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:04.183074 10859 consensus_queue.cc:237] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.183307 10040 ts_tablet_manager.cc:1929] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:04.183414 10040 log.cc:1199] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/71dc6b7a6ca146c6b0c58d9ad0ecbefa
I20251027 21:08:04.183758 10040 ts_tablet_manager.cc:1950] T 71dc6b7a6ca146c6b0c58d9ad0ecbefa P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:04.185030 9769 raft_consensus.cc:2468] T 87d5af4607f04f92abcc7292b4176cbd P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:04.187669 10040 tablet_replica.cc:333] T 24ec2e0ba1994ab59c53977583bd2058 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:04.184499 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 71dc6b7a6ca146c6b0c58d9ad0ecbefa (table default.Table_8fa71eed37b040fcb5f63d829b4a6315 [id=de5974f2ae38479fa2d403d5ac6e1dc5]) successfully deleted
I20251027 21:08:04.188584 10040 raft_consensus.cc:2243] T 24ec2e0ba1994ab59c53977583bd2058 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:04.188743 10040 raft_consensus.cc:2272] T 24ec2e0ba1994ab59c53977583bd2058 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:04.193315 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_34bd7efc93714dca9c8bf72307608cf1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:04.193719 10040 ts_tablet_manager.cc:1916] T 24ec2e0ba1994ab59c53977583bd2058 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:04.193787 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34bd7efc93714dca9c8bf72307608cf1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.195298 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c2d713d72fd4b5ca34cb3e828c75761 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:04.195279 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_25435d2d53084e0aa737382bf0fa0e2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.195737 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25435d2d53084e0aa737382bf0fa0e2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.196272 10032 ts_tablet_manager.cc:1434] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20251027 21:08:04.199115 10029 tablet_bootstrap.cc:492] T 67bb12de9f044c73ae9ab26d10498c90 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:04.199314 10029 ts_tablet_manager.cc:1403] T 67bb12de9f044c73ae9ab26d10498c90 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.087s user 0.009s sys 0.027s
I20251027 21:08:04.199115 10043 ts_tablet_manager.cc:1929] T 2532bf7190064443a898e47e843fd9ac P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:04.199347 11018 consensus_queue.cc:1048] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:04.199576 10043 log.cc:1199] T 2532bf7190064443a898e47e843fd9ac P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/2532bf7190064443a898e47e843fd9ac
I20251027 21:08:04.200060 10043 ts_tablet_manager.cc:1950] T 2532bf7190064443a898e47e843fd9ac P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:04.201574 10032 tablet_bootstrap.cc:492] T d672118ecd6f4094855adf106e5f7115 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:04.202257 10031 tablet_bootstrap.cc:492] T a8328318a4f14b38a4ce36d183e99dfb P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:04.202431 10031 ts_tablet_manager.cc:1403] T a8328318a4f14b38a4ce36d183e99dfb P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.005s
I20251027 21:08:04.202920 10907 raft_consensus.cc:493] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.203073 10907 raft_consensus.cc:515] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.203277 10031 raft_consensus.cc:359] T a8328318a4f14b38a4ce36d183e99dfb P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.203469 10031 raft_consensus.cc:385] T a8328318a4f14b38a4ce36d183e99dfb P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.203544 10031 raft_consensus.cc:740] T a8328318a4f14b38a4ce36d183e99dfb P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.203786 10907 leader_election.cc:290] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:04.204059 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a26ded6012174348b7295fc7c0e05e6e" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:04.204488 9566 leader_election.cc:343] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.204520 10031 consensus_queue.cc:260] T a8328318a4f14b38a4ce36d183e99dfb P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.204957 10031 ts_tablet_manager.cc:1434] T a8328318a4f14b38a4ce36d183e99dfb P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:04.205245 10031 tablet_bootstrap.cc:492] T 6600774e4c7b48479a2fdb02614ba505 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:04.206892 10031 tablet_bootstrap.cc:654] T 6600774e4c7b48479a2fdb02614ba505 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.209826 10029 raft_consensus.cc:359] T 67bb12de9f044c73ae9ab26d10498c90 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.209993 10029 raft_consensus.cc:385] T 67bb12de9f044c73ae9ab26d10498c90 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.210083 10029 raft_consensus.cc:740] T 67bb12de9f044c73ae9ab26d10498c90 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.210294 10029 consensus_queue.cc:260] T 67bb12de9f044c73ae9ab26d10498c90 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.210743 10029 ts_tablet_manager.cc:1434] T 67bb12de9f044c73ae9ab26d10498c90 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251027 21:08:04.213594 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a46825cc888a4a0898d6db70b02e5ab6" } modify_external_catalogs: true
I20251027 21:08:04.213594 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_abc7f8be9f414322ad9064340647399f" } modify_external_catalogs: true
I20251027 21:08:04.213824 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_abc7f8be9f414322ad9064340647399f" } modify_external_catalogs: true
I20251027 21:08:04.213970 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 2532bf7190064443a898e47e843fd9ac (table default.Table_89e2320f599341e882a2d09c0985b220 [id=c6f20d8985134d9a8d3c5bb4a51b45fb]) successfully deleted
I20251027 21:08:04.214408 10040 ts_tablet_manager.cc:1929] T 24ec2e0ba1994ab59c53977583bd2058 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:04.214481 10029 tablet_bootstrap.cc:492] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:04.214524 10040 log.cc:1199] T 24ec2e0ba1994ab59c53977583bd2058 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/24ec2e0ba1994ab59c53977583bd2058
I20251027 21:08:04.214856 10040 ts_tablet_manager.cc:1950] T 24ec2e0ba1994ab59c53977583bd2058 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
W20251027 21:08:04.214996 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 19f3e1a9db434c519e26e864eb790706 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19f3e1a9db434c519e26e864eb790706 already in progress: creating tablet
W20251027 21:08:04.215378 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 6bbee5f1fb704709b6f166a153e69808 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bbee5f1fb704709b6f166a153e69808 already in progress: creating tablet
I20251027 21:08:04.215500 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 24ec2e0ba1994ab59c53977583bd2058 (table default.Table_2992b8e207024e06957ce7befb657acb [id=decffeb1481e4d798cffd8c9883be59b]) successfully deleted
W20251027 21:08:04.215724 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet d03918b19787449a88e0f17a925ce599 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d03918b19787449a88e0f17a925ce599 already in progress: creating tablet
I20251027 21:08:04.215755 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a46825cc888a4a0898d6db70b02e5ab6" } modify_external_catalogs: true
W20251027 21:08:04.216159 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 9a2432f5c24b4bbca744d5f4806f2b58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a2432f5c24b4bbca744d5f4806f2b58 already in progress: creating tablet
I20251027 21:08:04.216343 10031 tablet_bootstrap.cc:492] T 6600774e4c7b48479a2fdb02614ba505 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:04.216491 10031 ts_tablet_manager.cc:1403] T 6600774e4c7b48479a2fdb02614ba505 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
W20251027 21:08:04.216708 9568 leader_election.cc:343] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: a26ded6012174348b7295fc7c0e05e6e
I20251027 21:08:04.217154 10031 raft_consensus.cc:359] T 6600774e4c7b48479a2fdb02614ba505 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.217327 10031 raft_consensus.cc:385] T 6600774e4c7b48479a2fdb02614ba505 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.217410 10031 raft_consensus.cc:740] T 6600774e4c7b48479a2fdb02614ba505 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.217734 10031 consensus_queue.cc:260] T 6600774e4c7b48479a2fdb02614ba505 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.218225 10031 ts_tablet_manager.cc:1434] T 6600774e4c7b48479a2fdb02614ba505 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:04.218606 10031 tablet_bootstrap.cc:492] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:04.220180 10031 tablet_bootstrap.cc:654] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.220971 10040 tablet_replica.cc:333] T d872cf3ef39042ff9a615b55e7195afa P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:04.221153 10040 raft_consensus.cc:2243] T d872cf3ef39042ff9a615b55e7195afa P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:04.221311 10040 raft_consensus.cc:2272] T d872cf3ef39042ff9a615b55e7195afa P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:04.222007 10040 ts_tablet_manager.cc:1916] T d872cf3ef39042ff9a615b55e7195afa P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:04.222244 10032 tablet_bootstrap.cc:654] T d672118ecd6f4094855adf106e5f7115 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.224922 10031 tablet_bootstrap.cc:492] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:04.225042 10031 ts_tablet_manager.cc:1403] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20251027 21:08:04.226676 9568 leader_election.cc:304] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:04.227119 10859 raft_consensus.cc:2749] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:04.228236 10032 tablet_bootstrap.cc:492] T d672118ecd6f4094855adf106e5f7115 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:04.228369 10032 ts_tablet_manager.cc:1403] T d672118ecd6f4094855adf106e5f7115 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.001s
I20251027 21:08:04.229080 10032 raft_consensus.cc:359] T d672118ecd6f4094855adf106e5f7115 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:04.229261 10032 raft_consensus.cc:385] T d672118ecd6f4094855adf106e5f7115 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.229316 10032 raft_consensus.cc:740] T d672118ecd6f4094855adf106e5f7115 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.221294 10029 tablet_bootstrap.cc:654] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.234299 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_48f4510204084408aef690097faeaca1" } new_table_name: "default.Table_802fea5b939f43549f0e180cd01cfa4b" modify_external_catalogs: true
I20251027 21:08:04.225759 10031 raft_consensus.cc:359] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.238844 10031 raft_consensus.cc:385] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.238920 10031 raft_consensus.cc:740] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.240881 10829 raft_consensus.cc:493] T 45568792d7954b498581ce69fb611948 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.241055 10829 raft_consensus.cc:515] T 45568792d7954b498581ce69fb611948 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.241622 10829 leader_election.cc:290] T 45568792d7954b498581ce69fb611948 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:04.242178 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45568792d7954b498581ce69fb611948" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:04.242377 9925 raft_consensus.cc:2393] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77b0513f17c0472088901a1ce8b08218 in current term 1: Already voted for candidate 51902b6b61ff4d5584ff21896e847e14 in this term.
I20251027 21:08:04.242395 9308 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 345d3795a37b4a40b40967cfacf0f893 (table default.Table_ef79e4d09fb741429811f81e1efd90cb [id=93d6307da3a34f87a67018a06825cd15]). Expected version 2 got 0
I20251027 21:08:04.242518 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45568792d7954b498581ce69fb611948" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:04.243211 9710 leader_election.cc:304] T 45568792d7954b498581ce69fb611948 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:04.245291 10829 raft_consensus.cc:3060] T 45568792d7954b498581ce69fb611948 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:04.247879 10829 raft_consensus.cc:2749] T 45568792d7954b498581ce69fb611948 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:04.248329 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 196a594025ad4564b028b39f1afae48f
I20251027 21:08:04.249020 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 196a594025ad4564b028b39f1afae48f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.249051 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 070167be7d8d4ddf8da561df14448360
I20251027 21:08:04.249522 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 196a594025ad4564b028b39f1afae48f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:04.249871 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 070167be7d8d4ddf8da561df14448360 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.250280 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet 196a594025ad4564b028b39f1afae48f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.250478 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 070167be7d8d4ddf8da561df14448360 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:04.250640 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1a6d4a92fb514fdf8f76dcf2757f2b70" } new_table_name: "default.Table_dbc2147f95cc4823bad51330a3e07498" modify_external_catalogs: true
I20251027 21:08:04.250826 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet 070167be7d8d4ddf8da561df14448360 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:04.251210 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 196a594025ad4564b028b39f1afae48f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 196a594025ad4564b028b39f1afae48f already in progress: creating tablet
W20251027 21:08:04.251611 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 070167be7d8d4ddf8da561df14448360 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 070167be7d8d4ddf8da561df14448360 already in progress: creating tablet
I20251027 21:08:04.254499 10031 consensus_queue.cc:260] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.254889 9303 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet abd3b8e71d3041b4b9e5f6663841bfcf (table default.Table_fd331f3cfc07425082d58ab5af3c1094 [id=f3e3147411e5488bae105ed755f5b50f]). Expected version 2 got 0
I20251027 21:08:04.254998 10031 ts_tablet_manager.cc:1434] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.030s user 0.001s sys 0.000s
I20251027 21:08:04.255230 10032 consensus_queue.cc:260] T d672118ecd6f4094855adf106e5f7115 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:04.255765 10032 ts_tablet_manager.cc:1434] T d672118ecd6f4094855adf106e5f7115 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.027s user 0.001s sys 0.000s
I20251027 21:08:04.256028 10032 tablet_bootstrap.cc:492] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:04.256443 9310 catalog_manager.cc:5649] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
W20251027 21:08:04.256820 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce9bad29c7b24f4cb729c8ef2e163c72 consensus state has no leader
I20251027 21:08:04.256927 9310 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 6600774e4c7b48479a2fdb02614ba505 (table default.Table_d7530773668f42c5a07a921df02a95ab [id=914b728e358e4fdba6d38d6c7beadaca]). Expected version 1 got 0
I20251027 21:08:04.257119 9310 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet b8856c5dc8f144358445165324bfb477 (table default.Table_203191e192be4e3a9a7477b4313ef4f4 [id=5da6bd4463e24c03accd9985f72b6ebb]). Expected version 3 got 0
I20251027 21:08:04.263940 10031 tablet_bootstrap.cc:492] T d56d323ef0e74344a479940b93fea11a P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:04.266661 11018 raft_consensus.cc:493] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.266831 11018 raft_consensus.cc:515] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.267573 10032 tablet_bootstrap.cc:654] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.267760 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca477937278f45b4b115fec715f32a92" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:04.268261 9566 leader_election.cc:343] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.268426 11018 leader_election.cc:290] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:04.268937 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca477937278f45b4b115fec715f32a92" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:04.269126 9925 raft_consensus.cc:2468] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:04.269268 10031 tablet_bootstrap.cc:654] T d56d323ef0e74344a479940b93fea11a P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.269521 9568 leader_election.cc:304] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:04.270539 10639 rpc_server.cc:307] RPC server started. Bound to: 127.9.14.124:45071
I20251027 21:08:04.273258 9304 master_service.cc:954] ReplaceTablet: received request to replace tablet 0ac1aed5f56844639b848f90c8eb5b6c from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:04.274400 10639 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-2/data/info.pb
I20251027 21:08:04.270537 11039 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.14.124:45071 every 8 connection(s)
I20251027 21:08:04.276005 10031 tablet_bootstrap.cc:492] T d56d323ef0e74344a479940b93fea11a P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
W20251027 21:08:04.276347 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e740ddf3b394d3a823bc95bad48fa99 consensus state has no leader
I20251027 21:08:04.276582 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 96a680e8f16343b1908829a67e960b60
I20251027 21:08:04.277637 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_342bf4fd6c7f4ffbb80cc0f87baf3768" } new_table_name: "default.Table_03f648dd39ab4c9cae71aa2c0121ced4" modify_external_catalogs: true
I20251027 21:08:04.278088 10041 raft_consensus.cc:2272] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:04.278637 10041 ts_tablet_manager.cc:1916] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:04.278908 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1376974f614548189208cb9cf0fd5353" } new_table_name: "default.Table_01dfe3b2ed8847b1b093719f06ad74f0" modify_external_catalogs: true
I20251027 21:08:04.279629 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 96a680e8f16343b1908829a67e960b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:04.283033 9273 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu as pid 10639
I20251027 21:08:04.283525 9307 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet d672118ecd6f4094855adf106e5f7115 (table default.Table_1b066b58b2c649a2887ed91461f15d7c [id=887ea33c2fbe4573a1d7169dfd226567]). Expected version 2 got 0
I20251027 21:08:04.278507 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 96a680e8f16343b1908829a67e960b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.281283 9748 tablet_service.cc:1505] Processing CreateTablet for tablet fdc215e390ca4b93bdee902f959843fb (DEFAULT_TABLE table=default.Table_df2d8d09ba3147afb5c9271da5e23d45 [id=a868a439c6b142068daad14573426d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.284379 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c6a189aa22d048858e9f30bf4276859d
I20251027 21:08:04.284471 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdc215e390ca4b93bdee902f959843fb. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:04.285138 9299 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet d7955e31f10449718964a8ced3c18672 (table default.Table_836e232c1c5546ee84d694e0f866f646 [id=593dff7ab03d4186b1e58a46ff1db6fc], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d7955e31f10449718964a8ced3c18672 consensus state has no leader
I20251027 21:08:04.285493 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet c6a189aa22d048858e9f30bf4276859d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:04.285920 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet c6a189aa22d048858e9f30bf4276859d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.288193 9311 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet abd3b8e71d3041b4b9e5f6663841bfcf (table default.Table_fd331f3cfc07425082d58ab5af3c1094 [id=f3e3147411e5488bae105ed755f5b50f]). Expected version 2 got 0
I20251027 21:08:04.293062 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_aaeb60868697426bbaa517f52ea223e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:04.293165 9308 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 6600774e4c7b48479a2fdb02614ba505 (table default.Table_d7530773668f42c5a07a921df02a95ab [id=914b728e358e4fdba6d38d6c7beadaca]). Expected version 1 got 0
I20251027 21:08:04.293462 9308 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 87d5af4607f04f92abcc7292b4176cbd (table default.Table_c78ef9ebe86b4da08443e37c83d4e80d [id=cdb023218cda4e9ea04a226b37a220d9]). Expected version 2 got 0
W20251027 21:08:04.293591 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aaeb60868697426bbaa517f52ea223e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.295609 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet a8328318a4f14b38a4ce36d183e99dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.295697 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 53cac7817db440e79e7416d625a9b2c8 with delete_type TABLET_DATA_DELETED (replaced by tablet ed66f9eeeb2b4b4bb77062c241d2140f) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.295609 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet 2532bf7190064443a898e47e843fd9ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.296628 10031 ts_tablet_manager.cc:1403] T d56d323ef0e74344a479940b93fea11a P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.002s
I20251027 21:08:04.298828 11018 consensus_queue.cc:1048] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:04.301869 9308 catalog_manager.cc:5649] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:04.303238 9890 tablet_service.cc:1505] Processing CreateTablet for tablet fdc215e390ca4b93bdee902f959843fb (DEFAULT_TABLE table=default.Table_df2d8d09ba3147afb5c9271da5e23d45 [id=a868a439c6b142068daad14573426d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.303622 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdc215e390ca4b93bdee902f959843fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.305310 9308 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 6600774e4c7b48479a2fdb02614ba505 (table default.Table_d7530773668f42c5a07a921df02a95ab [id=914b728e358e4fdba6d38d6c7beadaca]). Expected version 1 got 0
I20251027 21:08:04.305652 9308 catalog_manager.cc:5649] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
W20251027 21:08:04.307174 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f7fbffe7cd3c45d1b8fecec85f2bbf92 consensus state has no leader
I20251027 21:08:04.307925 9308 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 345d3795a37b4a40b40967cfacf0f893 (table default.Table_ef79e4d09fb741429811f81e1efd90cb [id=93d6307da3a34f87a67018a06825cd15]). Expected version 2 got 0
I20251027 21:08:04.308275 9308 catalog_manager.cc:5649] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:04.308863 9308 catalog_manager.cc:5649] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:04.310506 10029 tablet_bootstrap.cc:492] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:04.310688 10029 ts_tablet_manager.cc:1403] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.096s user 0.002s sys 0.001s
I20251027 21:08:04.312047 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet f9534e65d8bb4b3a88df934871792119
I20251027 21:08:04.309226 9308 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 87d5af4607f04f92abcc7292b4176cbd (table default.Table_c78ef9ebe86b4da08443e37c83d4e80d [id=cdb023218cda4e9ea04a226b37a220d9]). Expected version 2 got 0
I20251027 21:08:04.313933 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet f9534e65d8bb4b3a88df934871792119 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:04.314752 9308 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet ca477937278f45b4b115fec715f32a92 (table default.Table_69627cab65c2407b9226a05106b7bdf4 [id=fc5a0b5a96e6400da438307d26f73614]). Expected version 3 got 0
I20251027 21:08:04.314889 10029 raft_consensus.cc:359] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.315109 10029 raft_consensus.cc:385] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.315177 10029 raft_consensus.cc:740] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.315441 10029 consensus_queue.cc:260] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.315543 9308 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet d56d323ef0e74344a479940b93fea11a (table default.Table_cf11a8c344c544c5ba54672fee38e4aa [id=71e02b21c9654e45a41d3e450ac3fbe1]). Expected version 2 got 0
I20251027 21:08:04.317440 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_514660e85ec84a728ae8fa885f7c0532"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:04.317516 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c55370d227fd4e1baaca17e66bbaf3f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.318162 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_514660e85ec84a728ae8fa885f7c0532 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:04.318578 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c55370d227fd4e1baaca17e66bbaf3f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.303411 9739 tablet_service.cc:1505] Processing CreateTablet for tablet f0bbd665213c473e8e5c90bd9cb89d45 (DEFAULT_TABLE table=default.Table_ae5235e10b154ddd95f2aa9bb7d34a6f [id=1bfd7a5489dd4beda43d1a0caeb14b66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.319110 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0bbd665213c473e8e5c90bd9cb89d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.320382 10029 ts_tablet_manager.cc:1434] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251027 21:08:04.323690 10029 tablet_bootstrap.cc:492] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:04.325183 10029 tablet_bootstrap.cc:654] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.308508 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet a339c0dee0994788b8644912f74c1b82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.317126 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet f9534e65d8bb4b3a88df934871792119 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.328712 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3455942c45f2475ca47232041641c48f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.329319 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3455942c45f2475ca47232041641c48f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.333945 10031 raft_consensus.cc:359] T d56d323ef0e74344a479940b93fea11a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.334235 10031 raft_consensus.cc:385] T d56d323ef0e74344a479940b93fea11a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.334326 10031 raft_consensus.cc:740] T d56d323ef0e74344a479940b93fea11a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.334569 10031 consensus_queue.cc:260] T d56d323ef0e74344a479940b93fea11a P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.335196 10031 ts_tablet_manager.cc:1434] T d56d323ef0e74344a479940b93fea11a P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:04.335542 10029 tablet_bootstrap.cc:492] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:04.335577 10031 tablet_bootstrap.cc:492] T 4debc3f13f3a42939b6b83f6523aab7c P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:04.335690 10029 ts_tablet_manager.cc:1403] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
I20251027 21:08:04.337239 11018 raft_consensus.cc:2804] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:04.337329 11018 raft_consensus.cc:493] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:04.337397 11018 raft_consensus.cc:3060] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:04.340200 11018 raft_consensus.cc:515] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.341022 11018 leader_election.cc:290] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:04.341475 10029 raft_consensus.cc:359] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.341672 10029 raft_consensus.cc:385] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.341753 10029 raft_consensus.cc:740] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.341979 10029 consensus_queue.cc:260] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.342026 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca477937278f45b4b115fec715f32a92" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:04.346446 10029 ts_tablet_manager.cc:1434] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
W20251027 21:08:04.342633 9566 leader_election.cc:343] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:04.342417 9568 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 72952.3us
I20251027 21:08:04.347512 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca477937278f45b4b115fec715f32a92" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:04.347690 9924 raft_consensus.cc:3060] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:04.352257 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4996016747cd484cbda5f11e40e269b8 consensus state has no leader
I20251027 21:08:04.352702 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 0ac1aed5f56844639b848f90c8eb5b6c
I20251027 21:08:04.282214 10040 ts_tablet_manager.cc:1929] T d872cf3ef39042ff9a615b55e7195afa P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:04.352941 10040 log.cc:1199] T d872cf3ef39042ff9a615b55e7195afa P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/d872cf3ef39042ff9a615b55e7195afa
I20251027 21:08:04.353193 9304 catalog_manager.cc:6719] ReplaceTablet: tablet 0ac1aed5f56844639b848f90c8eb5b6c deleted and replaced by tablet febc011f42244fdfa1954dbb630c3d0c
I20251027 21:08:04.353364 10040 ts_tablet_manager.cc:1950] T d872cf3ef39042ff9a615b55e7195afa P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:04.353910 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ac1aed5f56844639b848f90c8eb5b6c with delete_type TABLET_DATA_DELETED (replaced by tablet febc011f42244fdfa1954dbb630c3d0c) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.354054 9924 raft_consensus.cc:2468] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:04.354357 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet d872cf3ef39042ff9a615b55e7195afa (table default.Table_2c2ec3f9d23142c9b8c2d66d1c979525 [id=59de715802f547bca16f181917fcc418]) successfully deleted
I20251027 21:08:04.354933 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ac1aed5f56844639b848f90c8eb5b6c with delete_type TABLET_DATA_DELETED (replaced by tablet febc011f42244fdfa1954dbb630c3d0c) from {username='slave'} at 127.0.0.1:34644
W20251027 21:08:04.355100 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 196a594025ad4564b028b39f1afae48f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 196a594025ad4564b028b39f1afae48f already in progress: creating tablet
I20251027 21:08:04.355394 9307 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet abd3b8e71d3041b4b9e5f6663841bfcf (table default.Table_fd331f3cfc07425082d58ab5af3c1094 [id=f3e3147411e5488bae105ed755f5b50f]). Expected version 2 got 0
I20251027 21:08:04.355492 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet 96a680e8f16343b1908829a67e960b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:04.355728 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 070167be7d8d4ddf8da561df14448360 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 070167be7d8d4ddf8da561df14448360 already in progress: creating tablet
W20251027 21:08:04.356993 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 96a680e8f16343b1908829a67e960b60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96a680e8f16343b1908829a67e960b60 already in progress: creating tablet
W20251027 21:08:04.357758 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c6a189aa22d048858e9f30bf4276859d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6a189aa22d048858e9f30bf4276859d already in progress: creating tablet
I20251027 21:08:04.358100 10040 tablet_replica.cc:333] T a8328318a4f14b38a4ce36d183e99dfb P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:04.358381 10040 raft_consensus.cc:2243] T a8328318a4f14b38a4ce36d183e99dfb P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:04.358604 10040 raft_consensus.cc:2272] T a8328318a4f14b38a4ce36d183e99dfb P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:04.359282 10029 tablet_bootstrap.cc:492] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:04.359879 9601 tablet_service.cc:1505] Processing CreateTablet for tablet fdc215e390ca4b93bdee902f959843fb (DEFAULT_TABLE table=default.Table_df2d8d09ba3147afb5c9271da5e23d45 [id=a868a439c6b142068daad14573426d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.360219 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdc215e390ca4b93bdee902f959843fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.361608 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 8b1e5ab297da49058e5f8d27f2fba65d (DEFAULT_TABLE table=default.Table_555cf66116b243de86d5cdd0daf42151 [id=10fb7362e0034567b2f121befc2da872]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.361990 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b1e5ab297da49058e5f8d27f2fba65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.363286 9890 tablet_service.cc:1505] Processing CreateTablet for tablet f0bbd665213c473e8e5c90bd9cb89d45 (DEFAULT_TABLE table=default.Table_ae5235e10b154ddd95f2aa9bb7d34a6f [id=1bfd7a5489dd4beda43d1a0caeb14b66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.363653 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0bbd665213c473e8e5c90bd9cb89d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.366343 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f22296338b56467193b136c872ba30c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.366894 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f22296338b56467193b136c872ba30c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.369772 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f396e1e1ebc44cf9a774f6523942ff0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.370281 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f396e1e1ebc44cf9a774f6523942ff0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.373308 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9cd91730e4c4415e9f65d748f4592366" } new_table_name: "default.Table_508fcc20bab34856b1fb148b6fd6bf3c" modify_external_catalogs: true
I20251027 21:08:04.379751 10032 tablet_bootstrap.cc:492] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:04.379969 10032 ts_tablet_manager.cc:1403] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.124s user 0.004s sys 0.010s
I20251027 21:08:04.380606 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0393cf4c52244b19b2f59cc4a13f8dee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.381064 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0393cf4c52244b19b2f59cc4a13f8dee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.381528 10032 raft_consensus.cc:359] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:04.381775 10032 raft_consensus.cc:385] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.381796 11018 consensus_queue.cc:1048] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:04.381871 10032 raft_consensus.cc:740] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.381893 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 8b1e5ab297da49058e5f8d27f2fba65d (DEFAULT_TABLE table=default.Table_555cf66116b243de86d5cdd0daf42151 [id=10fb7362e0034567b2f121befc2da872]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.383639 9742 tablet_service.cc:1505] Processing CreateTablet for tablet b260fe61fd6d43d59ef7fd77a2532cc2 (DEFAULT_TABLE table=default.Table_d3be7b0de84549a38b43f01c7efd058a [id=e9131618c01649e8a5ea1a9b4e713075]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.384135 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b260fe61fd6d43d59ef7fd77a2532cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.382766 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b1e5ab297da49058e5f8d27f2fba65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.382241 10032 consensus_queue.cc:260] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:04.388188 10032 ts_tablet_manager.cc:1434] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20251027 21:08:04.388852 10032 tablet_bootstrap.cc:492] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:04.390074 10032 tablet_bootstrap.cc:654] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.391214 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_401e3aad77324534a4e47f63fe64de89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.391708 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_401e3aad77324534a4e47f63fe64de89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.396188 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet fde0980aa64c4496861a662f7ed3ef75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:04.398017 9682 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:08:04.408924 10032 tablet_bootstrap.cc:492] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:04.409045 10032 ts_tablet_manager.cc:1403] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20251027 21:08:04.410137 10032 raft_consensus.cc:359] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:04.410346 10032 raft_consensus.cc:385] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.410413 10032 raft_consensus.cc:740] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.410626 10032 consensus_queue.cc:260] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:04.411497 10032 ts_tablet_manager.cc:1434] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251027 21:08:04.412503 10032 tablet_bootstrap.cc:492] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:04.413064 9820 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:08:04.413842 10032 tablet_bootstrap.cc:654] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.415858 9970 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:08:04.418393 9895 tablet_service.cc:1505] Processing CreateTablet for tablet b260fe61fd6d43d59ef7fd77a2532cc2 (DEFAULT_TABLE table=default.Table_d3be7b0de84549a38b43f01c7efd058a [id=e9131618c01649e8a5ea1a9b4e713075]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.419075 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b260fe61fd6d43d59ef7fd77a2532cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.419991 10040 ts_tablet_manager.cc:1916] T a8328318a4f14b38a4ce36d183e99dfb P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:04.420230 10955 raft_consensus.cc:493] T e6b58b1f4f63461aa96e416cae879be8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.420342 10955 raft_consensus.cc:515] T e6b58b1f4f63461aa96e416cae879be8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:04.420832 10955 leader_election.cc:290] T e6b58b1f4f63461aa96e416cae879be8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:04.425343 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6b58b1f4f63461aa96e416cae879be8" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:04.425894 9859 leader_election.cc:343] T e6b58b1f4f63461aa96e416cae879be8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.426472 9304 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet abd3b8e71d3041b4b9e5f6663841bfcf (table default.Table_fd331f3cfc07425082d58ab5af3c1094 [id=f3e3147411e5488bae105ed755f5b50f]). Expected version 2 got 0
I20251027 21:08:04.426559 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d64a13667194c12864fa388eb88a380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:04.426678 9304 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet d56d323ef0e74344a479940b93fea11a (table default.Table_cf11a8c344c544c5ba54672fee38e4aa [id=71e02b21c9654e45a41d3e450ac3fbe1]). Expected version 2 got 0
I20251027 21:08:04.434515 9820 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:08:04.441490 10031 tablet_bootstrap.cc:654] T 4debc3f13f3a42939b6b83f6523aab7c P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.442755 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet c6a189aa22d048858e9f30bf4276859d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.443614 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet f9534e65d8bb4b3a88df934871792119 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.444077 9890 tablet_service.cc:1505] Processing CreateTablet for tablet bc505d437e774afcbd030a4e88375649 (DEFAULT_TABLE table=default.Table_303556148a4445a191ca828fb6582bd2 [id=3cf765f6ab1a4479b2b26b63ad522b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.444447 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc505d437e774afcbd030a4e88375649. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.444787 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3fa040a0070246308ad0028af0f86ae2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.445474 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fa040a0070246308ad0028af0f86ae2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.447791 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2374f13424064567a276cca554d2aff8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.448324 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2374f13424064567a276cca554d2aff8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.449627 9748 tablet_service.cc:1505] Processing CreateTablet for tablet bc505d437e774afcbd030a4e88375649 (DEFAULT_TABLE table=default.Table_303556148a4445a191ca828fb6582bd2 [id=3cf765f6ab1a4479b2b26b63ad522b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.450059 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc505d437e774afcbd030a4e88375649. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.451130 10029 tablet_bootstrap.cc:654] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.452037 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ac1aed5f56844639b848f90c8eb5b6c with delete_type TABLET_DATA_DELETED (replaced by tablet febc011f42244fdfa1954dbb630c3d0c) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.452548 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_19e1b54d4b7f4d24b1de893d1229bcdb" } modify_external_catalogs: true
I20251027 21:08:04.452786 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_19e1b54d4b7f4d24b1de893d1229bcdb" } modify_external_catalogs: true
I20251027 21:08:04.452993 9601 tablet_service.cc:1505] Processing CreateTablet for tablet f0bbd665213c473e8e5c90bd9cb89d45 (DEFAULT_TABLE table=default.Table_ae5235e10b154ddd95f2aa9bb7d34a6f [id=1bfd7a5489dd4beda43d1a0caeb14b66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.453315 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0bbd665213c473e8e5c90bd9cb89d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.454963 9739 tablet_service.cc:1505] Processing CreateTablet for tablet e2645a6c003a4523ac32787f6c6d3d1d (DEFAULT_TABLE table=default.Table_439c4d078d484b7da5f47d76aca5a4a2 [id=70a81b05d2bd4324b6527d1668db07d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.455404 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2645a6c003a4523ac32787f6c6d3d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.458662 9308 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 45568792d7954b498581ce69fb611948 (table default.Table_3e3e0c17a3bf452fbb217c6e3ee8bdcb [id=2eb093cd58e643a58b14e564d5914fbc]). Expected version 1 got 0
I20251027 21:08:04.459939 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_37d4aa59b4364a9ebcfe4096d3250e49" } modify_external_catalogs: true
I20251027 21:08:04.460238 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_37d4aa59b4364a9ebcfe4096d3250e49" } modify_external_catalogs: true
I20251027 21:08:04.460502 9609 tablet_service.cc:1505] Processing CreateTablet for tablet b260fe61fd6d43d59ef7fd77a2532cc2 (DEFAULT_TABLE table=default.Table_d3be7b0de84549a38b43f01c7efd058a [id=e9131618c01649e8a5ea1a9b4e713075]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.460793 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b260fe61fd6d43d59ef7fd77a2532cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.466985 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet a26ded6012174348b7295fc7c0e05e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.467518 10043 tablet_replica.cc:333] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:04.467746 10043 raft_consensus.cc:2243] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:04.467772 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 67bb12de9f044c73ae9ab26d10498c90 with delete_type TABLET_DATA_DELETED (replaced by tablet ea02ab2fed634dbcba3ca8b04808b694) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.467892 10043 raft_consensus.cc:2272] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:04.468603 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet a8328318a4f14b38a4ce36d183e99dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.468640 10043 ts_tablet_manager.cc:1916] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:04.469509 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 8b1e5ab297da49058e5f8d27f2fba65d (DEFAULT_TABLE table=default.Table_555cf66116b243de86d5cdd0daf42151 [id=10fb7362e0034567b2f121befc2da872]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.469779 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet a339c0dee0994788b8644912f74c1b82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.470089 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet e9edb62a8e4b4dfea6a31a2202545a6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.471974 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b1e5ab297da49058e5f8d27f2fba65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.475996 10829 raft_consensus.cc:493] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.476177 10829 raft_consensus.cc:515] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.476279 10997 raft_consensus.cc:493] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.476925 10997 raft_consensus.cc:515] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.477567 10997 leader_election.cc:290] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:04.478055 9711 leader_election.cc:343] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 2532bf7190064443a898e47e843fd9ac
I20251027 21:08:04.476943 10829 leader_election.cc:290] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:04.478269 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccf10ef559f74e2db28023fd520c8de9" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:04.478495 9924 raft_consensus.cc:2393] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77b0513f17c0472088901a1ce8b08218 in current term 1: Already voted for candidate 51902b6b61ff4d5584ff21896e847e14 in this term.
I20251027 21:08:04.480770 9890 tablet_service.cc:1505] Processing CreateTablet for tablet e2645a6c003a4523ac32787f6c6d3d1d (DEFAULT_TABLE table=default.Table_439c4d078d484b7da5f47d76aca5a4a2 [id=70a81b05d2bd4324b6527d1668db07d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.481208 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2645a6c003a4523ac32787f6c6d3d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.481884 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e9721dd5a67f48559827c11b5944ba0b" } new_table_name: "default.Table_c6bc5b44dd844823b76718c93f89a53d" modify_external_catalogs: true
I20251027 21:08:04.477614 10031 tablet_bootstrap.cc:492] T 4debc3f13f3a42939b6b83f6523aab7c P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:04.484577 10031 ts_tablet_manager.cc:1403] T 4debc3f13f3a42939b6b83f6523aab7c P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.149s user 0.004s sys 0.000s
I20251027 21:08:04.470484 9615 tablet_service.cc:1505] Processing CreateTablet for tablet bc505d437e774afcbd030a4e88375649 (DEFAULT_TABLE table=default.Table_303556148a4445a191ca828fb6582bd2 [id=3cf765f6ab1a4479b2b26b63ad522b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.485702 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc505d437e774afcbd030a4e88375649. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.485635 10031 raft_consensus.cc:359] T 4debc3f13f3a42939b6b83f6523aab7c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.485850 10031 raft_consensus.cc:385] T 4debc3f13f3a42939b6b83f6523aab7c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.485942 10031 raft_consensus.cc:740] T 4debc3f13f3a42939b6b83f6523aab7c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.486284 10031 consensus_queue.cc:260] T 4debc3f13f3a42939b6b83f6523aab7c P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.470252 9568 leader_election.cc:304] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:04.487922 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 96a680e8f16343b1908829a67e960b60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96a680e8f16343b1908829a67e960b60 already in progress: creating tablet
I20251027 21:08:04.488473 10043 ts_tablet_manager.cc:1929] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:04.488561 10043 log.cc:1199] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/a26ded6012174348b7295fc7c0e05e6e
I20251027 21:08:04.488821 10043 ts_tablet_manager.cc:1950] T a26ded6012174348b7295fc7c0e05e6e P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
W20251027 21:08:04.489503 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet c6a189aa22d048858e9f30bf4276859d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6a189aa22d048858e9f30bf4276859d already in progress: creating tablet
I20251027 21:08:04.489568 10043 tablet_replica.cc:333] T 67bb12de9f044c73ae9ab26d10498c90 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:04.489744 10043 raft_consensus.cc:2243] T 67bb12de9f044c73ae9ab26d10498c90 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:04.489877 10043 raft_consensus.cc:2272] T 67bb12de9f044c73ae9ab26d10498c90 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:04.489959 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet f9534e65d8bb4b3a88df934871792119 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9534e65d8bb4b3a88df934871792119 already in progress: creating tablet
W20251027 21:08:04.490419 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 0ac1aed5f56844639b848f90c8eb5b6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ac1aed5f56844639b848f90c8eb5b6c already in progress: creating tablet
I20251027 21:08:04.491669 9609 tablet_service.cc:1505] Processing CreateTablet for tablet e2645a6c003a4523ac32787f6c6d3d1d (DEFAULT_TABLE table=default.Table_439c4d078d484b7da5f47d76aca5a4a2 [id=70a81b05d2bd4324b6527d1668db07d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.492120 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2645a6c003a4523ac32787f6c6d3d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.493063 10907 raft_consensus.cc:2804] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:04.494395 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet a26ded6012174348b7295fc7c0e05e6e (table default.Table_69f618a7f3c74a1d87cb5e0c17f38f94 [id=d37bad05f12e4d0da8934baeb7627de6]) successfully deleted
I20251027 21:08:04.496310 10955 raft_consensus.cc:493] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.496454 10955 raft_consensus.cc:515] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:04.497036 10955 leader_election.cc:290] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:04.497575 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "631ff72613734913818a3cad24f20dbc" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:04.498153 9859 leader_election.cc:343] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.503363 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c9999fc8fde44b77aecd45e35a92bdbc" } new_table_name: "default.Table_3cb6bc9f33ea40aa9d58e82f709c64b7" modify_external_catalogs: true
I20251027 21:08:04.505939 9739 tablet_service.cc:1505] Processing CreateTablet for tablet ece28e82383540bea3457b1cbc65558b (DEFAULT_TABLE table=default.Table_0668ac874959449eb530dd484f0c1d73 [id=7fbba4a31725468aacff7a9684d8fd4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.506318 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ece28e82383540bea3457b1cbc65558b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.509162 9905 tablet_service.cc:1505] Processing CreateTablet for tablet ece28e82383540bea3457b1cbc65558b (DEFAULT_TABLE table=default.Table_0668ac874959449eb530dd484f0c1d73 [id=7fbba4a31725468aacff7a9684d8fd4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.509932 9748 tablet_service.cc:1505] Processing CreateTablet for tablet ba2519c12a6e49b994ca2709cf3c2ba4 (DEFAULT_TABLE table=default.Table_c22259490f91403e8b54952e22c9924b [id=b69d97562e264e52bc874d621eefec53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.510447 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba2519c12a6e49b994ca2709cf3c2ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.510921 10955 raft_consensus.cc:493] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.511045 10955 raft_consensus.cc:515] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:04.511595 10955 leader_election.cc:290] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:04.511826 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 901d4f08b2a24b9b8cddcff6255ea98f (DEFAULT_TABLE table=default.Table_c9eb751ca4b7420a9d7557aa95e30ed6 [id=c17f423ce5f248de86ce7f1fe820f9bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.511906 9898 tablet_service.cc:1505] Processing CreateTablet for tablet ba2519c12a6e49b994ca2709cf3c2ba4 (DEFAULT_TABLE table=default.Table_c22259490f91403e8b54952e22c9924b [id=b69d97562e264e52bc874d621eefec53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.512146 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 901d4f08b2a24b9b8cddcff6255ea98f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:04.517467 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9d19e9f63acf4c8c894af38a02225fc6 consensus state has no leader
I20251027 21:08:04.519553 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fa39697360c46138ef4cd21e8c7a950" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:04.522776 9859 leader_election.cc:343] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:04.531639 9710 leader_election.cc:343] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 2532bf7190064443a898e47e843fd9ac
I20251027 21:08:04.531767 9710 leader_election.cc:304] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:04.531947 10955 raft_consensus.cc:493] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.532142 10907 raft_consensus.cc:697] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:04.532100 10955 raft_consensus.cc:515] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:04.532455 10907 consensus_queue.cc:237] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.532621 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6b58b1f4f63461aa96e416cae879be8" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:04.532714 10955 leader_election.cc:290] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:04.533064 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c2d713d72fd4b5ca34cb3e828c75761" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:04.502017 10043 ts_tablet_manager.cc:1916] T 67bb12de9f044c73ae9ab26d10498c90 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:04.504817 11051 consensus_queue.cc:1048] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:04.507891 9609 tablet_service.cc:1505] Processing CreateTablet for tablet ece28e82383540bea3457b1cbc65558b (DEFAULT_TABLE table=default.Table_0668ac874959449eb530dd484f0c1d73 [id=7fbba4a31725468aacff7a9684d8fd4f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:04.534719 9859 leader_election.cc:343] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.536226 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ece28e82383540bea3457b1cbc65558b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.540623 10043 ts_tablet_manager.cc:1929] T 67bb12de9f044c73ae9ab26d10498c90 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251027 21:08:04.541167 9860 leader_election.cc:343] T e6b58b1f4f63461aa96e416cae879be8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.510378 9601 tablet_service.cc:1505] Processing CreateTablet for tablet ba2519c12a6e49b994ca2709cf3c2ba4 (DEFAULT_TABLE table=default.Table_c22259490f91403e8b54952e22c9924b [id=b69d97562e264e52bc874d621eefec53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.541289 9860 leader_election.cc:304] T e6b58b1f4f63461aa96e416cae879be8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:04.541479 10955 raft_consensus.cc:2749] T e6b58b1f4f63461aa96e416cae879be8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:04.542032 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba2519c12a6e49b994ca2709cf3c2ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.530203 11057 raft_consensus.cc:493] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.530869 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccf10ef559f74e2db28023fd520c8de9" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:04.544173 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c2d713d72fd4b5ca34cb3e828c75761" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:04.544219 11057 raft_consensus.cc:515] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.531162 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "631ff72613734913818a3cad24f20dbc" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:04.531263 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fa39697360c46138ef4cd21e8c7a950" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:04.544750 11057 leader_election.cc:290] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:04.545049 9860 leader_election.cc:343] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.545146 9860 leader_election.cc:304] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:04.545362 10955 raft_consensus.cc:2749] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:04.545687 9568 leader_election.cc:343] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: e9edb62a8e4b4dfea6a31a2202545a6e
W20251027 21:08:04.545835 9860 leader_election.cc:343] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.545939 9860 leader_election.cc:304] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:04.546154 10955 raft_consensus.cc:2749] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:04.546398 9860 leader_election.cc:343] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.546504 9860 leader_election.cc:304] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:04.546665 10955 raft_consensus.cc:2749] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:04.552268 10029 tablet_bootstrap.cc:492] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:04.552430 10029 ts_tablet_manager.cc:1403] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.193s user 0.003s sys 0.000s
I20251027 21:08:04.553287 10029 raft_consensus.cc:359] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.553484 10029 raft_consensus.cc:385] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.553565 10029 raft_consensus.cc:740] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.553776 10029 consensus_queue.cc:260] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.554258 10029 ts_tablet_manager.cc:1434] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:04.554524 10029 tablet_bootstrap.cc:492] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:04.554592 9710 leader_election.cc:304] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:04.555989 10029 tablet_bootstrap.cc:654] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.556178 10997 raft_consensus.cc:2749] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:04.556428 10997 raft_consensus.cc:3060] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:04.559927 10997 raft_consensus.cc:2749] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:04.560999 10829 raft_consensus.cc:493] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.561182 10829 raft_consensus.cc:515] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.561858 10829 leader_election.cc:290] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:04.562299 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9edb62a8e4b4dfea6a31a2202545a6e" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:04.563161 9566 leader_election.cc:343] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.563273 9566 leader_election.cc:304] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:04.565071 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f1d1bd52556463fa306e51eb253af40" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:04.565384 9925 raft_consensus.cc:2393] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77b0513f17c0472088901a1ce8b08218 in current term 1: Already voted for candidate 51902b6b61ff4d5584ff21896e847e14 in this term.
I20251027 21:08:04.567406 11018 raft_consensus.cc:2749] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:04.568311 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f1d1bd52556463fa306e51eb253af40" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:04.568912 10040 ts_tablet_manager.cc:1929] T a8328318a4f14b38a4ce36d183e99dfb P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:04.569083 10040 log.cc:1199] T a8328318a4f14b38a4ce36d183e99dfb P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/a8328318a4f14b38a4ce36d183e99dfb
I20251027 21:08:04.569480 10040 ts_tablet_manager.cc:1950] T a8328318a4f14b38a4ce36d183e99dfb P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:04.570487 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet a8328318a4f14b38a4ce36d183e99dfb (table default.Table_526d1cbb28784221b43a6719685c5e2a [id=287175ad40544f73a586c736fcce910b]) successfully deleted
I20251027 21:08:04.571174 10955 raft_consensus.cc:493] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.571298 10955 raft_consensus.cc:515] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:04.571465 9710 leader_election.cc:304] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:04.571812 10829 raft_consensus.cc:3060] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:04.571908 10955 leader_election.cc:290] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:04.575238 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c34c8f1433b40d491ae05e7c6e60ba1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:04.575881 9859 leader_election.cc:343] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.577212 10829 raft_consensus.cc:2749] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:04.578419 11040 sys_catalog.cc:263] Verifying existing consensus state
I20251027 21:08:04.580703 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c34c8f1433b40d491ae05e7c6e60ba1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:04.581851 10031 ts_tablet_manager.cc:1434] T 4debc3f13f3a42939b6b83f6523aab7c P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.097s user 0.002s sys 0.000s
I20251027 21:08:04.582237 10955 raft_consensus.cc:493] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251027 21:08:04.582278 9860 leader_election.cc:343] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.582374 9860 leader_election.cc:304] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:04.582351 10955 raft_consensus.cc:515] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:04.582836 10955 leader_election.cc:290] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:04.582954 10955 raft_consensus.cc:2749] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:04.584900 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83f5fbf5757a49df93ab949b7657c61f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:04.585482 9859 leader_election.cc:343] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.588423 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 901d4f08b2a24b9b8cddcff6255ea98f (DEFAULT_TABLE table=default.Table_c9eb751ca4b7420a9d7557aa95e30ed6 [id=c17f423ce5f248de86ce7f1fe820f9bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.588821 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 901d4f08b2a24b9b8cddcff6255ea98f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.590416 10031 tablet_bootstrap.cc:492] T ca477937278f45b4b115fec715f32a92 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:04.592605 10031 tablet_bootstrap.cc:654] T ca477937278f45b4b115fec715f32a92 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.605091 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83f5fbf5757a49df93ab949b7657c61f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:04.607826 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 7062871e266c4603bab851a837ef9aec (DEFAULT_TABLE table=default.Table_7aab2b63c44043aa9aea339ec0b5763e [id=9defb05f895749e9a76b7653ad9ad789]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.608342 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7062871e266c4603bab851a837ef9aec. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:04.613389 9860 leader_election.cc:343] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.613557 9860 leader_election.cc:304] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:04.614076 10955 raft_consensus.cc:2749] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:04.616851 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 7062871e266c4603bab851a837ef9aec (DEFAULT_TABLE table=default.Table_7aab2b63c44043aa9aea339ec0b5763e [id=9defb05f895749e9a76b7653ad9ad789]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.617501 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 901d4f08b2a24b9b8cddcff6255ea98f (DEFAULT_TABLE table=default.Table_c9eb751ca4b7420a9d7557aa95e30ed6 [id=c17f423ce5f248de86ce7f1fe820f9bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.624895 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 719321644ea3479ea4529a8a5d048754
I20251027 21:08:04.625569 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 719321644ea3479ea4529a8a5d048754 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.626184 10043 log.cc:1199] T 67bb12de9f044c73ae9ab26d10498c90 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/67bb12de9f044c73ae9ab26d10498c90
I20251027 21:08:04.626575 10043 ts_tablet_manager.cc:1950] T 67bb12de9f044c73ae9ab26d10498c90 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:04.626709 9888 tablet_service.cc:1552] Processing DeleteTablet for tablet 719321644ea3479ea4529a8a5d048754 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:04.627386 10043 tablet_replica.cc:333] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:04.627497 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 67bb12de9f044c73ae9ab26d10498c90 (table default.Table_0ce16ec9dc8b44eab80cedc074577cba [id=c22fee46f319480e88ab7745f6a95126]) successfully deleted
I20251027 21:08:04.627557 10043 raft_consensus.cc:2243] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:04.627526 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 719321644ea3479ea4529a8a5d048754 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.627714 10043 raft_consensus.cc:2272] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:04.628172 10043 ts_tablet_manager.cc:1916] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:04.628477 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet b83daa33ef0f45c6a78c70001e532ea2
I20251027 21:08:04.629045 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet b83daa33ef0f45c6a78c70001e532ea2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.630131 9888 tablet_service.cc:1552] Processing DeleteTablet for tablet b83daa33ef0f45c6a78c70001e532ea2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:04.630822 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet b83daa33ef0f45c6a78c70001e532ea2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.631868 10041 ts_tablet_manager.cc:1929] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:04.632167 10041 log.cc:1199] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/a9339c6b1154400eace53d765528fdf9
I20251027 21:08:04.632864 10041 ts_tablet_manager.cc:1950] T a9339c6b1154400eace53d765528fdf9 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:04.578663 10040 tablet_replica.cc:333] T 53cac7817db440e79e7416d625a9b2c8 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:04.634298 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ece28e82383540bea3457b1cbc65558b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.634691 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba2519c12a6e49b994ca2709cf3c2ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.635596 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 901d4f08b2a24b9b8cddcff6255ea98f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.640365 11040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: Bootstrap starting.
I20251027 21:08:04.644197 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7062871e266c4603bab851a837ef9aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:04.650322 10040 raft_consensus.cc:2243] T 53cac7817db440e79e7416d625a9b2c8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:04.650561 10040 raft_consensus.cc:2272] T 53cac7817db440e79e7416d625a9b2c8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:04.651166 10040 ts_tablet_manager.cc:1916] T 53cac7817db440e79e7416d625a9b2c8 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:04.652310 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e07797daedc47ee9dfb5daa36b15eef consensus state has no leader
I20251027 21:08:04.653259 9769 raft_consensus.cc:3060] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:04.671337 10043 ts_tablet_manager.cc:1929] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:04.671461 10043 log.cc:1199] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/a8328318a4f14b38a4ce36d183e99dfb
I20251027 21:08:04.672583 9769 raft_consensus.cc:1275] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Refusing update from remote peer 51902b6b61ff4d5584ff21896e847e14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 21:08:04.673524 10829 raft_consensus.cc:493] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.673664 11018 consensus_queue.cc:1048] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:04.673723 10829 raft_consensus.cc:515] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.674000 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 907) took 1183 ms. Trace:
I20251027 21:08:04.675100 10029 tablet_bootstrap.cc:492] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:04.675218 10029 ts_tablet_manager.cc:1403] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.121s user 0.006s sys 0.001s
I20251027 21:08:04.675937 10029 raft_consensus.cc:359] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.676084 10029 raft_consensus.cc:385] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.674881 10041 rpcz_store.cc:276] 1027 21:08:03.490072 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:03.490826 (+ 754us) service_pool.cc:225] Handling call
1027 21:08:04.134556 (+643730us) ts_tablet_manager.cc:1151] Deleting tablet a9339c6b1154400eace53d765528fdf9
1027 21:08:04.134566 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:04.281568 (+147002us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:04.631762 (+350194us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:04.673610 (+ 41848us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":643558}
I20251027 21:08:04.676133 10029 raft_consensus.cc:740] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.676337 10029 consensus_queue.cc:260] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.676951 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet a9339c6b1154400eace53d765528fdf9 (table default.Table_5a6211c8465040dbad1cc86377a370ea [id=91fb9f95fe9343fca1d8cb0d0f781afb]) successfully deleted
I20251027 21:08:04.677804 10829 leader_election.cc:290] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:04.678454 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "345d3795a37b4a40b40967cfacf0f893" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:04.678267 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 910) took 1171 ms. Trace:
I20251027 21:08:04.678752 9925 raft_consensus.cc:2468] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:04.679445 9711 leader_election.cc:304] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:04.678964 10041 rpcz_store.cc:276] 1027 21:08:03.506396 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:03.514544 (+ 8148us) service_pool.cc:225] Handling call
1027 21:08:04.677234 (+1162690us) ts_tablet_manager.cc:1151] Deleting tablet 8db666d9b7eb458b843ab5910cd7dff4
1027 21:08:04.677243 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:04.678250 (+ 1007us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1162460}
I20251027 21:08:04.679975 10829 raft_consensus.cc:2804] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251027 21:08:04.680055 10829 raft_consensus.cc:493] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:04.680125 10829 raft_consensus.cc:3060] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Advancing to term 2
I20251027 21:08:04.681339 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "345d3795a37b4a40b40967cfacf0f893" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:04.681401 10041 tablet_replica.cc:333] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
W20251027 21:08:04.682614 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 8db666d9b7eb458b843ab5910cd7dff4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8db666d9b7eb458b843ab5910cd7dff4 already in progress: creating tablet
I20251027 21:08:04.682685 10041 raft_consensus.cc:2243] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:04.683832 10029 ts_tablet_manager.cc:1434] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251027 21:08:04.683758 10041 raft_consensus.cc:2272] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:04.687739 10041 ts_tablet_manager.cc:1916] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:04.688541 11018 consensus_queue.cc:1048] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:04.688201 9924 raft_consensus.cc:1275] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Refusing update from remote peer 51902b6b61ff4d5584ff21896e847e14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 21:08:04.689942 11018 consensus_queue.cc:1048] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:04.691807 10029 tablet_bootstrap.cc:492] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:04.695639 10029 tablet_bootstrap.cc:654] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.696394 10043 ts_tablet_manager.cc:1950] T a8328318a4f14b38a4ce36d183e99dfb P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:04.719156 10041 ts_tablet_manager.cc:1929] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20251027 21:08:04.719522 10041 log.cc:1199] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/d13d00353b6740b5bed5a153936d2059
I20251027 21:08:04.720952 10041 ts_tablet_manager.cc:1950] T d13d00353b6740b5bed5a153936d2059 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:04.724098 11051 raft_consensus.cc:493] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.724261 11051 raft_consensus.cc:515] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.725005 11051 leader_election.cc:290] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:04.725397 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4debc3f13f3a42939b6b83f6523aab7c" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:04.725600 9772 raft_consensus.cc:2468] T 4debc3f13f3a42939b6b83f6523aab7c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:04.725986 9566 leader_election.cc:304] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:04.726298 11057 raft_consensus.cc:2804] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:04.726385 11057 raft_consensus.cc:493] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:04.726456 11057 raft_consensus.cc:3060] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:04.726194 11071 tablet.cc:1722] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251027 21:08:04.730119 9568 leader_election.cc:343] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 4debc3f13f3a42939b6b83f6523aab7c
I20251027 21:08:04.730408 10043 tablet_replica.cc:333] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:04.730573 10043 raft_consensus.cc:2243] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:04.730727 10043 raft_consensus.cc:2272] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:04.731187 10043 ts_tablet_manager.cc:1916] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:04.731379 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet a8328318a4f14b38a4ce36d183e99dfb (table default.Table_526d1cbb28784221b43a6719685c5e2a [id=287175ad40544f73a586c736fcce910b]) successfully deleted
I20251027 21:08:04.731635 10907 raft_consensus.cc:493] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.731771 10907 raft_consensus.cc:515] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.732367 10907 leader_election.cc:290] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:04.734191 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3b9a461567a48fcb4af514ecdf618b4" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:04.734309 9568 leader_election.cc:343] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: c3b9a461567a48fcb4af514ecdf618b4
W20251027 21:08:04.734597 9566 leader_election.cc:343] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.734726 9566 leader_election.cc:304] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:04.734927 11051 raft_consensus.cc:2749] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:04.735402 11051 raft_consensus.cc:493] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.735531 11051 raft_consensus.cc:515] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.736186 11051 leader_election.cc:290] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:04.737506 9568 leader_election.cc:343] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 32f80614e30d41128f9fb2e0ea165223
I20251027 21:08:04.737874 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32f80614e30d41128f9fb2e0ea165223" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:04.738473 9566 leader_election.cc:343] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.738582 9566 leader_election.cc:304] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:04.739235 11051 raft_consensus.cc:2749] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:04.745466 11051 consensus_queue.cc:1048] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:04.748149 10829 raft_consensus.cc:515] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.749962 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "345d3795a37b4a40b40967cfacf0f893" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:04.750042 9769 raft_consensus.cc:1240] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [term 2 FOLLOWER]: Rejecting Update request from peer 51902b6b61ff4d5584ff21896e847e14 for earlier term 1. Current term is 2. Ops: [1.1-1.2]
I20251027 21:08:04.751686 11051 consensus_queue.cc:1059] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:04.752207 10029 tablet_bootstrap.cc:492] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:04.752375 10029 ts_tablet_manager.cc:1403] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.061s user 0.001s sys 0.002s
I20251027 21:08:04.753070 10029 raft_consensus.cc:359] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.753248 10029 raft_consensus.cc:385] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.753330 10029 raft_consensus.cc:740] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.753558 10829 leader_election.cc:290] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 2 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:04.753554 10029 consensus_queue.cc:260] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.753600 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "345d3795a37b4a40b40967cfacf0f893" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:04.754379 9711 leader_election.cc:304] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:04.754734 10997 raft_consensus.cc:2749] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251027 21:08:04.755618 10029 ts_tablet_manager.cc:1434] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:08:04.755865 10029 tablet_bootstrap.cc:492] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:04.758631 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_456ae12ae5d04554b252e71a72c3d9c1" } modify_external_catalogs: true
I20251027 21:08:04.758881 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_456ae12ae5d04554b252e71a72c3d9c1" } modify_external_catalogs: true
I20251027 21:08:04.760361 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0a195bfbfcbc4c1484188d25d0399353" } new_table_name: "default.Table_6bb3152125264bc6a28eda3f88cee649" modify_external_catalogs: true
I20251027 21:08:04.762480 10029 tablet_bootstrap.cc:654] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.763028 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 45568792d7954b498581ce69fb611948 (table default.Table_3e3e0c17a3bf452fbb217c6e3ee8bdcb [id=2eb093cd58e643a58b14e564d5914fbc]). Expected version 1 got 0
I20251027 21:08:04.763309 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet abd3b8e71d3041b4b9e5f6663841bfcf (table default.Table_fd331f3cfc07425082d58ab5af3c1094 [id=f3e3147411e5488bae105ed755f5b50f]). Expected version 2 got 0
I20251027 21:08:04.763614 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet b8856c5dc8f144358445165324bfb477 (table default.Table_203191e192be4e3a9a7477b4313ef4f4 [id=5da6bd4463e24c03accd9985f72b6ebb]). Expected version 3 got 0
I20251027 21:08:04.765413 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 32f80614e30d41128f9fb2e0ea165223 with delete_type TABLET_DATA_DELETED (replaced by tablet a7d8d060916941a5b7e944e8b08a2882) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.765576 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet 4debc3f13f3a42939b6b83f6523aab7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.767621 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_94b3855384ef4019bf69b5ff2ccc6dd6" } modify_external_catalogs: true
I20251027 21:08:04.767810 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_94b3855384ef4019bf69b5ff2ccc6dd6" } modify_external_catalogs: true
I20251027 21:08:04.769507 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_bff95c9aec0642ed94a4b1355a7a32af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.770031 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bff95c9aec0642ed94a4b1355a7a32af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.771324 9303 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 45568792d7954b498581ce69fb611948 (table default.Table_3e3e0c17a3bf452fbb217c6e3ee8bdcb [id=2eb093cd58e643a58b14e564d5914fbc]). Expected version 1 got 0
I20251027 21:08:04.771556 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_17e67d6acbdd470d9b351522b7908579" } modify_external_catalogs: true
I20251027 21:08:04.771766 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_17e67d6acbdd470d9b351522b7908579" } modify_external_catalogs: true
I20251027 21:08:04.772454 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7ff9f51c77a1418faf1e35ad182c91d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.772950 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ff9f51c77a1418faf1e35ad182c91d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.774327 10907 raft_consensus.cc:493] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.774432 10907 raft_consensus.cc:515] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.774947 10907 leader_election.cc:290] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:04.776264 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ea4ca66f1a144318a185c1324b2f3f61" } new_table_name: "default.Table_c63e136bf25642a696d1ad5db0b7a997" modify_external_catalogs: true
I20251027 21:08:04.779129 9312 catalog_manager.cc:5649] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:04.779582 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet ca477937278f45b4b115fec715f32a92 (table default.Table_69627cab65c2407b9226a05106b7bdf4 [id=fc5a0b5a96e6400da438307d26f73614]). Expected version 3 got 0
I20251027 21:08:04.781205 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d56d323ef0e74344a479940b93fea11a" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:04.781394 9769 raft_consensus.cc:2468] T d56d323ef0e74344a479940b93fea11a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:04.781682 9566 leader_election.cc:304] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:04.782370 10859 raft_consensus.cc:2804] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:04.782438 10859 raft_consensus.cc:493] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:04.782495 10859 raft_consensus.cc:3060] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:04.784124 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ccd419ff40a94f9587b024ec87a2517e consensus state has no leader
I20251027 21:08:04.785511 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 1d1d4031a14e448ea5a60327bd37d003
I20251027 21:08:04.786191 9306 master_service.cc:954] ReplaceTablet: received request to replace tablet 532ea47b15b544dfb05ddb2bdacf3953 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:04.786517 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d1d4031a14e448ea5a60327bd37d003 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.786849 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d1d4031a14e448ea5a60327bd37d003 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:04.788645 11065 raft_consensus.cc:493] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.788772 11065 raft_consensus.cc:515] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:04.789378 11065 leader_election.cc:290] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:04.790262 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9861a0efcba64141aae450663c5d83de" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:04.790263 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9861a0efcba64141aae450663c5d83de" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:04.790341 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c3d07b891a248efb0860cd000e44432 has no consensus state
W20251027 21:08:04.790796 9859 leader_election.cc:343] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:04.791265 9860 leader_election.cc:343] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.791378 9860 leader_election.cc:304] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:04.791726 11065 raft_consensus.cc:2749] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:04.792421 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet ee0c598904004dd18df058da483aa45c
I20251027 21:08:04.793849 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet ee0c598904004dd18df058da483aa45c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.793951 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet ee0c598904004dd18df058da483aa45c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:04.794659 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet ee0c598904004dd18df058da483aa45c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.798363 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 906) took 1308 ms. Trace:
I20251027 21:08:04.798442 10041 rpcz_store.cc:276] 1027 21:08:03.489975 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:03.521085 (+ 31110us) service_pool.cc:225] Handling call
1027 21:08:04.681131 (+1160046us) ts_tablet_manager.cc:1151] Deleting tablet d13d00353b6740b5bed5a153936d2059
1027 21:08:04.681146 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:04.712382 (+ 31236us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:04.719022 (+ 6640us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:04.798347 (+ 79325us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1159577}
I20251027 21:08:04.798753 10041 tablet_replica.cc:333] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:04.798931 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet ccf10ef559f74e2db28023fd520c8de9
I20251027 21:08:04.798954 10041 raft_consensus.cc:2243] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:04.799068 10041 raft_consensus.cc:2272] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:04.799467 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet a4d26f73b8534f08bcaa386d4de75c01 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:04.799516 10029 tablet_bootstrap.cc:492] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:04.799598 10041 ts_tablet_manager.cc:1916] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:04.799640 10029 ts_tablet_manager.cc:1403] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.000s
I20251027 21:08:04.800279 10029 raft_consensus.cc:359] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.800431 10029 raft_consensus.cc:385] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.800478 10029 raft_consensus.cc:740] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.800675 10029 consensus_queue.cc:260] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.801081 10029 ts_tablet_manager.cc:1434] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251027 21:08:04.801311 10029 tablet_bootstrap.cc:492] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:04.802278 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet ccf10ef559f74e2db28023fd520c8de9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.802924 10029 tablet_bootstrap.cc:654] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:04.804577 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e06ae559ca14b8bb0630e1571ca4852 consensus state has no leader
I20251027 21:08:04.808128 10043 ts_tablet_manager.cc:1929] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:04.809818 10043 log.cc:1199] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/e9edb62a8e4b4dfea6a31a2202545a6e
I20251027 21:08:04.810202 10043 ts_tablet_manager.cc:1950] T e9edb62a8e4b4dfea6a31a2202545a6e P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:04.810264 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet d13d00353b6740b5bed5a153936d2059 (table default.Table_c4824ada6d0f433dbb4770562a66e1ad [id=f2edcf33b0b74923b4ddbbc2419e14ca]) successfully deleted
I20251027 21:08:04.810592 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet ccf10ef559f74e2db28023fd520c8de9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:04.815227 11065 raft_consensus.cc:493] T d672118ecd6f4094855adf106e5f7115 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.815353 11065 raft_consensus.cc:515] T d672118ecd6f4094855adf106e5f7115 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:04.815946 11065 leader_election.cc:290] T d672118ecd6f4094855adf106e5f7115 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:04.816442 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d672118ecd6f4094855adf106e5f7115" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:04.817132 9859 leader_election.cc:343] T d672118ecd6f4094855adf106e5f7115 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.818372 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_baa19560a48f42c7ab0ecad353eaa283"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.818890 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_baa19560a48f42c7ab0ecad353eaa283 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.820374 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d672118ecd6f4094855adf106e5f7115" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:04.821169 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7cde4cdee7294ac8ab1c8dfcbb93abcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.821696 9860 leader_election.cc:343] T d672118ecd6f4094855adf106e5f7115 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:04.821756 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7cde4cdee7294ac8ab1c8dfcbb93abcf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.821802 9860 leader_election.cc:304] T d672118ecd6f4094855adf106e5f7115 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:04.822675 11065 raft_consensus.cc:2749] T d672118ecd6f4094855adf106e5f7115 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:04.823519 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d9decf014b434b80b9d5529bd7aafce3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.824151 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9decf014b434b80b9d5529bd7aafce3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.827301 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_1812ea722de14512bad256f85c0596ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:04.827702 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet ccf10ef559f74e2db28023fd520c8de9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:04.827940 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1812ea722de14512bad256f85c0596ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.828063 11057 raft_consensus.cc:515] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.829135 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4debc3f13f3a42939b6b83f6523aab7c" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:04.830045 9769 raft_consensus.cc:3060] T 4debc3f13f3a42939b6b83f6523aab7c P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:04.830260 11057 leader_election.cc:290] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:04.830339 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_bfc1511d230f4df297686bac2acbeac2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.830960 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bfc1511d230f4df297686bac2acbeac2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.832788 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e6dc3cc1aeef48a48107dc699cbb2238" } new_table_name: "default.Table_7be07f1db7b548f3b1fb67fd289088f5" modify_external_catalogs: true
I20251027 21:08:04.831705 10029 tablet_bootstrap.cc:492] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:04.833158 11057 consensus_queue.cc:1048] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:04.833170 10029 ts_tablet_manager.cc:1403] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.004s
I20251027 21:08:04.835011 11065 raft_consensus.cc:493] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.835143 11065 raft_consensus.cc:515] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:04.835799 10029 raft_consensus.cc:359] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.835872 11065 leader_election.cc:290] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:04.835945 10029 raft_consensus.cc:385] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.835978 9307 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 45568792d7954b498581ce69fb611948 (table default.Table_3e3e0c17a3bf452fbb217c6e3ee8bdcb [id=2eb093cd58e643a58b14e564d5914fbc]). Expected version 1 got 0
I20251027 21:08:04.836004 10029 raft_consensus.cc:740] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.836100 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fde0980aa64c4496861a662f7ed3ef75" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:04.836238 10029 consensus_queue.cc:260] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:04.836511 9859 leader_election.cc:343] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.836759 10029 ts_tablet_manager.cc:1434] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:04.837041 10029 tablet_bootstrap.cc:492] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:04.837249 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fde0980aa64c4496861a662f7ed3ef75" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:04.837785 9860 leader_election.cc:343] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:04.837879 9860 leader_election.cc:304] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:04.839125 10029 tablet_bootstrap.cc:654] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.839651 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet c3b9a461567a48fcb4af514ecdf618b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.839973 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a4aa1fe05114c1287ca580c1f411d3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.840176 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet da26c83976df40328cf9b11734fcfa44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.840695 11065 raft_consensus.cc:2749] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:04.841900 9568 leader_election.cc:343] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 4debc3f13f3a42939b6b83f6523aab7c
I20251027 21:08:04.843865 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 980eb6755aee4bcfb2ea44d84009021f (table default.Table_bb56319b6fe0405991a4d171477621e7 [id=42c8fedcabe0435dbaa11427141228ed]). Expected version 1 got 0
I20251027 21:08:04.848325 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b30052ff19744a9bf4bfaaa1bc77b36 with delete_type TABLET_DATA_DELETED (replaced by tablet 283b72f626644ac2b9b61186387b3839) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.848698 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d56d323ef0e74344a479940b93fea11a" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:04.848875 9925 raft_consensus.cc:2468] T d56d323ef0e74344a479940b93fea11a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:04.849895 9774 raft_consensus.cc:1240] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [term 2 FOLLOWER]: Rejecting Update request from peer 51902b6b61ff4d5584ff21896e847e14 for earlier term 1. Current term is 2. Ops: [1.1-1.2]
I20251027 21:08:04.850517 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d1d4031a14e448ea5a60327bd37d003 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.852005 11051 consensus_queue.cc:1059] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:04.852300 11018 raft_consensus.cc:3055] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Stepping down as leader of term 1
I20251027 21:08:04.852423 11018 raft_consensus.cc:740] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:04.852764 11018 consensus_queue.cc:260] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.853305 11018 raft_consensus.cc:3060] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Advancing to term 2
I20251027 21:08:04.848726 11070 tablet.cc:1722] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:04.855273 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_152a2bed589f47858b8a99ab7df9984c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.856570 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_152a2bed589f47858b8a99ab7df9984c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.857221 10029 tablet_bootstrap.cc:492] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:04.857333 10029 ts_tablet_manager.cc:1403] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.001s
I20251027 21:08:04.858114 10029 raft_consensus.cc:359] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:04.858283 10029 raft_consensus.cc:385] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.858337 10029 raft_consensus.cc:740] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.858533 10029 consensus_queue.cc:260] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:04.858960 10029 ts_tablet_manager.cc:1434] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 21:08:04.859176 10029 tablet_bootstrap.cc:492] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:04.860160 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 4debc3f13f3a42939b6b83f6523aab7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.860555 10029 tablet_bootstrap.cc:654] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.866698 11071 tablet.cc:1722] T abd3b8e71d3041b4b9e5f6663841bfcf P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:04.874043 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 532ea47b15b544dfb05ddb2bdacf3953
I20251027 21:08:04.874536 9306 catalog_manager.cc:6719] ReplaceTablet: tablet 532ea47b15b544dfb05ddb2bdacf3953 deleted and replaced by tablet 6c05b69509414fe8bfc809cd307d5f2e
I20251027 21:08:04.876336 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 532ea47b15b544dfb05ddb2bdacf3953 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c05b69509414fe8bfc809cd307d5f2e) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:04.876487 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet 532ea47b15b544dfb05ddb2bdacf3953 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c05b69509414fe8bfc809cd307d5f2e) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:04.876801 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 532ea47b15b544dfb05ddb2bdacf3953 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c05b69509414fe8bfc809cd307d5f2e) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:04.878265 10029 tablet_bootstrap.cc:492] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:04.878379 10029 ts_tablet_manager.cc:1403] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
I20251027 21:08:04.878545 10829 raft_consensus.cc:493] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:04.878712 10829 raft_consensus.cc:515] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.878981 10029 raft_consensus.cc:359] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.879150 10029 raft_consensus.cc:385] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.879205 10029 raft_consensus.cc:740] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.879446 10829 leader_election.cc:290] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:04.879426 10029 consensus_queue.cc:260] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.880062 10029 ts_tablet_manager.cc:1434] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:04.880373 10029 tablet_bootstrap.cc:492] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:04.880862 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a339c0dee0994788b8644912f74c1b82" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:04.881065 9635 raft_consensus.cc:2468] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
W20251027 21:08:04.881268 9711 leader_election.cc:343] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: a339c0dee0994788b8644912f74c1b82
I20251027 21:08:04.881916 9710 leader_election.cc:304] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:04.882252 10029 tablet_bootstrap.cc:654] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.882411 10829 raft_consensus.cc:2804] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:04.882531 10829 raft_consensus.cc:493] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:04.882618 10829 raft_consensus.cc:3060] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:04.885681 9305 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 345d3795a37b4a40b40967cfacf0f893 (table default.Table_ef79e4d09fb741429811f81e1efd90cb [id=93d6307da3a34f87a67018a06825cd15]). Expected version 2 got 0
I20251027 21:08:04.888299 10043 tablet_replica.cc:333] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:04.888473 10043 raft_consensus.cc:2243] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:04.888603 10043 raft_consensus.cc:2272] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:04.889156 10043 ts_tablet_manager.cc:1916] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:04.893175 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet e9edb62a8e4b4dfea6a31a2202545a6e (table default.Table_2d874641d88e434c8022d7026802972f [id=7b94fb706c8e429b911e1e19265a3883]) successfully deleted
W20251027 21:08:04.895787 9847 kernel_stack_watchdog.cc:198] Thread 10032 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 297ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fc050a6d980 (unknown) at ??:0
@ 0x93d797 __sanitizer::internal_sched_yield() at ??:?
@ 0x8a2e4d __interceptor_pthread_create at ??:?
@ 0x7fc04d456be1 kudu::Thread::StartThread() at ??:0
@ 0xa75107 kudu::Thread::Create() at ??:?
@ 0x7fc04d476a2b kudu::ThreadPool::CreateThread() at ??:0
@ 0x7fc04d47244d kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7fc04d47777f kudu::ThreadPool::Submit() at ??:0
@ 0x7fc058b9388d kudu::log::SegmentAllocator::AsyncAllocateSegmentUnlocked() at ??:0
@ 0x7fc058b9287f kudu::log::SegmentAllocator::AllocateSegmentAndRollOver() at ??:0
@ 0x7fc058b9243e kudu::log::SegmentAllocator::Init() at ??:0
@ 0x7fc058b9a4b5 kudu::log::Log::Init() at ??:0
@ 0x7fc058b99811 kudu::log::Log::Open() at ??:0
@ 0x7fc05a5e9581 kudu::tablet::TabletBootstrap::OpenNewLog() at ??:0
@ 0x7fc05a5e58ec kudu::tablet::TabletBootstrap::RunBootstrap() at ??:0
@ 0x7fc05a5e406f kudu::tablet::TabletBootstrap::Bootstrap() at ??:0
I20251027 21:08:04.902356 10029 tablet_bootstrap.cc:492] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:04.902453 10029 ts_tablet_manager.cc:1403] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.003s
I20251027 21:08:04.903105 10029 raft_consensus.cc:359] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.903251 10029 raft_consensus.cc:385] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.903294 10029 raft_consensus.cc:740] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.903473 10029 consensus_queue.cc:260] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.903955 10029 ts_tablet_manager.cc:1434] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 21:08:04.904172 10029 tablet_bootstrap.cc:492] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:04.905613 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 7062871e266c4603bab851a837ef9aec (DEFAULT_TABLE table=default.Table_7aab2b63c44043aa9aea339ec0b5763e [id=9defb05f895749e9a76b7653ad9ad789]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.906064 10029 tablet_bootstrap.cc:654] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.906708 9742 tablet_service.cc:1505] Processing CreateTablet for tablet a21a6a3725934266bbfdfc70b40de687 (DEFAULT_TABLE table=default.Table_7d0f803e9bb44ef6b7f03e11e0dae838 [id=d351ca009aca461a99411b479ce9651c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.915268 9774 raft_consensus.cc:3060] T 6600774e4c7b48479a2fdb02614ba505 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:04.922925 10029 tablet_bootstrap.cc:492] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:04.923017 10029 ts_tablet_manager.cc:1403] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20251027 21:08:04.923609 10029 raft_consensus.cc:359] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.923740 10029 raft_consensus.cc:385] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.923787 10029 raft_consensus.cc:740] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.923975 10029 consensus_queue.cc:260] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.924481 10029 ts_tablet_manager.cc:1434] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251027 21:08:04.924678 10029 tablet_bootstrap.cc:492] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:04.926012 10029 tablet_bootstrap.cc:654] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.928977 9615 tablet_service.cc:1505] Processing CreateTablet for tablet a21a6a3725934266bbfdfc70b40de687 (DEFAULT_TABLE table=default.Table_7d0f803e9bb44ef6b7f03e11e0dae838 [id=d351ca009aca461a99411b479ce9651c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.931082 9905 tablet_service.cc:1505] Processing CreateTablet for tablet a21a6a3725934266bbfdfc70b40de687 (DEFAULT_TABLE table=default.Table_7d0f803e9bb44ef6b7f03e11e0dae838 [id=d351ca009aca461a99411b479ce9651c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.931851 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 4e35961fed9c4a4b974140324fe6cc05 (DEFAULT_TABLE table=default.Table_6a6205df84c04bf8a9fb02bef162312c [id=4da1e0966a744542800f03c1055016eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:04.933624 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 4e35961fed9c4a4b974140324fe6cc05 (DEFAULT_TABLE table=default.Table_6a6205df84c04bf8a9fb02bef162312c [id=4da1e0966a744542800f03c1055016eb]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:04.944748 10032 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.109s sys 0.212s
W20251027 21:08:04.944829 10032 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.530s user 0.109s sys 0.212s
I20251027 21:08:04.946875 11074 tablet.cc:1722] T abd3b8e71d3041b4b9e5f6663841bfcf P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:04.947026 10032 tablet_bootstrap.cc:492] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:04.947125 10032 ts_tablet_manager.cc:1403] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.535s user 0.111s sys 0.212s
I20251027 21:08:04.947913 10032 raft_consensus.cc:359] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.948063 10032 raft_consensus.cc:385] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.948109 10032 raft_consensus.cc:740] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.948417 10032 consensus_queue.cc:260] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.949060 10032 ts_tablet_manager.cc:1434] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:04.949417 10032 tablet_bootstrap.cc:492] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:04.950902 10032 tablet_bootstrap.cc:654] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.956454 10029 tablet_bootstrap.cc:492] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:04.956569 10029 ts_tablet_manager.cc:1403] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.003s
I20251027 21:08:04.957222 10029 raft_consensus.cc:359] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.957376 10029 raft_consensus.cc:385] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.957422 10029 raft_consensus.cc:740] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.957705 10029 consensus_queue.cc:260] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.958181 10029 ts_tablet_manager.cc:1434] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:04.958412 10029 tablet_bootstrap.cc:492] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:04.959327 11075 consensus_queue.cc:1048] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:04.959784 10029 tablet_bootstrap.cc:654] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.966324 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f690a4140360442b92080c8d55fdc1e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.967051 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f690a4140360442b92080c8d55fdc1e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.967634 10029 tablet_bootstrap.cc:492] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:04.967758 10029 ts_tablet_manager.cc:1403] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20251027 21:08:04.968523 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3da5a86d9f104d1ba39726b5a6dcec8a" } modify_external_catalogs: true
I20251027 21:08:04.968554 10029 raft_consensus.cc:359] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.968716 10029 raft_consensus.cc:385] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:04.968699 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3da5a86d9f104d1ba39726b5a6dcec8a" } modify_external_catalogs: true
I20251027 21:08:04.968791 10029 raft_consensus.cc:740] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:04.969758 11086 tablet.cc:1722] T abd3b8e71d3041b4b9e5f6663841bfcf P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:04.971939 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e9839f0986064565be4d73e81a8671da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:04.972518 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9839f0986064565be4d73e81a8671da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:04.974517 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_50b757fa69504e57872e1ecaee13583c" } modify_external_catalogs: true
I20251027 21:08:04.974701 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_50b757fa69504e57872e1ecaee13583c" } modify_external_catalogs: true
I20251027 21:08:04.975287 10029 consensus_queue.cc:260] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:04.978075 10029 ts_tablet_manager.cc:1434] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251027 21:08:04.978399 10029 tablet_bootstrap.cc:492] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:04.981613 10907 consensus_queue.cc:1048] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:04.989979 10029 tablet_bootstrap.cc:654] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:04.996210 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 4e35961fed9c4a4b974140324fe6cc05 (DEFAULT_TABLE table=default.Table_6a6205df84c04bf8a9fb02bef162312c [id=4da1e0966a744542800f03c1055016eb]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:04.997840 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2741d50c7654407194488e14b55460ad consensus state has no leader
I20251027 21:08:04.998371 9307 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_ef79e4d09fb741429811f81e1efd90cb [id=93d6307da3a34f87a67018a06825cd15] alter complete (version 2)
I20251027 21:08:05.001046 11057 raft_consensus.cc:493] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:05.002097 11057 raft_consensus.cc:515] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.003206 11065 raft_consensus.cc:493] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:05.003345 11065 raft_consensus.cc:515] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.004655 11065 leader_election.cc:290] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:05.004828 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04c579bb008448e9baa841135b6dbc61" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:05.005035 11057 leader_election.cc:290] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:05.005220 9568 leader_election.cc:343] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 04c579bb008448e9baa841135b6dbc61
W20251027 21:08:05.005329 9566 leader_election.cc:343] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:05.005391 9568 leader_election.cc:304] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:05.006927 11057 raft_consensus.cc:2749] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:05.009523 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d18ff6d8ddb465fbf479fdc7ebe7a05" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:05.009864 9860 leader_election.cc:343] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:05.009227 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d18ff6d8ddb465fbf479fdc7ebe7a05" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:05.010360 9859 leader_election.cc:343] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:05.010447 9859 leader_election.cc:304] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:05.010759 11065 raft_consensus.cc:2749] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:05.011987 9739 tablet_service.cc:1505] Processing CreateTablet for tablet dc1cb2149b5f49e8a1f78d820a74cea7 (DEFAULT_TABLE table=default.Table_52c44a2a153c46ec823f80c2c4fbb600 [id=36cd43fd6b4a4eb0871b5cb16dd464fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.012606 10029 tablet_bootstrap.cc:492] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:05.012717 10029 ts_tablet_manager.cc:1403] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20251027 21:08:05.013350 10029 raft_consensus.cc:359] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.013468 10029 raft_consensus.cc:385] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:05.013504 10029 raft_consensus.cc:740] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:05.007743 11057 consensus_queue.cc:1048] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20251027 21:08:05.018424 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_190963122a7245dfa310ab8c69edf80f" } modify_external_catalogs: true
I20251027 21:08:05.018615 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_190963122a7245dfa310ab8c69edf80f" } modify_external_catalogs: true
I20251027 21:08:05.019876 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2c2ec3f9d23142c9b8c2d66d1c979525" } new_table_name: "default.Table_58c9d142451744d080d11bda5ff47a05" modify_external_catalogs: true
I20251027 21:08:05.021186 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_aaeb60868697426bbaa517f52ea223e6" } modify_external_catalogs: true
I20251027 21:08:05.021346 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_aaeb60868697426bbaa517f52ea223e6" } modify_external_catalogs: true
I20251027 21:08:05.022375 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_140424c161ea4581bed312cf5deb5ab5" } new_table_name: "default.Table_85f8ddead4f94d889f32e6c27e524f21" modify_external_catalogs: true
I20251027 21:08:05.026820 9304 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 94f2819dd0394530ae93754bdd1e482f (table default.Table_090fa7c40bdb40d09777fd8458acd6b2 [id=63147f61013c4d6dbc96fd25769b6136]). Expected version 2 got 0
I20251027 21:08:05.027030 9304 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet aaa8910d9fdb4146945621ec4ebf969e (table default.Table_d8da07f0725347338012d992737c5d04 [id=bb676562b97f48088767941239cfbf33]). Expected version 2 got 0
I20251027 21:08:05.027264 9304 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 9fde4fd91943498d8888d1783005e492 (table default.Table_87ac731a20bf4f6bbce8d82c1ad3bd97 [id=3a0c8b19df934d1580349b061fe0e8f5]). Expected version 1 got 0
I20251027 21:08:05.028730 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 59b6818d29624da5a49dd1a7d20994c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:05.029340 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet c9efffa9766e46aca8eb286fc2ad48d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:05.029695 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 04c579bb008448e9baa841135b6dbc61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:05.029688 10907 consensus_queue.cc:1048] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:05.031436 9304 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 87d5af4607f04f92abcc7292b4176cbd (table default.Table_c78ef9ebe86b4da08443e37c83d4e80d [id=cdb023218cda4e9ea04a226b37a220d9]). Expected version 2 got 0
I20251027 21:08:05.033027 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9b06fbc2554c4780a754ed9cfb964fa9" } modify_external_catalogs: true
I20251027 21:08:05.033178 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9b06fbc2554c4780a754ed9cfb964fa9" } modify_external_catalogs: true
I20251027 21:08:05.034606 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_602a5846b60c4860b33575da44b5015b" } new_table_name: "default.Table_ae51ca0900c0448a8852598394e0e912" modify_external_catalogs: true
I20251027 21:08:05.015117 10029 consensus_queue.cc:260] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.046561 10029 ts_tablet_manager.cc:1434] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20251027 21:08:05.046936 10029 tablet_bootstrap.cc:492] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:05.050021 10029 tablet_bootstrap.cc:654] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:05.057477 10029 tablet_bootstrap.cc:492] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:05.057576 10029 ts_tablet_manager.cc:1403] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20251027 21:08:05.058368 10029 raft_consensus.cc:359] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:05.058503 10029 raft_consensus.cc:385] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:05.058542 10029 raft_consensus.cc:740] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:05.058738 10029 consensus_queue.cc:260] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:05.059140 10029 ts_tablet_manager.cc:1434] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251027 21:08:05.059372 10029 tablet_bootstrap.cc:492] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:05.060321 9600 tablet_service.cc:1505] Processing CreateTablet for tablet dc1cb2149b5f49e8a1f78d820a74cea7 (DEFAULT_TABLE table=default.Table_52c44a2a153c46ec823f80c2c4fbb600 [id=36cd43fd6b4a4eb0871b5cb16dd464fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.060956 9890 tablet_service.cc:1505] Processing CreateTablet for tablet dc1cb2149b5f49e8a1f78d820a74cea7 (DEFAULT_TABLE table=default.Table_52c44a2a153c46ec823f80c2c4fbb600 [id=36cd43fd6b4a4eb0871b5cb16dd464fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.061422 10029 tablet_bootstrap.cc:654] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:05.061988 9745 tablet_service.cc:1505] Processing CreateTablet for tablet dccbd5e574564e85b7f5b6abd2e8857d (DEFAULT_TABLE table=default.Table_6f780e2be9194c1e9ad52558415f7043 [id=979d48352f5347cdac6c0589eac598b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.063292 9611 tablet_service.cc:1505] Processing CreateTablet for tablet dccbd5e574564e85b7f5b6abd2e8857d (DEFAULT_TABLE table=default.Table_6f780e2be9194c1e9ad52558415f7043 [id=979d48352f5347cdac6c0589eac598b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.064363 9888 tablet_service.cc:1505] Processing CreateTablet for tablet dccbd5e574564e85b7f5b6abd2e8857d (DEFAULT_TABLE table=default.Table_6f780e2be9194c1e9ad52558415f7043 [id=979d48352f5347cdac6c0589eac598b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.065549 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 3fef7b60497446f3af446069494ebc88 (DEFAULT_TABLE table=default.Table_32b31b9bf26542e69fb03fea9edd07bc [id=8608708934294043a468e6b1ea2f014e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.066634 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 3fef7b60497446f3af446069494ebc88 (DEFAULT_TABLE table=default.Table_32b31b9bf26542e69fb03fea9edd07bc [id=8608708934294043a468e6b1ea2f014e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.067854 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 3fef7b60497446f3af446069494ebc88 (DEFAULT_TABLE table=default.Table_32b31b9bf26542e69fb03fea9edd07bc [id=8608708934294043a468e6b1ea2f014e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.069048 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 9a650cc13a9e4cdbbcb145fa75c15937 (DEFAULT_TABLE table=default.Table_01b9845ac6a54380a39c0a76e0db231e [id=7a8fd1dcb5874722a89cc33789f18ba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.070264 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 9a650cc13a9e4cdbbcb145fa75c15937 (DEFAULT_TABLE table=default.Table_01b9845ac6a54380a39c0a76e0db231e [id=7a8fd1dcb5874722a89cc33789f18ba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.072283 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e35961fed9c4a4b974140324fe6cc05. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.072779 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1cb2149b5f49e8a1f78d820a74cea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.073257 10859 raft_consensus.cc:515] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.073740 10859 leader_election.cc:290] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:05.074301 9769 raft_consensus.cc:2468] T 4debc3f13f3a42939b6b83f6523aab7c P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:05.074687 9566 leader_election.cc:304] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:05.075116 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fef7b60497446f3af446069494ebc88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.075810 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a650cc13a9e4cdbbcb145fa75c15937. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.076222 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dccbd5e574564e85b7f5b6abd2e8857d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.076532 10859 raft_consensus.cc:2804] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:05.076645 10859 raft_consensus.cc:697] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:05.076849 10859 consensus_queue.cc:237] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.080775 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a21a6a3725934266bbfdfc70b40de687. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.080897 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a21a6a3725934266bbfdfc70b40de687. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.081235 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e35961fed9c4a4b974140324fe6cc05. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.083909 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1cb2149b5f49e8a1f78d820a74cea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.084271 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dccbd5e574564e85b7f5b6abd2e8857d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.084830 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fef7b60497446f3af446069494ebc88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.085996 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d56d323ef0e74344a479940b93fea11a" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:05.086160 9925 raft_consensus.cc:3060] T d56d323ef0e74344a479940b93fea11a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:05.094864 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7062871e266c4603bab851a837ef9aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.096588 11096 raft_consensus.cc:493] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:05.096712 11096 raft_consensus.cc:515] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:05.097762 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9efffa9766e46aca8eb286fc2ad48d2" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:05.098308 9566 leader_election.cc:343] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:05.098529 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a21a6a3725934266bbfdfc70b40de687. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.098989 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e35961fed9c4a4b974140324fe6cc05. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.099325 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dccbd5e574564e85b7f5b6abd2e8857d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.100600 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1cb2149b5f49e8a1f78d820a74cea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.101276 11096 leader_election.cc:290] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:05.103781 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a650cc13a9e4cdbbcb145fa75c15937. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.104259 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fef7b60497446f3af446069494ebc88. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:05.109598 9568 leader_election.cc:343] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: c9efffa9766e46aca8eb286fc2ad48d2
I20251027 21:08:05.109755 9568 leader_election.cc:304] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:05.115240 11096 raft_consensus.cc:2749] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:05.118685 11070 tablet.cc:1722] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251027 21:08:05.120591 11071 tablet.cc:1722] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251027 21:08:05.130559 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d56d323ef0e74344a479940b93fea11a" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:05.130764 9772 raft_consensus.cc:3060] T d56d323ef0e74344a479940b93fea11a P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:05.151237 11040 log.cc:826] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: Log is configured to *not* fsync() on all Append() calls
I20251027 21:08:05.151731 10829 raft_consensus.cc:515] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.152343 10829 leader_election.cc:290] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
W20251027 21:08:05.153275 9711 leader_election.cc:343] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: a339c0dee0994788b8644912f74c1b82
I20251027 21:08:05.155647 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a339c0dee0994788b8644912f74c1b82" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
W20251027 21:08:05.156145 9710 leader_election.cc:343] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: Tablet not RUNNING: STOPPED
I20251027 21:08:05.156275 9710 leader_election.cc:304] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:05.156689 10829 raft_consensus.cc:2749] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251027 21:08:05.161291 9925 raft_consensus.cc:2468] T d56d323ef0e74344a479940b93fea11a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:05.163237 9568 leader_election.cc:304] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:05.163487 11018 raft_consensus.cc:2804] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:05.163584 11018 raft_consensus.cc:697] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:05.163792 11018 consensus_queue.cc:237] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.180356 10031 tablet_bootstrap.cc:492] T ca477937278f45b4b115fec715f32a92 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:05.180524 10031 ts_tablet_manager.cc:1403] T ca477937278f45b4b115fec715f32a92 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.590s user 0.005s sys 0.000s
I20251027 21:08:05.181274 10031 raft_consensus.cc:359] T ca477937278f45b4b115fec715f32a92 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.181447 10031 raft_consensus.cc:385] T ca477937278f45b4b115fec715f32a92 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:05.181504 10031 raft_consensus.cc:740] T ca477937278f45b4b115fec715f32a92 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:05.190884 10031 consensus_queue.cc:260] T ca477937278f45b4b115fec715f32a92 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.194186 11086 tablet.cc:1722] T b8856c5dc8f144358445165324bfb477 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251027 21:08:05.198915 10031 ts_tablet_manager.cc:1434] T ca477937278f45b4b115fec715f32a92 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20251027 21:08:05.200105 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet d94c695320f24a7f9776c493e462ed24
I20251027 21:08:05.203071 10997 raft_consensus.cc:493] T 6600774e4c7b48479a2fdb02614ba505 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:05.203174 10997 raft_consensus.cc:515] T 6600774e4c7b48479a2fdb02614ba505 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.203833 10997 leader_election.cc:290] T 6600774e4c7b48479a2fdb02614ba505 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:05.204787 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6600774e4c7b48479a2fdb02614ba505" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:05.206768 9310 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 87d5af4607f04f92abcc7292b4176cbd (table default.Table_c78ef9ebe86b4da08443e37c83d4e80d [id=cdb023218cda4e9ea04a226b37a220d9]). Expected version 2 got 0
I20251027 21:08:05.211562 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_645e7862b8ef4097aa7152d5984b7fb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:05.212108 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_645e7862b8ef4097aa7152d5984b7fb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:05.214440 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_98ca10ad3609477aaf22dc563a68742e" } new_table_name: "default.Table_e35dbbb1f48a4d2998d79cff38d71088" modify_external_catalogs: true
I20251027 21:08:05.215978 10041 ts_tablet_manager.cc:1929] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:05.216086 10041 log.cc:1199] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/01e9db9eaf8e41c6a2ee13b9ed127421
I20251027 21:08:05.216361 10041 ts_tablet_manager.cc:1950] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:05.220764 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet bbb4116d1edd492893adfbf5fb3ddeef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:05.221093 9888 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:05.224560 11097 consensus_queue.cc:1048] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:05.229532 11051 raft_consensus.cc:493] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:05.461984 11051 raft_consensus.cc:515] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.234668 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet d94c695320f24a7f9776c493e462ed24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:05.462559 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 6919e57d8dae48a890c041109ed253c7 (DEFAULT_TABLE table=default.Table_7b39d3d222b7499fa27b4230bc87dc5f [id=0144f2f8abbd4ce6b444b78024fb786e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.462805 11051 leader_election.cc:290] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:05.462977 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6919e57d8dae48a890c041109ed253c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.234836 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet d94c695320f24a7f9776c493e462ed24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:05.464258 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 17a19ce92da8448bba471f94e3ee4647 (DEFAULT_TABLE table=default.Table_e9839f0986064565be4d73e81a8671da [id=06369c47e7ac421dbf721c768d0c208e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.464597 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17a19ce92da8448bba471f94e3ee4647. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.224814 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet d94c695320f24a7f9776c493e462ed24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:05.233942 10031 tablet_bootstrap.cc:492] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:05.466481 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 6919e57d8dae48a890c041109ed253c7 (DEFAULT_TABLE table=default.Table_7b39d3d222b7499fa27b4230bc87dc5f [id=0144f2f8abbd4ce6b444b78024fb786e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.466845 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6919e57d8dae48a890c041109ed253c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.467911 10031 tablet_bootstrap.cc:654] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:05.237752 10997 raft_consensus.cc:493] T ca477937278f45b4b115fec715f32a92 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:05.468832 10997 raft_consensus.cc:515] T ca477937278f45b4b115fec715f32a92 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.469512 10997 leader_election.cc:290] T ca477937278f45b4b115fec715f32a92 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:05.245357 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 01049640e486408e962fed8c6ea557f5 (DEFAULT_TABLE table=default.Table_c6b87d4cea6a4f8eb5226bbdd13f1434 [id=690432cfd1bb488184b2f0e1b1b9b870]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.259184 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 91d29318b91341e2a7c00a0841c6def6 (DEFAULT_TABLE table=default.Table_233d6aa47dba4a72beb99b61ba19983b [id=48683266249b45f99e83117329cdbc2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.287055 9744 tablet_service.cc:1505] Processing CreateTablet for tablet b7c9623ff35e4ae9a2ead45bef605e49 (DEFAULT_TABLE table=default.Table_dbd96c7982f647c6990d16cbec0a6a04 [id=1b79b89efe404e9bbf247d643b4b3453]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.305321 9772 raft_consensus.cc:2468] T d56d323ef0e74344a479940b93fea11a P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:05.469933 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01049640e486408e962fed8c6ea557f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.223738 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 83f5fbf5757a49df93ab949b7657c61f
W20251027 21:08:05.305655 9748 consensus_meta.cc:287] T 7062871e266c4603bab851a837ef9aec P 77b0513f17c0472088901a1ce8b08218: Time spent flushing consensus metadata: real 0.657s user 0.003s sys 0.000s
I20251027 21:08:05.472787 9306 catalog_manager.cc:5649] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:05.306990 10040 ts_tablet_manager.cc:1929] T 53cac7817db440e79e7416d625a9b2c8 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:05.473192 9306 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet d56d323ef0e74344a479940b93fea11a (table default.Table_cf11a8c344c544c5ba54672fee38e4aa [id=71e02b21c9654e45a41d3e450ac3fbe1]). Expected version 2 got 0
I20251027 21:08:05.473304 10040 log.cc:1199] T 53cac7817db440e79e7416d625a9b2c8 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/53cac7817db440e79e7416d625a9b2c8
I20251027 21:08:05.473546 9306 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 87d5af4607f04f92abcc7292b4176cbd (table default.Table_c78ef9ebe86b4da08443e37c83d4e80d [id=cdb023218cda4e9ea04a226b37a220d9]). Expected version 2 got 0
I20251027 21:08:05.473592 10040 ts_tablet_manager.cc:1950] T 53cac7817db440e79e7416d625a9b2c8 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:05.318336 9739 tablet_service.cc:1505] Processing CreateTablet for tablet cb5ac0fff70b491e8c8842f7a331005e (DEFAULT_TABLE table=default.Table_c305a5155cce4754bcbc792f3900b690 [id=82b0b6e6d2e145a7870a13ed5b407b47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.324214 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 4540d5fbb5bc4611bc0104bf840bd12d (DEFAULT_TABLE table=default.Table_b29f906d3a774523839fd36ba1f5d2e4 [id=6e0cb64609ac43a7b474c2306e5fefe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.326126 9755 tablet_service.cc:1505] Processing CreateTablet for tablet dcf5c7d38e6143a8869227f85ae37270 (DEFAULT_TABLE table=default.Table_e186a89da49149d3921fe0646ac3f5c2 [id=a082a674803a4b0eacd140fcbce0d4bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.331867 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 511fdda39953415aaa65d88cb75ad382 (DEFAULT_TABLE table=default.Table_7453f892d0514227b015e34cf2461acd [id=f79bde1a71234b53b5a960c983481cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.334894 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 8fddac33cd6247ee81a0117c8a2e5e8b (DEFAULT_TABLE table=default.Table_4e44eb5928a2489baf1bf9d68ea91d18 [id=e0d6b59ba51f4b4eae20b20d4c76731d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.338160 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 4fe24988809f4300934efe90ec326559 (DEFAULT_TABLE table=default.Table_54ec9385aca7411ba2d3d140802ee661 [id=a820c8074c6a4de192f2782cc079c74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.347407 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 45d67d0ef7294b3ea63529db2cdaafaf (DEFAULT_TABLE table=default.Table_4cbc59c030d044d8a097385ed57a0b5f [id=83566d870a90461686826b2073b99f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.247642 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 01049640e486408e962fed8c6ea557f5 (DEFAULT_TABLE table=default.Table_c6b87d4cea6a4f8eb5226bbdd13f1434 [id=690432cfd1bb488184b2f0e1b1b9b870]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.350205 9738 tablet_service.cc:1505] Processing CreateTablet for tablet c5fd809f50e2494bb6d815b548f18481 (DEFAULT_TABLE table=default.Table_563019081d2f4beebbe8dce71dca1b65 [id=a804408c2221413590a3ca8b69067c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.474411 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01049640e486408e962fed8c6ea557f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.224699 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 8649a3b611cc48ff8832c0fdeb128efe
I20251027 21:08:05.476052 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca477937278f45b4b115fec715f32a92" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:05.251619 11057 raft_consensus.cc:493] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:05.261416 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 91d29318b91341e2a7c00a0841c6def6 (DEFAULT_TABLE table=default.Table_233d6aa47dba4a72beb99b61ba19983b [id=48683266249b45f99e83117329cdbc2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.273139 10029 tablet_bootstrap.cc:492] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:05.477121 9923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca477937278f45b4b115fec715f32a92" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:05.477324 9923 raft_consensus.cc:2393] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77b0513f17c0472088901a1ce8b08218 in current term 1: Already voted for candidate 51902b6b61ff4d5584ff21896e847e14 in this term.
I20251027 21:08:05.477319 10029 ts_tablet_manager.cc:1403] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.418s user 0.020s sys 0.053s
I20251027 21:08:05.477254 11057 raft_consensus.cc:515] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.477424 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91d29318b91341e2a7c00a0841c6def6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.477890 11057 leader_election.cc:290] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:05.356979 9751 tablet_service.cc:1505] Processing CreateTablet for tablet 5e372257733148199d2dbe70d0887cbb (DEFAULT_TABLE table=default.Table_02a5b78b84f1409b80d09d02e5069b9c [id=b3524d24371a43038db124bb4f0c5793]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.359094 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 5de5213c0a78413782c0e9d1bc6c3849 (DEFAULT_TABLE table=default.Table_368edee03055430fafa818f57694096b [id=c7f283a2ceca482d903dd69802590b2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.361388 9741 tablet_service.cc:1505] Processing CreateTablet for tablet 3604203c122a4df291ecdc1d8477dca0 (DEFAULT_TABLE table=default.Table_279dfb819b974d2085826df64b14d411 [id=ba4c471be2664b04b733d23b0481a34e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.363547 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 47d809c7d0c9412e81fa9cc497063e23 (DEFAULT_TABLE table=default.Table_92fa3ef69db74af5964d6b6f170b03c5 [id=77b56a563c144efe852d04ff779280fb]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:05.225790 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f901626832f4adb948ccb65f8573e86 consensus state has no leader
I20251027 21:08:05.235451 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6600774e4c7b48479a2fdb02614ba505" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:05.241488 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 9a650cc13a9e4cdbbcb145fa75c15937 (DEFAULT_TABLE table=default.Table_01b9845ac6a54380a39c0a76e0db231e [id=7a8fd1dcb5874722a89cc33789f18ba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.478901 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f7947222dc5043359f1891191346329b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:05.479039 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a650cc13a9e4cdbbcb145fa75c15937. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:05.479370 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7947222dc5043359f1891191346329b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:05.288583 9615 tablet_service.cc:1505] Processing CreateTablet for tablet b7c9623ff35e4ae9a2ead45bef605e49 (DEFAULT_TABLE table=default.Table_dbd96c7982f647c6990d16cbec0a6a04 [id=1b79b89efe404e9bbf247d643b4b3453]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.480624 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7c9623ff35e4ae9a2ead45bef605e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.247709 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 01049640e486408e962fed8c6ea557f5 (DEFAULT_TABLE table=default.Table_c6b87d4cea6a4f8eb5226bbdd13f1434 [id=690432cfd1bb488184b2f0e1b1b9b870]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.482403 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01049640e486408e962fed8c6ea557f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.342032 11119 tablet.cc:1722] T 995284883d834bbb9ef583b736ca8402 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251027 21:08:05.271960 10032 tablet_bootstrap.cc:492] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:05.483423 10032 ts_tablet_manager.cc:1403] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.534s user 0.003s sys 0.027s
I20251027 21:08:05.226714 9307 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet ecc7753ea44a45c39f2f326b918f5c99
I20251027 21:08:05.484206 10032 raft_consensus.cc:359] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.484364 10032 raft_consensus.cc:385] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:05.484277 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_557acd3dc3344fff87c5b92798fcfa66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:05.484408 10032 raft_consensus.cc:740] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:05.484575 10032 consensus_queue.cc:260] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:05.484762 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_557acd3dc3344fff87c5b92798fcfa66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:05.485059 10032 ts_tablet_manager.cc:1434] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:05.485411 10032 tablet_bootstrap.cc:492] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:05.301740 11070 tablet.cc:1722] T 995284883d834bbb9ef583b736ca8402 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251027 21:08:05.283578 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 91d29318b91341e2a7c00a0841c6def6 (DEFAULT_TABLE table=default.Table_233d6aa47dba4a72beb99b61ba19983b [id=48683266249b45f99e83117329cdbc2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.310495 9611 tablet_service.cc:1505] Processing CreateTablet for tablet cb5ac0fff70b491e8c8842f7a331005e (DEFAULT_TABLE table=default.Table_c305a5155cce4754bcbc792f3900b690 [id=82b0b6e6d2e145a7870a13ed5b407b47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.230306 9311 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet ca477937278f45b4b115fec715f32a92 (table default.Table_69627cab65c2407b9226a05106b7bdf4 [id=fc5a0b5a96e6400da438307d26f73614]). Expected version 3 got 0
I20251027 21:08:05.486498 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb5ac0fff70b491e8c8842f7a331005e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.230647 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f433bb2ad7be45fc82c323420fe9808f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:05.487721 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f433bb2ad7be45fc82c323420fe9808f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:05.235257 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 6f1d1bd52556463fa306e51eb253af40
I20251027 21:08:05.489230 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3b8f6db604d34f05b8dde4b29bd4cd3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:05.489475 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 0167d1954bfb47deb89c890f1de063f3 (DEFAULT_TABLE table=default.Table_f690a4140360442b92080c8d55fdc1e2 [id=73a236fad2434ef6ae33417069474719]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:05.489696 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b8f6db604d34f05b8dde4b29bd4cd3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:05.489838 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0167d1954bfb47deb89c890f1de063f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.283679 9970 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:08:05.316387 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 4540d5fbb5bc4611bc0104bf840bd12d (DEFAULT_TABLE table=default.Table_b29f906d3a774523839fd36ba1f5d2e4 [id=6e0cb64609ac43a7b474c2306e5fefe5]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:05.250866 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 54c2b10e79e149c2a637475e1f518907 consensus state has no leader
I20251027 21:08:05.490768 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4540d5fbb5bc4611bc0104bf840bd12d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.490942 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3b772db224e14e8993234e232491c8c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:05.491360 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b772db224e14e8993234e232491c8c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:05.478104 10029 raft_consensus.cc:359] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:08:05.255278 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 953b137f14a44955ae41ab503d04b43c consensus state has no leader
I20251027 21:08:05.492210 10029 raft_consensus.cc:385] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:05.492327 10029 raft_consensus.cc:740] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:05.492520 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_46c804a5818a4a4ca815f2449ccbe842" } new_table_name: "default.Table_8dfcee06548d49648483336e11f746f9" modify_external_catalogs: true
I20251027 21:08:05.492561 10029 consensus_queue.cc:260] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:05.493103 10029 ts_tablet_manager.cc:1434] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20251027 21:08:05.493379 10029 tablet_bootstrap.cc:492] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:05.256067 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2aad78a009654cb88858d59b51ee78cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:05.493755 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2aad78a009654cb88858d59b51ee78cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:05.292198 9296 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 54c2b10e79e149c2a637475e1f518907 (table default.Table_ae51ca0900c0448a8852598394e0e912 [id=9f1e545ab3174e8a893b9183cf95768d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 54c2b10e79e149c2a637475e1f518907 consensus state has no leader
I20251027 21:08:05.295974 11107 tablet.cc:1722] T 995284883d834bbb9ef583b736ca8402 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251027 21:08:05.326512 9601 tablet_service.cc:1505] Processing CreateTablet for tablet dcf5c7d38e6143a8869227f85ae37270 (DEFAULT_TABLE table=default.Table_e186a89da49149d3921fe0646ac3f5c2 [id=a082a674803a4b0eacd140fcbce0d4bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.494803 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcf5c7d38e6143a8869227f85ae37270. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.330354 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 511fdda39953415aaa65d88cb75ad382 (DEFAULT_TABLE table=default.Table_7453f892d0514227b015e34cf2461acd [id=f79bde1a71234b53b5a960c983481cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.496397 10029 tablet_bootstrap.cc:654] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:05.496603 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 511fdda39953415aaa65d88cb75ad382. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.300526 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 912) took 1753 ms. Trace:
I20251027 21:08:05.498246 10041 rpcz_store.cc:276] 1027 21:08:03.547175 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:03.557279 (+ 10104us) service_pool.cc:225] Handling call
1027 21:08:04.798696 (+1241417us) ts_tablet_manager.cc:1151] Deleting tablet 01e9db9eaf8e41c6a2ee13b9ed127421
1027 21:08:04.798705 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:05.077793 (+279088us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:05.215867 (+138074us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:05.300506 (+ 84639us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1241103}
I20251027 21:08:05.368983 9743 tablet_service.cc:1505] Processing CreateTablet for tablet f838fb6e148c4b16b062e831c57cd454 (DEFAULT_TABLE table=default.Table_fdf2451e0a2a48f9b902827e2a291e90 [id=6eae55ab35bf40bca718266926b1187f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.370383 9747 tablet_service.cc:1505] Processing CreateTablet for tablet 32c32d85389f4255ae2041e4481a3765 (DEFAULT_TABLE table=default.Table_b00669c899a945d283e7e68bcbc4384b [id=a48366ec5c504099813e34a8c40dd7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.375178 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 348898b63dca46f4a5c9a7e34b545d86 (DEFAULT_TABLE table=default.Table_35f4f0dcffb34a5e9b565ebd86d78445 [id=18775ec32f87470b8120d071386e8b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.463943 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da26c83976df40328cf9b11734fcfa44" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:05.335497 9595 tablet_service.cc:1505] Processing CreateTablet for tablet 8fddac33cd6247ee81a0117c8a2e5e8b (DEFAULT_TABLE table=default.Table_4e44eb5928a2489baf1bf9d68ea91d18 [id=e0d6b59ba51f4b4eae20b20d4c76731d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.504602 9595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fddac33cd6247ee81a0117c8a2e5e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.506026 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fde4fd91943498d8888d1783005e492" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:05.506500 9566 leader_election.cc:343] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:05.506867 9566 leader_election.cc:343] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:05.507128 9711 leader_election.cc:304] T ca477937278f45b4b115fec715f32a92 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:05.507522 9711 leader_election.cc:304] T 6600774e4c7b48479a2fdb02614ba505 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:05.507773 10829 raft_consensus.cc:3060] T ca477937278f45b4b115fec715f32a92 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:05.300808 9898 tablet_service.cc:1505] Processing CreateTablet for tablet b7c9623ff35e4ae9a2ead45bef605e49 (DEFAULT_TABLE table=default.Table_dbd96c7982f647c6990d16cbec0a6a04 [id=1b79b89efe404e9bbf247d643b4b3453]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.338325 9608 tablet_service.cc:1505] Processing CreateTablet for tablet 4fe24988809f4300934efe90ec326559 (DEFAULT_TABLE table=default.Table_54ec9385aca7411ba2d3d140802ee661 [id=a820c8074c6a4de192f2782cc079c74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.510303 10032 tablet_bootstrap.cc:654] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:05.510555 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe24988809f4300934efe90ec326559. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.516206 9600 tablet_service.cc:1505] Processing CreateTablet for tablet b1c06652f7bd435b823e07198c713a10 (DEFAULT_TABLE table=default.Table_d9decf014b434b80b9d5529bd7aafce3 [id=7a385fd969d74bee9f340cd2afb4c575]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.311313 11065 raft_consensus.cc:493] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:05.518571 11065 raft_consensus.cc:515] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:05.519217 11065 leader_election.cc:290] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:05.312500 9905 tablet_service.cc:1505] Processing CreateTablet for tablet cb5ac0fff70b491e8c8842f7a331005e (DEFAULT_TABLE table=default.Table_c305a5155cce4754bcbc792f3900b690 [id=82b0b6e6d2e145a7870a13ed5b407b47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.323344 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 4540d5fbb5bc4611bc0104bf840bd12d (DEFAULT_TABLE table=default.Table_b29f906d3a774523839fd36ba1f5d2e4 [id=6e0cb64609ac43a7b474c2306e5fefe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.328200 9899 tablet_service.cc:1505] Processing CreateTablet for tablet dcf5c7d38e6143a8869227f85ae37270 (DEFAULT_TABLE table=default.Table_e186a89da49149d3921fe0646ac3f5c2 [id=a082a674803a4b0eacd140fcbce0d4bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.332082 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 511fdda39953415aaa65d88cb75ad382 (DEFAULT_TABLE table=default.Table_7453f892d0514227b015e34cf2461acd [id=f79bde1a71234b53b5a960c983481cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.519542 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d64a13667194c12864fa388eb88a380" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:05.336321 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 8fddac33cd6247ee81a0117c8a2e5e8b (DEFAULT_TABLE table=default.Table_4e44eb5928a2489baf1bf9d68ea91d18 [id=e0d6b59ba51f4b4eae20b20d4c76731d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.341254 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 4fe24988809f4300934efe90ec326559 (DEFAULT_TABLE table=default.Table_54ec9385aca7411ba2d3d140802ee661 [id=a820c8074c6a4de192f2782cc079c74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.343619 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 45d67d0ef7294b3ea63529db2cdaafaf (DEFAULT_TABLE table=default.Table_4cbc59c030d044d8a097385ed57a0b5f [id=83566d870a90461686826b2073b99f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.352625 9886 tablet_service.cc:1505] Processing CreateTablet for tablet c5fd809f50e2494bb6d815b548f18481 (DEFAULT_TABLE table=default.Table_563019081d2f4beebbe8dce71dca1b65 [id=a804408c2221413590a3ca8b69067c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.357522 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 5e372257733148199d2dbe70d0887cbb (DEFAULT_TABLE table=default.Table_02a5b78b84f1409b80d09d02e5069b9c [id=b3524d24371a43038db124bb4f0c5793]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.519910 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7c9623ff35e4ae9a2ead45bef605e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.520124 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 17a19ce92da8448bba471f94e3ee4647 (DEFAULT_TABLE table=default.Table_e9839f0986064565be4d73e81a8671da [id=06369c47e7ac421dbf721c768d0c208e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.520414 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17a19ce92da8448bba471f94e3ee4647. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.360769 9901 tablet_service.cc:1505] Processing CreateTablet for tablet 5de5213c0a78413782c0e9d1bc6c3849 (DEFAULT_TABLE table=default.Table_368edee03055430fafa818f57694096b [id=c7f283a2ceca482d903dd69802590b2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.362955 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 3604203c122a4df291ecdc1d8477dca0 (DEFAULT_TABLE table=default.Table_279dfb819b974d2085826df64b14d411 [id=ba4c471be2664b04b733d23b0481a34e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.367369 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 47d809c7d0c9412e81fa9cc497063e23 (DEFAULT_TABLE table=default.Table_92fa3ef69db74af5964d6b6f170b03c5 [id=77b56a563c144efe852d04ff779280fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.369932 9903 tablet_service.cc:1505] Processing CreateTablet for tablet f838fb6e148c4b16b062e831c57cd454 (DEFAULT_TABLE table=default.Table_fdf2451e0a2a48f9b902827e2a291e90 [id=6eae55ab35bf40bca718266926b1187f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.371613 9894 tablet_service.cc:1505] Processing CreateTablet for tablet 32c32d85389f4255ae2041e4481a3765 (DEFAULT_TABLE table=default.Table_b00669c899a945d283e7e68bcbc4384b [id=a48366ec5c504099813e34a8c40dd7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.376400 9900 tablet_service.cc:1505] Processing CreateTablet for tablet 348898b63dca46f4a5c9a7e34b545d86 (DEFAULT_TABLE table=default.Table_35f4f0dcffb34a5e9b565ebd86d78445 [id=18775ec32f87470b8120d071386e8b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.522554 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 918) took 1227 ms. Trace:
I20251027 21:08:05.522630 10040 rpcz_store.cc:276] 1027 21:08:04.295327 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.295633 (+ 306us) service_pool.cc:225] Handling call
1027 21:08:04.578593 (+282960us) ts_tablet_manager.cc:1151] Deleting tablet 53cac7817db440e79e7416d625a9b2c8
1027 21:08:04.578602 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:05.099666 (+521064us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:05.306876 (+207210us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:05.522535 (+215659us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":282814}
I20251027 21:08:05.522939 10040 tablet_replica.cc:333] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:05.523104 10040 raft_consensus.cc:2243] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:05.523236 10040 raft_consensus.cc:2272] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:05.523705 10040 ts_tablet_manager.cc:1916] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:05.351962 9594 tablet_service.cc:1505] Processing CreateTablet for tablet c5fd809f50e2494bb6d815b548f18481 (DEFAULT_TABLE table=default.Table_563019081d2f4beebbe8dce71dca1b65 [id=a804408c2221413590a3ca8b69067c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.353318 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 5e372257733148199d2dbe70d0887cbb (DEFAULT_TABLE table=default.Table_02a5b78b84f1409b80d09d02e5069b9c [id=b3524d24371a43038db124bb4f0c5793]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.359773 9613 tablet_service.cc:1505] Processing CreateTablet for tablet 5de5213c0a78413782c0e9d1bc6c3849 (DEFAULT_TABLE table=default.Table_368edee03055430fafa818f57694096b [id=c7f283a2ceca482d903dd69802590b2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.361987 9598 tablet_service.cc:1505] Processing CreateTablet for tablet 3604203c122a4df291ecdc1d8477dca0 (DEFAULT_TABLE table=default.Table_279dfb819b974d2085826df64b14d411 [id=ba4c471be2664b04b733d23b0481a34e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.366691 9612 tablet_service.cc:1505] Processing CreateTablet for tablet 47d809c7d0c9412e81fa9cc497063e23 (DEFAULT_TABLE table=default.Table_92fa3ef69db74af5964d6b6f170b03c5 [id=77b56a563c144efe852d04ff779280fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.368032 9604 tablet_service.cc:1505] Processing CreateTablet for tablet f838fb6e148c4b16b062e831c57cd454 (DEFAULT_TABLE table=default.Table_fdf2451e0a2a48f9b902827e2a291e90 [id=6eae55ab35bf40bca718266926b1187f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.524295 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5fd809f50e2494bb6d815b548f18481. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.370960 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 32c32d85389f4255ae2041e4481a3765 (DEFAULT_TABLE table=default.Table_b00669c899a945d283e7e68bcbc4384b [id=a48366ec5c504099813e34a8c40dd7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.375833 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 348898b63dca46f4a5c9a7e34b545d86 (DEFAULT_TABLE table=default.Table_35f4f0dcffb34a5e9b565ebd86d78445 [id=18775ec32f87470b8120d071386e8b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.378785 9603 tablet_service.cc:1505] Processing CreateTablet for tablet 6919e57d8dae48a890c041109ed253c7 (DEFAULT_TABLE table=default.Table_7b39d3d222b7499fa27b4230bc87dc5f [id=0144f2f8abbd4ce6b444b78024fb786e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.397420 10043 ts_tablet_manager.cc:1929] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:05.526448 10043 log.cc:1199] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/a339c0dee0994788b8644912f74c1b82
I20251027 21:08:05.526657 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb5ac0fff70b491e8c8842f7a331005e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.526829 10043 ts_tablet_manager.cc:1950] T a339c0dee0994788b8644912f74c1b82 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:05.232481 11075 raft_consensus.cc:493] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:05.527148 11075 raft_consensus.cc:515] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.527802 11075 leader_election.cc:290] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:05.528326 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de5213c0a78413782c0e9d1bc6c3849. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.529975 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4540d5fbb5bc4611bc0104bf840bd12d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.533308 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcf5c7d38e6143a8869227f85ae37270. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.536439 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 511fdda39953415aaa65d88cb75ad382. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.537689 9613 tablet_service.cc:1505] Processing CreateTablet for tablet 0ab8cd9b46d446b3922909c715964ee8 (DEFAULT_TABLE table=default.Table_0393cf4c52244b19b2f59cc4a13f8dee [id=f151f5778c0a4bee8b442885ae087ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.538278 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 0167d1954bfb47deb89c890f1de063f3 (DEFAULT_TABLE table=default.Table_f690a4140360442b92080c8d55fdc1e2 [id=73a236fad2434ef6ae33417069474719]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.538890 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 17a19ce92da8448bba471f94e3ee4647 (DEFAULT_TABLE table=default.Table_e9839f0986064565be4d73e81a8671da [id=06369c47e7ac421dbf721c768d0c208e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.539423 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 0167d1954bfb47deb89c890f1de063f3 (DEFAULT_TABLE table=default.Table_f690a4140360442b92080c8d55fdc1e2 [id=73a236fad2434ef6ae33417069474719]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.516723 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 52991b0eefbb4d1297266a95c17a4662 (DEFAULT_TABLE table=default.Table_bff95c9aec0642ed94a4b1355a7a32af [id=63bdb97aacc9414b9caaa496425dcdaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.342231 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 45d67d0ef7294b3ea63529db2cdaafaf (DEFAULT_TABLE table=default.Table_4cbc59c030d044d8a097385ed57a0b5f [id=83566d870a90461686826b2073b99f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.539780 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 911) took 1992 ms. Trace:
I20251027 21:08:05.539863 10041 rpcz_store.cc:276] 1027 21:08:03.547070 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:03.560462 (+ 13392us) service_pool.cc:225] Handling call
1027 21:08:05.498538 (+1938076us) ts_tablet_manager.cc:1151] Deleting tablet c0857f89bc74495081d35ee55e785835
1027 21:08:05.539655 (+ 41117us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:05.539763 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1937403}
I20251027 21:08:05.540148 10041 tablet_replica.cc:333] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:05.540459 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 931) took 1071 ms. Trace:
I20251027 21:08:05.540573 10043 rpcz_store.cc:276] 1027 21:08:04.468914 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.469747 (+ 833us) service_pool.cc:225] Handling call
1027 21:08:04.888261 (+418514us) ts_tablet_manager.cc:1151] Deleting tablet a339c0dee0994788b8644912f74c1b82
1027 21:08:04.888268 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:05.207392 (+319124us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:05.397311 (+189919us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:05.540414 (+143103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":416665}
I20251027 21:08:05.541200 10043 tablet_replica.cc:333] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:05.541416 10043 raft_consensus.cc:2243] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:05.541586 10043 raft_consensus.cc:2272] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:05.540294 10041 raft_consensus.cc:2243] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:05.542682 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d64a13667194c12864fa388eb88a380" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:05.542851 10041 raft_consensus.cc:2272] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:05.543313 10041 ts_tablet_manager.cc:1916] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:05.543818 9860 leader_election.cc:343] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:05.544914 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaa8910d9fdb4146945621ec4ebf969e" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:05.545437 9566 leader_election.cc:343] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:05.547435 9859 leader_election.cc:343] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:05.547538 9859 leader_election.cc:304] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:05.547796 11121 raft_consensus.cc:2749] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:05.548197 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 53cac7817db440e79e7416d625a9b2c8 (table default.Table_79a3420c8b6542868231c740293ab1fb [id=f3babbbf131543d2813ed9d5882e3d00]) successfully deleted
I20251027 21:08:05.553382 9888 tablet_service.cc:1505] Processing CreateTablet for tablet b1c06652f7bd435b823e07198c713a10 (DEFAULT_TABLE table=default.Table_d9decf014b434b80b9d5529bd7aafce3 [id=7a385fd969d74bee9f340cd2afb4c575]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.557986 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 0ca2708d87b7440ca97913f001deb38f (DEFAULT_TABLE table=default.Table_3455942c45f2475ca47232041641c48f [id=97fc7d303579467abbbd24405f76ff33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.558542 9611 tablet_service.cc:1505] Processing CreateTablet for tablet ebe32981d5e84af3b8a33c1a949dd97f (DEFAULT_TABLE table=default.Table_c55370d227fd4e1baaca17e66bbaf3f6 [id=82d1dce90ff848abab3052ed5bef7209]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.559829 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 2c067b84a1b6467eaf701d5ea2a4960f (DEFAULT_TABLE table=default.Table_514660e85ec84a728ae8fa885f7c0532 [id=235f236e2aea4ead8213e0a4f92a37fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.562991 10043 ts_tablet_manager.cc:1916] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:05.563386 10031 tablet_bootstrap.cc:492] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:05.563503 10031 ts_tablet_manager.cc:1403] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.330s user 0.001s sys 0.001s
I20251027 21:08:05.564240 10031 raft_consensus.cc:359] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.564380 10031 raft_consensus.cc:385] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:05.564429 10031 raft_consensus.cc:740] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:05.564605 10031 consensus_queue.cc:260] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.567251 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_877147b58eff438190609ce0142c08f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:05.567893 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_877147b58eff438190609ce0142c08f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:05.569890 9312 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 87d5af4607f04f92abcc7292b4176cbd (table default.Table_c78ef9ebe86b4da08443e37c83d4e80d [id=cdb023218cda4e9ea04a226b37a220d9]). Expected version 2 got 0
I20251027 21:08:05.570153 9312 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 6600774e4c7b48479a2fdb02614ba505 (table default.Table_d7530773668f42c5a07a921df02a95ab [id=914b728e358e4fdba6d38d6c7beadaca]). Expected version 1 got 0
I20251027 21:08:05.576066 10031 ts_tablet_manager.cc:1434] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.012s user 0.001s sys 0.005s
I20251027 21:08:05.576383 10031 tablet_bootstrap.cc:492] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:05.578028 10031 tablet_bootstrap.cc:654] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:05.578181 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fde4fd91943498d8888d1783005e492" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:05.578351 9924 raft_consensus.cc:2468] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:05.579713 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaa8910d9fdb4146945621ec4ebf969e" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:05.579849 9924 raft_consensus.cc:2468] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:05.583788 10029 tablet_bootstrap.cc:492] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:05.584555 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet a339c0dee0994788b8644912f74c1b82 (table default.Table_4a455030bbe945c391d97696ee50010e [id=fb030c6e79b44af4b3469a20e09a8e47]) successfully deleted
W20251027 21:08:05.584841 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 719321644ea3479ea4529a8a5d048754 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 719321644ea3479ea4529a8a5d048754 already in progress: creating tablet
W20251027 21:08:05.585052 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet b83daa33ef0f45c6a78c70001e532ea2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b83daa33ef0f45c6a78c70001e532ea2 already in progress: creating tablet
I20251027 21:08:05.589076 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 01e9db9eaf8e41c6a2ee13b9ed127421 (table default.Table_11fbadd9eed14cdba7664715e389f7e5 [id=83ec44dba5c444aba48846811a5f5fc9]) successfully deleted
W20251027 21:08:05.589552 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet c0857f89bc74495081d35ee55e785835 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0857f89bc74495081d35ee55e785835 already in progress: creating tablet
I20251027 21:08:05.592190 9311 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_fd331f3cfc07425082d58ab5af3c1094 [id=f3e3147411e5488bae105ed755f5b50f] alter complete (version 2)
I20251027 21:08:05.592670 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet f9534e65d8bb4b3a88df934871792119 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:05.595077 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_907ada415b694cf4817ef1e6f6f68e9d" } modify_external_catalogs: true
I20251027 21:08:05.595242 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_907ada415b694cf4817ef1e6f6f68e9d" } modify_external_catalogs: true
I20251027 21:08:05.597196 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_653e76a8cae24ff38c2eee3d9322aa6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:05.597709 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_653e76a8cae24ff38c2eee3d9322aa6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:05.599929 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_66beca448bc04e81aecbc35fd309fe12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:05.600399 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66beca448bc04e81aecbc35fd309fe12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:05.612283 10829 raft_consensus.cc:2749] T ca477937278f45b4b115fec715f32a92 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:05.615097 11057 raft_consensus.cc:493] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:05.615234 11057 raft_consensus.cc:515] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.615854 11057 leader_election.cc:290] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:05.616044 9745 tablet_service.cc:1505] Processing CreateTablet for tablet b1c06652f7bd435b823e07198c713a10 (DEFAULT_TABLE table=default.Table_d9decf014b434b80b9d5529bd7aafce3 [id=7a385fd969d74bee9f340cd2afb4c575]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.616452 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b30052ff19744a9bf4bfaaa1bc77b36" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:05.617347 9566 leader_election.cc:343] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:05.617695 11057 consensus_queue.cc:1048] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:05.621711 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91d29318b91341e2a7c00a0841c6def6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.622622 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17a19ce92da8448bba471f94e3ee4647. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.623133 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0167d1954bfb47deb89c890f1de063f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.623472 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe24988809f4300934efe90ec326559. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.623741 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fddac33cd6247ee81a0117c8a2e5e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.624192 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe24988809f4300934efe90ec326559. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.624348 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45d67d0ef7294b3ea63529db2cdaafaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.624575 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45d67d0ef7294b3ea63529db2cdaafaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.624722 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3604203c122a4df291ecdc1d8477dca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.624982 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5fd809f50e2494bb6d815b548f18481. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.625077 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f838fb6e148c4b16b062e831c57cd454. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.625401 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e372257733148199d2dbe70d0887cbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.625407 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4540d5fbb5bc4611bc0104bf840bd12d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.625763 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de5213c0a78413782c0e9d1bc6c3849. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.625783 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e372257733148199d2dbe70d0887cbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.626153 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3604203c122a4df291ecdc1d8477dca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.626153 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32c32d85389f4255ae2041e4481a3765. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.626490 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47d809c7d0c9412e81fa9cc497063e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.626499 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47d809c7d0c9412e81fa9cc497063e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.626866 9901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de5213c0a78413782c0e9d1bc6c3849. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.626883 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32c32d85389f4255ae2041e4481a3765. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.627228 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fddac33cd6247ee81a0117c8a2e5e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.627235 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f838fb6e148c4b16b062e831c57cd454. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.627611 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 348898b63dca46f4a5c9a7e34b545d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.627606 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcf5c7d38e6143a8869227f85ae37270. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.627980 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0167d1954bfb47deb89c890f1de063f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.627980 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb5ac0fff70b491e8c8842f7a331005e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.628324 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 511fdda39953415aaa65d88cb75ad382. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.628523 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 348898b63dca46f4a5c9a7e34b545d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.628858 9886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5fd809f50e2494bb6d815b548f18481. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.629169 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7c9623ff35e4ae9a2ead45bef605e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.629541 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91d29318b91341e2a7c00a0841c6def6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.630857 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1c06652f7bd435b823e07198c713a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.633482 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 52991b0eefbb4d1297266a95c17a4662 (DEFAULT_TABLE table=default.Table_bff95c9aec0642ed94a4b1355a7a32af [id=63bdb97aacc9414b9caaa496425dcdaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.633777 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52991b0eefbb4d1297266a95c17a4662. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.636937 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1c06652f7bd435b823e07198c713a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.638068 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1c06652f7bd435b823e07198c713a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.638311 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 348898b63dca46f4a5c9a7e34b545d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.638921 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ab8cd9b46d446b3922909c715964ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.639446 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52991b0eefbb4d1297266a95c17a4662. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.639947 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3604203c122a4df291ecdc1d8477dca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.640369 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e372257733148199d2dbe70d0887cbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.640792 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f838fb6e148c4b16b062e831c57cd454. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.641305 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6919e57d8dae48a890c041109ed253c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.641762 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32c32d85389f4255ae2041e4481a3765. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.647178 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47d809c7d0c9412e81fa9cc497063e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.647583 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45d67d0ef7294b3ea63529db2cdaafaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.650964 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ca2708d87b7440ca97913f001deb38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.652436 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 0ab8cd9b46d446b3922909c715964ee8 (DEFAULT_TABLE table=default.Table_0393cf4c52244b19b2f59cc4a13f8dee [id=f151f5778c0a4bee8b442885ae087ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.674290 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebe32981d5e84af3b8a33c1a949dd97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.677217 11057 raft_consensus.cc:493] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:05.677366 11057 raft_consensus.cc:515] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.678086 11057 leader_election.cc:290] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:05.688196 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c067b84a1b6467eaf701d5ea2a4960f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.657531 9739 tablet_service.cc:1505] Processing CreateTablet for tablet 0ca2708d87b7440ca97913f001deb38f (DEFAULT_TABLE table=default.Table_3455942c45f2475ca47232041641c48f [id=97fc7d303579467abbbd24405f76ff33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.698889 9733 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:40722 (request call id 149) took 1051 ms. Trace:
I20251027 21:08:05.699136 9733 rpcz_store.cc:276] 1027 21:08:04.647084 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.651721 (+ 4637us) service_pool.cc:225] Handling call
1027 21:08:05.117144 (+465423us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x616000135454. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d8fee1200 00007f8d93d9166f 00007f8d8e1b7c36 00007f8d8e1b7965 00007f8d8afe627e 00007f8d8afe515e 00007f8d8afeb395 00007f8d8afebcbf 00007f8d8afebae8 0000000000a8867c
1027 21:08:05.225392 (+108248us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x6160001b1654. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d8fee1200 00007f8d93d9166f 00007f8d8e1b7c36 00007f8d8e1b7965 00007f8d8afe627e 00007f8d8afe515e 00007f8d8afeb395 00007f8d8afebcbf 00007f8d8afebae8 0000000000a8867c
1027 21:08:05.499992 (+274600us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x6160001b1c54. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d8fee1200 00007f8d93d9166f 00007f8d8e1b7c36 00007f8d8e1b7965 00007f8d8afe627e 00007f8d8afe515e 00007f8d8afeb395 00007f8d8afebcbf 00007f8d8afebae8 0000000000a8867c
1027 21:08:05.683558 (+183566us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x6160002a1430. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d91269829 00007f8d93d91478 00007f8d8e1b7c36 00007f8d8e1b7965 00007f8d8afe627e 00007f8d8afe515e 00007f8d8afeb395 00007f8d8afebcbf 00007f8d8afebae8 0000000000a8867c
1027 21:08:05.698792 (+ 15234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1894950016}
W20251027 21:08:05.707080 9568 leader_election.cc:343] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: da26c83976df40328cf9b11734fcfa44
I20251027 21:08:05.707675 10040 ts_tablet_manager.cc:1929] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:05.707832 10040 log.cc:1199] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/2532bf7190064443a898e47e843fd9ac
I20251027 21:08:05.708176 10040 ts_tablet_manager.cc:1950] T 2532bf7190064443a898e47e843fd9ac P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:05.708894 9923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "980eb6755aee4bcfb2ea44d84009021f" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:05.709098 9923 raft_consensus.cc:2468] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:05.715154 9568 leader_election.cc:304] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:05.715927 11075 raft_consensus.cc:493] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:05.716161 11075 raft_consensus.cc:515] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.717422 11075 leader_election.cc:290] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:05.718020 11075 raft_consensus.cc:2749] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:05.718924 9568 leader_election.cc:304] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:05.719182 11075 raft_consensus.cc:2804] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:05.719242 11075 raft_consensus.cc:493] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:05.719293 11075 raft_consensus.cc:3060] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:05.722283 9568 leader_election.cc:304] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:05.722831 9568 leader_election.cc:304] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:05.726918 11097 raft_consensus.cc:2804] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:05.727005 11097 raft_consensus.cc:493] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:05.727065 11097 raft_consensus.cc:3060] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:05.731959 10032 tablet_bootstrap.cc:492] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:05.732079 10032 ts_tablet_manager.cc:1403] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.247s user 0.009s sys 0.013s
I20251027 21:08:05.732740 10032 raft_consensus.cc:359] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.732944 10032 raft_consensus.cc:385] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:05.733009 10032 raft_consensus.cc:740] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:05.733198 10032 consensus_queue.cc:260] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.733668 10032 ts_tablet_manager.cc:1434] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:05.733933 10032 tablet_bootstrap.cc:492] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:05.737737 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 919) took 1442 ms. Trace:
I20251027 21:08:05.737864 10040 rpcz_store.cc:276] 1027 21:08:04.295421 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.295575 (+ 154us) service_pool.cc:225] Handling call
1027 21:08:05.522900 (+1227325us) ts_tablet_manager.cc:1151] Deleting tablet 2532bf7190064443a898e47e843fd9ac
1027 21:08:05.522909 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:05.702364 (+179455us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:05.707588 (+ 5224us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:05.737717 (+ 30129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1227040}
I20251027 21:08:05.738267 10040 tablet_replica.cc:333] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:05.738442 10040 raft_consensus.cc:2243] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:05.738592 10040 raft_consensus.cc:2272] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:05.739485 10040 ts_tablet_manager.cc:1916] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:05.745859 10031 tablet_bootstrap.cc:492] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:05.745868 11018 raft_consensus.cc:2804] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:05.745951 11018 raft_consensus.cc:493] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:05.745990 10031 ts_tablet_manager.cc:1403] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.170s user 0.035s sys 0.027s
I20251027 21:08:05.746021 11018 raft_consensus.cc:3060] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:05.746749 10031 raft_consensus.cc:359] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.746908 10031 raft_consensus.cc:385] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:05.746970 10031 raft_consensus.cc:740] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:05.747160 10031 consensus_queue.cc:260] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.747648 10031 ts_tablet_manager.cc:1434] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 21:08:05.747949 10031 tablet_bootstrap.cc:492] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:05.749405 10031 tablet_bootstrap.cc:654] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:05.750699 10032 tablet_bootstrap.cc:654] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:05.757308 10029 ts_tablet_manager.cc:1403] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.264s user 0.022s sys 0.010s
I20251027 21:08:05.760339 10031 tablet_bootstrap.cc:492] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:05.760492 10031 ts_tablet_manager.cc:1403] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20251027 21:08:05.761323 10031 raft_consensus.cc:359] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:05.761500 10031 raft_consensus.cc:385] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:05.761592 10031 raft_consensus.cc:740] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:05.761829 10031 consensus_queue.cc:260] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:05.762455 10031 ts_tablet_manager.cc:1434] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 21:08:05.762732 10031 tablet_bootstrap.cc:492] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:05.764148 10031 tablet_bootstrap.cc:654] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:05.761516 10029 raft_consensus.cc:359] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:05.765333 10029 raft_consensus.cc:385] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:05.766041 10032 tablet_bootstrap.cc:492] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:05.765942 10029 raft_consensus.cc:740] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:05.766881 10032 ts_tablet_manager.cc:1403] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20251027 21:08:05.767414 10029 consensus_queue.cc:260] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:05.771487 10032 raft_consensus.cc:359] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.772431 10032 raft_consensus.cc:385] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:05.772843 10032 raft_consensus.cc:740] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:05.773587 10029 ts_tablet_manager.cc:1434] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20251027 21:08:05.773864 10032 consensus_queue.cc:260] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.775940 10032 ts_tablet_manager.cc:1434] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20251027 21:08:05.777498 10032 tablet_bootstrap.cc:492] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:05.779455 11051 raft_consensus.cc:493] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:05.779601 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4aa1fe05114c1287ca580c1f411d3f" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:05.779598 11051 raft_consensus.cc:515] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.780198 11051 leader_election.cc:290] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:05.784986 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b6818d29624da5a49dd1a7d20994c6" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:05.785012 9566 leader_election.cc:343] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:05.790625 9566 leader_election.cc:343] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:05.795517 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 2532bf7190064443a898e47e843fd9ac (table default.Table_89e2320f599341e882a2d09c0985b220 [id=c6f20d8985134d9a8d3c5bb4a51b45fb]) successfully deleted
I20251027 21:08:05.804502 11075 raft_consensus.cc:515] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.805064 11075 leader_election.cc:290] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:05.814911 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fde4fd91943498d8888d1783005e492" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:05.815400 9566 leader_election.cc:343] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:05.827353 10032 tablet_bootstrap.cc:654] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:05.840392 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 52991b0eefbb4d1297266a95c17a4662 (DEFAULT_TABLE table=default.Table_bff95c9aec0642ed94a4b1355a7a32af [id=63bdb97aacc9414b9caaa496425dcdaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.840773 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52991b0eefbb4d1297266a95c17a4662. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.840907 10032 tablet_bootstrap.cc:492] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:05.841024 10032 ts_tablet_manager.cc:1403] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.006s
I20251027 21:08:05.841796 10032 raft_consensus.cc:359] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.841964 10032 raft_consensus.cc:385] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:05.842029 10032 raft_consensus.cc:740] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:05.842235 10032 consensus_queue.cc:260] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.842664 10032 ts_tablet_manager.cc:1434] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:05.842971 10032 tablet_bootstrap.cc:492] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:05.844415 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 0ab8cd9b46d446b3922909c715964ee8 (DEFAULT_TABLE table=default.Table_0393cf4c52244b19b2f59cc4a13f8dee [id=f151f5778c0a4bee8b442885ae087ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.844764 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ab8cd9b46d446b3922909c715964ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.848426 10032 tablet_bootstrap.cc:654] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:05.848829 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 0ca2708d87b7440ca97913f001deb38f (DEFAULT_TABLE table=default.Table_3455942c45f2475ca47232041641c48f [id=97fc7d303579467abbbd24405f76ff33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.849172 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ca2708d87b7440ca97913f001deb38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.850512 10040 ts_tablet_manager.cc:1929] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:05.850637 10040 log.cc:1199] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/a339c0dee0994788b8644912f74c1b82
I20251027 21:08:05.850936 10040 ts_tablet_manager.cc:1950] T a339c0dee0994788b8644912f74c1b82 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:05.852948 9890 tablet_service.cc:1505] Processing CreateTablet for tablet ebe32981d5e84af3b8a33c1a949dd97f (DEFAULT_TABLE table=default.Table_c55370d227fd4e1baaca17e66bbaf3f6 [id=82d1dce90ff848abab3052ed5bef7209]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.853245 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebe32981d5e84af3b8a33c1a949dd97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.854346 11097 raft_consensus.cc:515] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.855074 11097 leader_election.cc:290] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:05.856462 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 2c067b84a1b6467eaf701d5ea2a4960f (DEFAULT_TABLE table=default.Table_514660e85ec84a728ae8fa885f7c0532 [id=235f236e2aea4ead8213e0a4f92a37fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.856807 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c067b84a1b6467eaf701d5ea2a4960f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.857565 9900 tablet_service.cc:1505] Processing CreateTablet for tablet d3a08a60c5e8461296ea8528eda90476 (DEFAULT_TABLE table=default.Table_25435d2d53084e0aa737382bf0fa0e2c [id=a253b23efcca4f01a7bea9d4e8319960]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.858172 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaa8910d9fdb4146945621ec4ebf969e" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:05.858700 9566 leader_election.cc:343] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:05.859673 10829 raft_consensus.cc:493] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:05.859802 10829 raft_consensus.cc:515] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.860596 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a08a60c5e8461296ea8528eda90476. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.860600 10829 leader_election.cc:290] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:05.861289 10032 tablet_bootstrap.cc:492] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:05.861445 10032 ts_tablet_manager.cc:1403] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
W20251027 21:08:05.862010 9710 leader_election.cc:343] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 67bb12de9f044c73ae9ab26d10498c90
I20251027 21:08:05.862582 10032 raft_consensus.cc:359] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.862772 10032 raft_consensus.cc:385] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:05.862844 10032 raft_consensus.cc:740] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:05.863050 10032 consensus_queue.cc:260] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.863500 10032 ts_tablet_manager.cc:1434] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:05.889536 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 2c067b84a1b6467eaf701d5ea2a4960f (DEFAULT_TABLE table=default.Table_514660e85ec84a728ae8fa885f7c0532 [id=235f236e2aea4ead8213e0a4f92a37fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.891611 9754 tablet_service.cc:1505] Processing CreateTablet for tablet d3a08a60c5e8461296ea8528eda90476 (DEFAULT_TABLE table=default.Table_25435d2d53084e0aa737382bf0fa0e2c [id=a253b23efcca4f01a7bea9d4e8319960]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.895412 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 8827e4746ecc408a8fea98b6b5ddd57b (DEFAULT_TABLE table=default.Table_9f371b16693a415896291c6a3edf3d25 [id=1efe103ddab14a8db22ecddcbf3f37c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.896086 11097 raft_consensus.cc:493] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:05.896214 11097 raft_consensus.cc:515] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.896795 11097 leader_election.cc:290] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:05.897954 11097 raft_consensus.cc:493] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:05.898061 11097 raft_consensus.cc:515] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.898612 11097 leader_election.cc:290] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:05.899602 11097 consensus_queue.cc:1048] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:05.678828 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "980eb6755aee4bcfb2ea44d84009021f" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:05.897805 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94f2819dd0394530ae93754bdd1e482f" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:05.899331 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bbb4116d1edd492893adfbf5fb3ddeef" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:05.901180 9566 leader_election.cc:343] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:05.901468 9566 leader_election.cc:343] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:05.901611 9566 leader_election.cc:343] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:05.901990 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8827e4746ecc408a8fea98b6b5ddd57b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.908896 9748 tablet_service.cc:1505] Processing CreateTablet for tablet b162734c536a40c08755201a85255f4c (DEFAULT_TABLE table=default.Table_b1f06adb64714ba5818da1b0c060fd16 [id=227e5aeea5f54cea9c17a0f286671216]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.909228 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b162734c536a40c08755201a85255f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.913245 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ab8cd9b46d446b3922909c715964ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.915273 9742 tablet_service.cc:1505] Processing CreateTablet for tablet f954daa93d5a42bfabfbf8fd4b72172b (DEFAULT_TABLE table=default.Table_f2e9f61a51bd4e1796f6bda56f48a6a1 [id=d2773f589a344817bd4d66df63f7abc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.915632 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c067b84a1b6467eaf701d5ea2a4960f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.917467 9747 tablet_service.cc:1505] Processing CreateTablet for tablet 218d244dc8a549408863110fefbc7b20 (DEFAULT_TABLE table=default.Table_6b527fc023dd45f4ab080eddeeb8bb76 [id=5888beb9284242a8bf5a7c8f2649d62c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.919237 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 60683685808643bcaf5845870d3f783d (DEFAULT_TABLE table=default.Table_3e61b43276ac41a2b626158c42642384 [id=2b6c44bf73684792a987f15a61c00a19]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.920908 9743 tablet_service.cc:1505] Processing CreateTablet for tablet 53c13ebc2e3f46e996f521f234c5b015 (DEFAULT_TABLE table=default.Table_12e5e752397f4aadae74519cfdf8f4d7 [id=3288cd17e43d44569d222b2c55676aaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:05.923694 11075 raft_consensus.cc:493] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:05.923843 11075 raft_consensus.cc:515] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:05.925051 11075 leader_election.cc:290] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:05.926110 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cef2bfee222f45e6b6f20c6d63ff44d4" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:05.926904 9566 leader_election.cc:343] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:05.938050 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ca2708d87b7440ca97913f001deb38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:05.955340 10041 ts_tablet_manager.cc:1929] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:05.955454 10041 log.cc:1199] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/a4d26f73b8534f08bcaa386d4de75c01
I20251027 21:08:05.955708 10041 ts_tablet_manager.cc:1950] T a4d26f73b8534f08bcaa386d4de75c01 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:05.959707 11018 raft_consensus.cc:515] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:05.960491 11018 leader_election.cc:290] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:05.969926 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "980eb6755aee4bcfb2ea44d84009021f" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:05.973254 9566 leader_election.cc:343] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:05.660243 9755 tablet_service.cc:1505] Processing CreateTablet for tablet ebe32981d5e84af3b8a33c1a949dd97f (DEFAULT_TABLE table=default.Table_c55370d227fd4e1baaca17e66bbaf3f6 [id=82d1dce90ff848abab3052ed5bef7209]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:05.982750 9774 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.000s sys 0.005s
W20251027 21:08:05.983086 9774 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.516s user 0.000s sys 0.005s
I20251027 21:08:05.988811 10997 raft_consensus.cc:2749] T 6600774e4c7b48479a2fdb02614ba505 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251027 21:08:06.002480 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 921) took 1699 ms. Trace:
I20251027 21:08:06.002590 10040 rpcz_store.cc:276] 1027 21:08:04.303012 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.308447 (+ 5435us) service_pool.cc:225] Handling call
1027 21:08:05.738210 (+1429763us) ts_tablet_manager.cc:1151] Deleting tablet a339c0dee0994788b8644912f74c1b82
1027 21:08:05.738231 (+ 21us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:05.805326 (+ 67095us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:05.850396 (+ 45070us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.002454 (+152058us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1410215}
I20251027 21:08:06.003026 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 923) took 1686 ms. Trace:
I20251027 21:08:06.003095 10040 rpcz_store.cc:276] 1027 21:08:04.316890 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.317027 (+ 137us) service_pool.cc:225] Handling call
1027 21:08:06.002902 (+1685875us) ts_tablet_manager.cc:1151] Deleting tablet f9534e65d8bb4b3a88df934871792119
1027 21:08:06.002912 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:06.003007 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1674744}
I20251027 21:08:06.003340 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 924) took 1649 ms. Trace:
I20251027 21:08:06.003391 10040 rpcz_store.cc:276] 1027 21:08:04.353787 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.353838 (+ 51us) service_pool.cc:225] Handling call
1027 21:08:06.003268 (+1649430us) ts_tablet_manager.cc:1151] Deleting tablet 0ac1aed5f56844639b848f90c8eb5b6c
1027 21:08:06.003274 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:06.003329 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1649242}
I20251027 21:08:06.003613 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 933) took 1376 ms. Trace:
I20251027 21:08:06.003665 10040 rpcz_store.cc:276] 1027 21:08:04.626832 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.627450 (+ 618us) service_pool.cc:225] Handling call
1027 21:08:06.003541 (+1376091us) ts_tablet_manager.cc:1151] Deleting tablet 719321644ea3479ea4529a8a5d048754
1027 21:08:06.003548 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:06.003602 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1375895}
I20251027 21:08:06.003901 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 934) took 1374 ms. Trace:
I20251027 21:08:06.003952 10040 rpcz_store.cc:276] 1027 21:08:04.629684 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.630737 (+ 1053us) service_pool.cc:225] Handling call
1027 21:08:06.003817 (+1373080us) ts_tablet_manager.cc:1151] Deleting tablet b83daa33ef0f45c6a78c70001e532ea2
1027 21:08:06.003824 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:06.003889 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1372806}
I20251027 21:08:06.004212 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 937) took 1210 ms. Trace:
I20251027 21:08:06.004272 10040 rpcz_store.cc:276] 1027 21:08:04.793740 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.793789 (+ 49us) service_pool.cc:225] Handling call
1027 21:08:06.004110 (+1210321us) ts_tablet_manager.cc:1151] Deleting tablet ee0c598904004dd18df058da483aa45c
1027 21:08:06.004119 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:06.004198 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1210155}
I20251027 21:08:06.004468 10040 tablet_replica.cc:333] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:06.004622 10040 raft_consensus.cc:2243] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:06.004765 10040 raft_consensus.cc:2272] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:06.005565 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 07b8fd24b8af4f8db2a401c26cc65058 (DEFAULT_TABLE table=default.Table_fb6cfb8df03347efbf6dd0002e05bf86 [id=a9b801ae4df54e82a3748fb95f7d467d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.005950 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07b8fd24b8af4f8db2a401c26cc65058. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.006340 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a08a60c5e8461296ea8528eda90476. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.006877 10040 ts_tablet_manager.cc:1916] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:06.013084 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 93e8b973b39244be9de1107b14d6d93f (DEFAULT_TABLE table=default.Table_7ed5ee38ed7a4cf1a5a706eb6e233d41 [id=3bf292438c2749649a155a8b2332cd3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.015017 9740 tablet_service.cc:1505] Processing CreateTablet for tablet c35e5e38b9ab4bb88efa30fab9a9f9d3 (DEFAULT_TABLE table=default.Table_9e0da1160aff4c58887dbcf3ac217299 [id=7bdc02392ba542c7be65a5de5b0bdf8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.020797 9924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67bb12de9f044c73ae9ab26d10498c90" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:06.020998 9924 raft_consensus.cc:2468] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:08:06.021443 9711 leader_election.cc:304] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218; no voters: 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:06.021811 10997 raft_consensus.cc:2804] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:06.021886 10997 raft_consensus.cc:493] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:06.021941 10997 raft_consensus.cc:3060] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:06.025107 9903 tablet_service.cc:1505] Processing CreateTablet for tablet b162734c536a40c08755201a85255f4c (DEFAULT_TABLE table=default.Table_b1f06adb64714ba5818da1b0c060fd16 [id=227e5aeea5f54cea9c17a0f286671216]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.025257 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 8827e4746ecc408a8fea98b6b5ddd57b (DEFAULT_TABLE table=default.Table_9f371b16693a415896291c6a3edf3d25 [id=1efe103ddab14a8db22ecddcbf3f37c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.025496 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b162734c536a40c08755201a85255f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.026422 10032 tablet_bootstrap.cc:492] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:06.026500 9894 tablet_service.cc:1505] Processing CreateTablet for tablet f954daa93d5a42bfabfbf8fd4b72172b (DEFAULT_TABLE table=default.Table_f2e9f61a51bd4e1796f6bda56f48a6a1 [id=d2773f589a344817bd4d66df63f7abc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.014074 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet a339c0dee0994788b8644912f74c1b82 (table default.Table_4a455030bbe945c391d97696ee50010e [id=fb030c6e79b44af4b3469a20e09a8e47]) successfully deleted
W20251027 21:08:06.027572 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet f9534e65d8bb4b3a88df934871792119 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9534e65d8bb4b3a88df934871792119 already in progress: creating tablet
W20251027 21:08:06.027801 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 0ac1aed5f56844639b848f90c8eb5b6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ac1aed5f56844639b848f90c8eb5b6c already in progress: creating tablet
W20251027 21:08:06.027999 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 719321644ea3479ea4529a8a5d048754 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 719321644ea3479ea4529a8a5d048754 already in progress: creating tablet
W20251027 21:08:06.028287 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet b83daa33ef0f45c6a78c70001e532ea2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b83daa33ef0f45c6a78c70001e532ea2 already in progress: creating tablet
I20251027 21:08:06.028373 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f954daa93d5a42bfabfbf8fd4b72172b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.028460 10032 tablet_bootstrap.cc:654] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:06.028491 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet ee0c598904004dd18df058da483aa45c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee0c598904004dd18df058da483aa45c already in progress: creating tablet
I20251027 21:08:06.030937 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8827e4746ecc408a8fea98b6b5ddd57b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:06.040019 9568 leader_election.cc:343] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 8b30052ff19744a9bf4bfaaa1bc77b36
I20251027 21:08:06.040144 9568 leader_election.cc:304] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:06.040468 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aadae0184a8e454cbe40d0c8491d9bd5 consensus state has no leader
I20251027 21:08:06.042945 9608 tablet_service.cc:1505] Processing CreateTablet for tablet d3a08a60c5e8461296ea8528eda90476 (DEFAULT_TABLE table=default.Table_25435d2d53084e0aa737382bf0fa0e2c [id=a253b23efcca4f01a7bea9d4e8319960]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.048131 10032 tablet_bootstrap.cc:492] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:06.048287 10032 ts_tablet_manager.cc:1403] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.009s
I20251027 21:08:06.049048 10032 raft_consensus.cc:359] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.049255 10032 raft_consensus.cc:385] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:06.049329 10032 raft_consensus.cc:740] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:06.049593 10032 consensus_queue.cc:260] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.050271 10032 ts_tablet_manager.cc:1434] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251027 21:08:06.052953 10829 raft_consensus.cc:493] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:06.053068 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 8827e4746ecc408a8fea98b6b5ddd57b (DEFAULT_TABLE table=default.Table_9f371b16693a415896291c6a3edf3d25 [id=1efe103ddab14a8db22ecddcbf3f37c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.053113 10829 raft_consensus.cc:515] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:06.054127 10829 leader_election.cc:290] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:06.054970 11057 raft_consensus.cc:2749] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:06.055050 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9efffa9766e46aca8eb286fc2ad48d2" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:06.055280 9635 raft_consensus.cc:2468] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:08:06.055702 9710 leader_election.cc:304] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:06.061028 10997 raft_consensus.cc:515] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.061625 10997 leader_election.cc:290] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:06.061765 10997 raft_consensus.cc:2804] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:06.061823 10997 raft_consensus.cc:493] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:06.061894 10997 raft_consensus.cc:3060] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:06.064494 9710 leader_election.cc:343] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 67bb12de9f044c73ae9ab26d10498c90
I20251027 21:08:06.071513 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 8b63d498ce8e4c3cafbefa650b657250 (DEFAULT_TABLE table=default.Table_0a1b25591a094198b8123b3ac2af4715 [id=4b0f47c0f5344b1bafcef5e50f2d6cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.072674 9744 tablet_service.cc:1505] Processing CreateTablet for tablet fd511b6dd9544432bd4bec2822642906 (DEFAULT_TABLE table=default.Table_088cee7e137e418599e9dc3d73b42640 [id=a8254167aec04d8781120aabe34acca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.094693 11065 raft_consensus.cc:493] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:06.094851 11065 raft_consensus.cc:515] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.095485 11065 leader_election.cc:290] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:06.096498 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07a9c3b7ef1548ec9d5ae9f20545f263" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:06.097194 9860 leader_election.cc:343] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:06.103415 10029 tablet_bootstrap.cc:492] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:06.103601 9594 tablet_service.cc:1505] Processing CreateTablet for tablet b162734c536a40c08755201a85255f4c (DEFAULT_TABLE table=default.Table_b1f06adb64714ba5818da1b0c060fd16 [id=227e5aeea5f54cea9c17a0f286671216]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.104766 10029 tablet_bootstrap.cc:654] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:06.104710 9595 tablet_service.cc:1505] Processing CreateTablet for tablet f954daa93d5a42bfabfbf8fd4b72172b (DEFAULT_TABLE table=default.Table_f2e9f61a51bd4e1796f6bda56f48a6a1 [id=d2773f589a344817bd4d66df63f7abc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.107533 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 218d244dc8a549408863110fefbc7b20 (DEFAULT_TABLE table=default.Table_6b527fc023dd45f4ab080eddeeb8bb76 [id=5888beb9284242a8bf5a7c8f2649d62c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.107947 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 60683685808643bcaf5845870d3f783d (DEFAULT_TABLE table=default.Table_3e61b43276ac41a2b626158c42642384 [id=2b6c44bf73684792a987f15a61c00a19]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.108263 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 53c13ebc2e3f46e996f521f234c5b015 (DEFAULT_TABLE table=default.Table_12e5e752397f4aadae74519cfdf8f4d7 [id=3288cd17e43d44569d222b2c55676aaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.109208 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 07b8fd24b8af4f8db2a401c26cc65058 (DEFAULT_TABLE table=default.Table_fb6cfb8df03347efbf6dd0002e05bf86 [id=a9b801ae4df54e82a3748fb95f7d467d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:06.111575 9566 consensus_peers.cc:597] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:06.113229 10859 consensus_queue.cc:1048] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:06.076841 9749 tablet_service.cc:1505] Processing CreateTablet for tablet cbd77880ad934843b659d7b669eaafc7 (DEFAULT_TABLE table=default.Table_27e2b6a6c7684a7baefcca99d2f93e87 [id=f0ea1e2e6c04444ab1af5dab52fb98ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.080865 9751 tablet_service.cc:1505] Processing CreateTablet for tablet 933fb952d66d4cb1ade70c334fbbb3e8 (DEFAULT_TABLE table=default.Table_f0f82afba21846a89a2fa83897fde34f [id=5209bcda10e64660be987d47e95a738c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.082919 9732 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:40722 (request call id 148) took 1435 ms. Trace:
I20251027 21:08:06.084228 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet 83f5fbf5757a49df93ab949b7657c61f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:06.084252 10031 tablet_bootstrap.cc:492] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:06.121035 9732 rpcz_store.cc:276] 1027 21:08:04.647022 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.651377 (+ 4355us) service_pool.cc:225] Handling call
1027 21:08:05.089789 (+438412us) spinlock_profiling.cc:243] Waited 231 ms on lock 0x616000135454. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d8fee1200 00007f8d93d9166f 00007f8d8e1b7c36 00007f8d8e1b7965 00007f8d8afe627e 00007f8d8afe515e 00007f8d8afeb395 00007f8d8afebcbf 00007f8d8afebae8 0000000000a8867c
1027 21:08:05.205436 (+115647us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x6160001b1654. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d8fee1200 00007f8d93d9166f 00007f8d8e1b7c36 00007f8d8e1b7965 00007f8d8afe627e 00007f8d8afe515e 00007f8d8afeb395 00007f8d8afebcbf 00007f8d8afebae8 0000000000a8867c
1027 21:08:05.522620 (+317184us) spinlock_profiling.cc:243] Waited 315 ms on lock 0x6160001b1c54. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d8fee1200 00007f8d93d9166f 00007f8d8e1b7c36 00007f8d8e1b7965 00007f8d8afe627e 00007f8d8afe515e 00007f8d8afeb395 00007f8d8afebcbf 00007f8d8afebae8 0000000000a8867c
1027 21:08:05.616778 (+ 94158us) spinlock_profiling.cc:243] Waited 92.9 ms on lock 0x6160000d6f54. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d8fee1200 00007f8d93d9166f 00007f8d8e1b7c36 00007f8d8e1b7965 00007f8d8afe627e 00007f8d8afe515e 00007f8d8afeb395 00007f8d8afebcbf 00007f8d8afebae8 0000000000a8867c
1027 21:08:06.034075 (+417297us) spinlock_profiling.cc:243] Waited 413 ms on lock 0x6160002a1430. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d91269829 00007f8d93d91478 00007f8d8e1b7c36 00007f8d8e1b7965 00007f8d8afe627e 00007f8d8afe515e 00007f8d8afeb395 00007f8d8afebcbf 00007f8d8afebae8 0000000000a8867c
1027 21:08:06.082872 (+ 48797us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2686955904}
I20251027 21:08:06.121169 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f1d1bd52556463fa306e51eb253af40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:05 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:06.121191 10031 ts_tablet_manager.cc:1403] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.359s user 0.040s sys 0.112s
I20251027 21:08:06.121482 9738 tablet_service.cc:1505] Processing CreateTablet for tablet b8c368698cfb40799571567e8bfdd8dc (DEFAULT_TABLE table=default.Table_7057cb41215c437391bce90bc612d1df [id=f6000ae9faf749e28606aa7fd092174e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.122108 10031 raft_consensus.cc:359] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.122303 10031 raft_consensus.cc:385] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:06.122360 10031 raft_consensus.cc:740] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:06.122633 10031 consensus_queue.cc:260] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.123162 10031 ts_tablet_manager.cc:1434] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:06.123485 10031 tablet_bootstrap.cc:492] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:06.085691 9741 tablet_service.cc:1552] Processing DeleteTablet for tablet 8649a3b611cc48ff8832c0fdeb128efe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:05 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:06.096010 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07a9c3b7ef1548ec9d5ae9f20545f263" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:06.116211 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f954daa93d5a42bfabfbf8fd4b72172b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.124097 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c35e5e38b9ab4bb88efa30fab9a9f9d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.124258 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60683685808643bcaf5845870d3f783d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.126271 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53c13ebc2e3f46e996f521f234c5b015. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.126770 10031 tablet_bootstrap.cc:654] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:06.127903 9859 leader_election.cc:343] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:06.128010 9859 leader_election.cc:304] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:06.128413 11065 raft_consensus.cc:2749] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:06.124106 9741 tablet_service.cc:1505] Processing CreateTablet for tablet 5611e840dffc4ce6a17e2530fe78fed8 (DEFAULT_TABLE table=default.Table_baa19560a48f42c7ab0ecad353eaa283 [id=620c56eff478436185b0a178580a4548]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.128840 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5611e840dffc4ce6a17e2530fe78fed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.129330 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 218d244dc8a549408863110fefbc7b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.130254 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93e8b973b39244be9de1107b14d6d93f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.131928 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 933fb952d66d4cb1ade70c334fbbb3e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.134392 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8c368698cfb40799571567e8bfdd8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.137830 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b63d498ce8e4c3cafbefa650b657250. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.143023 10031 tablet_bootstrap.cc:492] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:06.143131 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebe32981d5e84af3b8a33c1a949dd97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.143143 10031 ts_tablet_manager.cc:1403] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.007s
I20251027 21:08:06.143886 10031 raft_consensus.cc:359] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.144045 10031 raft_consensus.cc:385] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:06.144109 10031 raft_consensus.cc:740] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:06.144322 10031 consensus_queue.cc:260] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.144897 10031 ts_tablet_manager.cc:1434] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:06.145272 10031 tablet_bootstrap.cc:492] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:06.146404 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd77880ad934843b659d7b669eaafc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.146878 10031 tablet_bootstrap.cc:654] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:06.150312 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd511b6dd9544432bd4bec2822642906. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.160809 10997 raft_consensus.cc:515] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:06.162191 10997 leader_election.cc:290] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:06.163039 10031 tablet_bootstrap.cc:492] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:06.163162 10031 ts_tablet_manager.cc:1403] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20251027 21:08:06.163138 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9efffa9766e46aca8eb286fc2ad48d2" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:06.163308 9635 raft_consensus.cc:3060] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:06.163967 10031 raft_consensus.cc:359] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.164122 10031 raft_consensus.cc:385] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:06.164180 10031 raft_consensus.cc:740] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:06.164445 10031 consensus_queue.cc:260] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.165267 10031 ts_tablet_manager.cc:1434] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:06.166034 10031 tablet_bootstrap.cc:492] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:06.168149 10031 tablet_bootstrap.cc:654] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:06.168793 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 473b7f84dfcd4f1cb89afcfeaab33b34 (DEFAULT_TABLE table=default.Table_4f5e79447c4b4bab8c8c12cf2adab418 [id=277e5426ed8f40abb73ca9fa3d86faca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.169127 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 473b7f84dfcd4f1cb89afcfeaab33b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.170935 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 3ef2fff1381b465a89eeb5f82ae4fd12 (DEFAULT_TABLE table=default.Table_685b53b410ef4403890d1494d6d77042 [id=5cd248bd134f425ea9fd20f3c483db51]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.171202 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef2fff1381b465a89eeb5f82ae4fd12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.174932 9745 tablet_service.cc:1505] Processing CreateTablet for tablet cddb9e2e3a914c8ab4515cf4505333ee (DEFAULT_TABLE table=default.Table_f22296338b56467193b136c872ba30c4 [id=6fd72bf290b34763b0eefcd420ac3894]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.176010 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cddb9e2e3a914c8ab4515cf4505333ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.166050 9743 tablet_service.cc:1505] Processing CreateTablet for tablet 60f4872159c54d7ebcf7987704018928 (DEFAULT_TABLE table=default.Table_34bd7efc93714dca9c8bf72307608cf1 [id=82bb4517750f4f42ae6f26ef5b9feafa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.182533 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60f4872159c54d7ebcf7987704018928. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:06.207453 9297 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 178660us
I20251027 21:08:06.218832 11040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: Bootstrap replayed 1/1 log segments. Stats: ops{read=782 overwritten=0 applied=782 ignored=0} inserts{seen=758 ignored=0} mutations{seen=879 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 21:08:06.227638 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a08a60c5e8461296ea8528eda90476. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.230340 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8827e4746ecc408a8fea98b6b5ddd57b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.231077 9612 tablet_service.cc:1505] Processing CreateTablet for tablet 93e8b973b39244be9de1107b14d6d93f (DEFAULT_TABLE table=default.Table_7ed5ee38ed7a4cf1a5a706eb6e233d41 [id=3bf292438c2749649a155a8b2332cd3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.231292 9613 tablet_service.cc:1505] Processing CreateTablet for tablet c35e5e38b9ab4bb88efa30fab9a9f9d3 (DEFAULT_TABLE table=default.Table_9e0da1160aff4c58887dbcf3ac217299 [id=7bdc02392ba542c7be65a5de5b0bdf8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.231444 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93e8b973b39244be9de1107b14d6d93f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.231582 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c35e5e38b9ab4bb88efa30fab9a9f9d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.235327 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b162734c536a40c08755201a85255f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.238885 11040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: Bootstrap complete.
I20251027 21:08:06.238940 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 218d244dc8a549408863110fefbc7b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.239686 9920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaa8910d9fdb4146945621ec4ebf969e" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:06.239892 9920 raft_consensus.cc:3060] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:06.241312 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fde4fd91943498d8888d1783005e492" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:06.241446 9925 raft_consensus.cc:3060] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:06.243916 9919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94f2819dd0394530ae93754bdd1e482f" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:06.244059 9919 raft_consensus.cc:2468] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:06.246577 9911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67bb12de9f044c73ae9ab26d10498c90" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:06.246760 9911 raft_consensus.cc:3060] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:06.247031 9711 leader_election.cc:343] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: c9efffa9766e46aca8eb286fc2ad48d2
I20251027 21:08:06.247009 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "980eb6755aee4bcfb2ea44d84009021f" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:06.247164 9909 raft_consensus.cc:3060] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:06.247233 9711 leader_election.cc:343] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: c9efffa9766e46aca8eb286fc2ad48d2
I20251027 21:08:06.248167 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 218d244dc8a549408863110fefbc7b20 (DEFAULT_TABLE table=default.Table_6b527fc023dd45f4ab080eddeeb8bb76 [id=5888beb9284242a8bf5a7c8f2649d62c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.248528 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 218d244dc8a549408863110fefbc7b20. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:06.249069 9568 leader_election.cc:343] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: bbb4116d1edd492893adfbf5fb3ddeef
I20251027 21:08:06.249167 9568 leader_election.cc:304] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:06.249584 9568 consensus_peers.cc:597] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4debc3f13f3a42939b6b83f6523aab7c. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:06.251674 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 60683685808643bcaf5845870d3f783d (DEFAULT_TABLE table=default.Table_3e61b43276ac41a2b626158c42642384 [id=2b6c44bf73684792a987f15a61c00a19]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.251983 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60683685808643bcaf5845870d3f783d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.254468 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 53c13ebc2e3f46e996f521f234c5b015 (DEFAULT_TABLE table=default.Table_12e5e752397f4aadae74519cfdf8f4d7 [id=3288cd17e43d44569d222b2c55676aaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.254755 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53c13ebc2e3f46e996f521f234c5b015. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.255481 11097 consensus_queue.cc:1048] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:06.255954 11097 raft_consensus.cc:2749] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:06.258076 9635 raft_consensus.cc:2468] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:06.260098 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 8b63d498ce8e4c3cafbefa650b657250 (DEFAULT_TABLE table=default.Table_0a1b25591a094198b8123b3ac2af4715 [id=4b0f47c0f5344b1bafcef5e50f2d6cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.260382 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b63d498ce8e4c3cafbefa650b657250. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.262354 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 07b8fd24b8af4f8db2a401c26cc65058 (DEFAULT_TABLE table=default.Table_fb6cfb8df03347efbf6dd0002e05bf86 [id=a9b801ae4df54e82a3748fb95f7d467d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.262475 9605 tablet_service.cc:1505] Processing CreateTablet for tablet fd511b6dd9544432bd4bec2822642906 (DEFAULT_TABLE table=default.Table_088cee7e137e418599e9dc3d73b42640 [id=a8254167aec04d8781120aabe34acca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.262631 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07b8fd24b8af4f8db2a401c26cc65058. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.262764 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd511b6dd9544432bd4bec2822642906. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.264707 9595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f954daa93d5a42bfabfbf8fd4b72172b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.268868 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 93e8b973b39244be9de1107b14d6d93f (DEFAULT_TABLE table=default.Table_7ed5ee38ed7a4cf1a5a706eb6e233d41 [id=3bf292438c2749649a155a8b2332cd3b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:06.271399 9554 kernel_stack_watchdog.cc:198] Thread 11141 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 341ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f1747b94980 (unknown) at ??:0
@ 0x93d797 __sanitizer::internal_sched_yield() at ??:?
@ 0x8a2e4d __interceptor_pthread_create at ??:?
@ 0x7f174457dbe1 kudu::Thread::StartThread() at ??:0
@ 0xa75107 kudu::Thread::Create() at ??:?
@ 0x7f174459da2b kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f174459944d kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f174459889f kudu::ThreadPoolToken::Submit() at ??:0
@ 0x7f175032dbe2 kudu::consensus::PeerMessageQueue::NotifyObserversOfPeerHealthChange() at ??:0
@ 0x7f175032d3e8 kudu::consensus::PeerMessageQueue::UpdatePeerHealthUnlocked() at ??:0
@ 0x7f175033e4cb kudu::consensus::PeerMessageQueue::RequestForPeer()::$_2::operator()() at ??:0
@ 0x7f1750332314 kudu::ScopedCleanup<>::~ScopedCleanup() at ??:0
@ 0x7f175032fbf0 kudu::consensus::PeerMessageQueue::RequestForPeer() at ??:0
@ 0x7f17502fca8c kudu::consensus::Peer::SendNextRequest() at ??:0
@ 0x7f1750307a22 kudu::consensus::Peer::SignalRequest()::$_2::operator()() at ??:0
@ 0x7f175030770e std::_Function_handler<>::_M_invoke() at ??:0
I20251027 21:08:06.273293 9602 tablet_service.cc:1505] Processing CreateTablet for tablet cbd77880ad934843b659d7b669eaafc7 (DEFAULT_TABLE table=default.Table_27e2b6a6c7684a7baefcca99d2f93e87 [id=f0ea1e2e6c04444ab1af5dab52fb98ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.273612 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd77880ad934843b659d7b669eaafc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.277738 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 83f5fbf5757a49df93ab949b7657c61f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:06.277921 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 8649a3b611cc48ff8832c0fdeb128efe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:05 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:06.278028 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f1d1bd52556463fa306e51eb253af40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:05 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:06.278118 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet cef2bfee222f45e6b6f20c6d63ff44d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:06.278401 9603 tablet_service.cc:1505] Processing CreateTablet for tablet 933fb952d66d4cb1ade70c334fbbb3e8 (DEFAULT_TABLE table=default.Table_f0f82afba21846a89a2fa83897fde34f [id=5209bcda10e64660be987d47e95a738c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.278707 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 933fb952d66d4cb1ade70c334fbbb3e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.282037 9604 tablet_service.cc:1505] Processing CreateTablet for tablet b8c368698cfb40799571567e8bfdd8dc (DEFAULT_TABLE table=default.Table_7057cb41215c437391bce90bc612d1df [id=f6000ae9faf749e28606aa7fd092174e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.282375 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8c368698cfb40799571567e8bfdd8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.285972 9598 tablet_service.cc:1505] Processing CreateTablet for tablet 5611e840dffc4ce6a17e2530fe78fed8 (DEFAULT_TABLE table=default.Table_baa19560a48f42c7ab0ecad353eaa283 [id=620c56eff478436185b0a178580a4548]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.286279 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5611e840dffc4ce6a17e2530fe78fed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.289497 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93e8b973b39244be9de1107b14d6d93f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.291142 11040 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:08:06.292241 11040 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e86ca5e83dd4df5b72f045e7b787e81, State: Initialized, Role: FOLLOWER
I20251027 21:08:06.271978 9925 raft_consensus.cc:2468] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:06.272748 9901 tablet_service.cc:1505] Processing CreateTablet for tablet 8b63d498ce8e4c3cafbefa650b657250 (DEFAULT_TABLE table=default.Table_0a1b25591a094198b8123b3ac2af4715 [id=4b0f47c0f5344b1bafcef5e50f2d6cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.292752 9901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b63d498ce8e4c3cafbefa650b657250. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.272748 9896 tablet_service.cc:1505] Processing CreateTablet for tablet c35e5e38b9ab4bb88efa30fab9a9f9d3 (DEFAULT_TABLE table=default.Table_9e0da1160aff4c58887dbcf3ac217299 [id=7bdc02392ba542c7be65a5de5b0bdf8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.273348 9887 tablet_service.cc:1505] Processing CreateTablet for tablet fd511b6dd9544432bd4bec2822642906 (DEFAULT_TABLE table=default.Table_088cee7e137e418599e9dc3d73b42640 [id=a8254167aec04d8781120aabe34acca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.273378 9902 tablet_service.cc:1505] Processing CreateTablet for tablet cbd77880ad934843b659d7b669eaafc7 (DEFAULT_TABLE table=default.Table_27e2b6a6c7684a7baefcca99d2f93e87 [id=f0ea1e2e6c04444ab1af5dab52fb98ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.273628 9889 tablet_service.cc:1552] Processing DeleteTablet for tablet 83f5fbf5757a49df93ab949b7657c61f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:06.293800 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c35e5e38b9ab4bb88efa30fab9a9f9d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.294072 9889 tablet_service.cc:1505] Processing CreateTablet for tablet cddb9e2e3a914c8ab4515cf4505333ee (DEFAULT_TABLE table=default.Table_f22296338b56467193b136c872ba30c4 [id=6fd72bf290b34763b0eefcd420ac3894]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.294379 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cddb9e2e3a914c8ab4515cf4505333ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.297353 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 60f4872159c54d7ebcf7987704018928 (DEFAULT_TABLE table=default.Table_34bd7efc93714dca9c8bf72307608cf1 [id=82bb4517750f4f42ae6f26ef5b9feafa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.273828 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 8649a3b611cc48ff8832c0fdeb128efe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:05 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:06.297673 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60f4872159c54d7ebcf7987704018928. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.297721 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 824ff6913b0142d5889c75aa8e3cba3e (DEFAULT_TABLE table=default.Table_152a2bed589f47858b8a99ab7df9984c [id=c1fcc453aee04cbca24ccb8bdfa3ba00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.298025 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 824ff6913b0142d5889c75aa8e3cba3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.298748 9710 leader_election.cc:304] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:06.299012 10997 raft_consensus.cc:2804] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:06.299105 10997 raft_consensus.cc:697] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:06.299316 10997 consensus_queue.cc:237] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:06.274019 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f1d1bd52556463fa306e51eb253af40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:05 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:06.301179 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 8ab9310bba1f425fbee6595e0182aae9 (DEFAULT_TABLE table=default.Table_7ff9f51c77a1418faf1e35ad182c91d7 [id=51b8c67b97dd461791faaa2cc6d9e5a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.301452 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ab9310bba1f425fbee6595e0182aae9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:06.302711 9298 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 699c6a8dd8d44d729df463b20cdcfdda (table default.Table_24fe105409a8463fa475647295017db9 [id=6e17bac6ca06444aa464cb19795ed793], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 699c6a8dd8d44d729df463b20cdcfdda consensus state has no leader
I20251027 21:08:06.302657 11040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 782, Last appended: 3.782, Last appended by leader: 782, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:08:06.304415 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd511b6dd9544432bd4bec2822642906. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.274220 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 915) took 2671 ms. Trace:
I20251027 21:08:06.307471 10041 rpcz_store.cc:276] 1027 21:08:03.602918 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:03.602996 (+ 78us) service_pool.cc:225] Handling call
1027 21:08:05.540113 (+1937117us) ts_tablet_manager.cc:1151] Deleting tablet a4d26f73b8534f08bcaa386d4de75c01
1027 21:08:05.540121 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:05.711040 (+170919us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:05.955247 (+244207us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.274205 (+318958us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1936962}
I20251027 21:08:06.307788 10041 tablet_replica.cc:333] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:06.307958 10041 raft_consensus.cc:2243] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:06.308069 10041 raft_consensus.cc:2272] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:06.308192 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07b8fd24b8af4f8db2a401c26cc65058. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.308387 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53c13ebc2e3f46e996f521f234c5b015. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.308516 10041 ts_tablet_manager.cc:1916] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:06.274362 9886 tablet_service.cc:1505] Processing CreateTablet for tablet 933fb952d66d4cb1ade70c334fbbb3e8 (DEFAULT_TABLE table=default.Table_f0f82afba21846a89a2fa83897fde34f [id=5209bcda10e64660be987d47e95a738c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.310432 9886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 933fb952d66d4cb1ade70c334fbbb3e8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:06.308073 9568 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 58365.9us
I20251027 21:08:06.311932 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60683685808643bcaf5845870d3f783d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:06.312142 9568 leader_election.cc:343] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 2a4aa1fe05114c1287ca580c1f411d3f
I20251027 21:08:06.312232 9568 leader_election.cc:304] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:06.312408 11097 raft_consensus.cc:2749] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:06.312714 9568 leader_election.cc:343] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 59b6818d29624da5a49dd1a7d20994c6
I20251027 21:08:06.312801 9568 leader_election.cc:304] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:06.312959 11097 raft_consensus.cc:2749] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:06.313225 9568 leader_election.cc:304] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:06.313385 11097 raft_consensus.cc:2804] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:06.313431 11097 raft_consensus.cc:493] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:06.313494 11097 raft_consensus.cc:3060] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:06.274729 9900 tablet_service.cc:1505] Processing CreateTablet for tablet b8c368698cfb40799571567e8bfdd8dc (DEFAULT_TABLE table=default.Table_7057cb41215c437391bce90bc612d1df [id=f6000ae9faf749e28606aa7fd092174e]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:06.314006 9568 leader_election.cc:343] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: cef2bfee222f45e6b6f20c6d63ff44d4
I20251027 21:08:06.314086 9568 leader_election.cc:304] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:06.314167 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8c368698cfb40799571567e8bfdd8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.314383 11018 raft_consensus.cc:2749] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:06.315132 11018 consensus_queue.cc:1048] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:06.315774 9568 leader_election.cc:304] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:06.315966 11018 raft_consensus.cc:2804] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:06.316170 11018 raft_consensus.cc:697] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:06.316388 11018 consensus_queue.cc:237] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.275113 9903 tablet_service.cc:1505] Processing CreateTablet for tablet 5611e840dffc4ce6a17e2530fe78fed8 (DEFAULT_TABLE table=default.Table_baa19560a48f42c7ab0ecad353eaa283 [id=620c56eff478436185b0a178580a4548]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.317291 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5611e840dffc4ce6a17e2530fe78fed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.275144 9894 tablet_service.cc:1505] Processing CreateTablet for tablet 60f4872159c54d7ebcf7987704018928 (DEFAULT_TABLE table=default.Table_34bd7efc93714dca9c8bf72307608cf1 [id=82bb4517750f4f42ae6f26ef5b9feafa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.318367 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60f4872159c54d7ebcf7987704018928. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.275542 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 473b7f84dfcd4f1cb89afcfeaab33b34 (DEFAULT_TABLE table=default.Table_4f5e79447c4b4bab8c8c12cf2adab418 [id=277e5426ed8f40abb73ca9fa3d86faca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.275933 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 3ef2fff1381b465a89eeb5f82ae4fd12 (DEFAULT_TABLE table=default.Table_685b53b410ef4403890d1494d6d77042 [id=5cd248bd134f425ea9fd20f3c483db51]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.281665 11065 raft_consensus.cc:493] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:06.282420 10032 tablet_bootstrap.cc:492] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:06.322543 11065 raft_consensus.cc:515] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.323129 11065 leader_election.cc:290] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:06.323851 10032 tablet_bootstrap.cc:654] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:06.324496 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f6789dc2f7cb4a3c826ac507d5c9a239" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:06.324954 9860 leader_election.cc:343] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:06.326033 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 473b7f84dfcd4f1cb89afcfeaab33b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.326627 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef2fff1381b465a89eeb5f82ae4fd12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.337183 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet a4d26f73b8534f08bcaa386d4de75c01 (table default.Table_a6cf2819ed9241cea09738477423c84a [id=f5ef094809e24355a7e021408e2ddd20]) successfully deleted
I20251027 21:08:06.337388 10032 tablet_bootstrap.cc:492] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:06.337519 10032 ts_tablet_manager.cc:1403] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.004s
I20251027 21:08:06.338485 10032 raft_consensus.cc:359] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.338603 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f6789dc2f7cb4a3c826ac507d5c9a239" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:06.338689 10032 raft_consensus.cc:385] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:06.338749 10032 raft_consensus.cc:740] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:06.338992 10032 consensus_queue.cc:260] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:06.339061 9859 leader_election.cc:343] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:06.339143 9859 leader_election.cc:304] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:06.339424 10032 ts_tablet_manager.cc:1434] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:06.339512 11065 raft_consensus.cc:2749] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:06.339673 10032 tablet_bootstrap.cc:492] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:06.341809 10032 tablet_bootstrap.cc:654] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:06.350900 10032 tablet_bootstrap.cc:492] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:06.351035 10032 ts_tablet_manager.cc:1403] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20251027 21:08:06.351861 10032 raft_consensus.cc:359] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.352043 10032 raft_consensus.cc:385] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:06.352097 10032 raft_consensus.cc:740] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:06.359138 10032 consensus_queue.cc:260] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.359771 10032 ts_tablet_manager.cc:1434] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251027 21:08:06.360070 10032 tablet_bootstrap.cc:492] T 882c58a45d46402eabe96591a7c1de71 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:06.360451 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd77880ad934843b659d7b669eaafc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.361665 10032 tablet_bootstrap.cc:654] T 882c58a45d46402eabe96591a7c1de71 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:06.362350 9911 raft_consensus.cc:2468] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:06.367889 9711 leader_election.cc:304] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218; no voters: 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:06.368263 10829 raft_consensus.cc:2804] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:06.371999 10032 tablet_bootstrap.cc:492] T 882c58a45d46402eabe96591a7c1de71 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:06.372115 10032 ts_tablet_manager.cc:1403] T 882c58a45d46402eabe96591a7c1de71 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20251027 21:08:06.373347 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 473b7f84dfcd4f1cb89afcfeaab33b34 (DEFAULT_TABLE table=default.Table_4f5e79447c4b4bab8c8c12cf2adab418 [id=277e5426ed8f40abb73ca9fa3d86faca]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:06.375100 11141 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.055s sys 0.113s
W20251027 21:08:06.375200 11141 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.602s user 0.055s sys 0.113s
I20251027 21:08:06.377660 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 3ef2fff1381b465a89eeb5f82ae4fd12 (DEFAULT_TABLE table=default.Table_685b53b410ef4403890d1494d6d77042 [id=5cd248bd134f425ea9fd20f3c483db51]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.381610 9608 tablet_service.cc:1505] Processing CreateTablet for tablet cddb9e2e3a914c8ab4515cf4505333ee (DEFAULT_TABLE table=default.Table_f22296338b56467193b136c872ba30c4 [id=6fd72bf290b34763b0eefcd420ac3894]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.382437 11160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:08:06.382786 11160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 21:08:06.384940 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 824ff6913b0142d5889c75aa8e3cba3e (DEFAULT_TABLE table=default.Table_152a2bed589f47858b8a99ab7df9984c [id=c1fcc453aee04cbca24ccb8bdfa3ba00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.388790 9611 tablet_service.cc:1505] Processing CreateTablet for tablet a24dbceaa95e4f43a5ea416a20e7041e (DEFAULT_TABLE table=default.Table_5a6211c8465040dbad1cc86377a370ea [id=91fb9f95fe9343fca1d8cb0d0f781afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.388818 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 8ab9310bba1f425fbee6595e0182aae9 (DEFAULT_TABLE table=default.Table_7ff9f51c77a1418faf1e35ad182c91d7 [id=51b8c67b97dd461791faaa2cc6d9e5a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.393929 10043 ts_tablet_manager.cc:1929] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:06.394027 10043 log.cc:1199] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/32f80614e30d41128f9fb2e0ea165223
I20251027 21:08:06.394320 10043 ts_tablet_manager.cc:1950] T 32f80614e30d41128f9fb2e0ea165223 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:06.401228 11040 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 21:08:06.403518 10032 raft_consensus.cc:359] T 882c58a45d46402eabe96591a7c1de71 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.403734 10032 raft_consensus.cc:385] T 882c58a45d46402eabe96591a7c1de71 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:06.403795 10032 raft_consensus.cc:740] T 882c58a45d46402eabe96591a7c1de71 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:06.404354 10032 consensus_queue.cc:260] T 882c58a45d46402eabe96591a7c1de71 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.404963 10032 ts_tablet_manager.cc:1434] T 882c58a45d46402eabe96591a7c1de71 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.033s user 0.001s sys 0.001s
I20251027 21:08:06.405269 10032 tablet_bootstrap.cc:492] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:06.410562 10032 tablet_bootstrap.cc:654] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:06.414639 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 473b7f84dfcd4f1cb89afcfeaab33b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.414848 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ab9310bba1f425fbee6595e0182aae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.415179 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef2fff1381b465a89eeb5f82ae4fd12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.415683 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cddb9e2e3a914c8ab4515cf4505333ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.416066 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a24dbceaa95e4f43a5ea416a20e7041e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.419472 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 824ff6913b0142d5889c75aa8e3cba3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.426263 10997 raft_consensus.cc:493] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:06.426389 10997 raft_consensus.cc:515] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:06.428716 9382 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.9.14.65:54623 (request call id 313) took 778 ms (client timeout 1000 ms). Trace:
W20251027 21:08:06.428871 9382 rpcz_store.cc:269] 1027 21:08:05.650531 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.937084 (+286553us) service_pool.cc:225] Handling call
1027 21:08:06.427988 (+490904us) spinlock_profiling.cc:243] Waited 490 ms on lock 0x616000041254. stack: 00007fcf0cae19f6 00007fcf0cae17b0 00007fcf0babd3ad 0000000000983f04 0000000000983ecd 00007fcf1bde86bd 00007fcf1bde87b3 00007fcf189af441 00007fcf206699dd 00007fcf207f0aa8 00007fcf17303ca6 00007fcf173039d5 00007fcf13a8027e 00007fcf13a7f15e 00007fcf13a85395 00007fcf13a85cbf
1027 21:08:06.428665 (+ 677us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1127291264}
I20251027 21:08:06.430874 11065 raft_consensus.cc:493] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:06.430990 11065 raft_consensus.cc:515] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.431259 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e9839f0986064565be4d73e81a8671da" } modify_external_catalogs: true
I20251027 21:08:06.431309 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b6554b3e96a841fb975ff74ff767069a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:06.431439 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e9839f0986064565be4d73e81a8671da" } modify_external_catalogs: true
I20251027 21:08:06.431569 11065 leader_election.cc:290] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:06.431885 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6554b3e96a841fb975ff74ff767069a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:06.432194 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2d26784bbb24ae3893103802c41affb" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:06.432269 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2d26784bbb24ae3893103802c41affb" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:06.432639 9859 leader_election.cc:343] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:06.432976 9860 leader_election.cc:343] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:06.433061 9860 leader_election.cc:304] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:06.433354 11065 raft_consensus.cc:493] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:06.433388 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_555cf66116b243de86d5cdd0daf42151" } new_table_name: "default.Table_f05b64422c2c441c996f4d8057c11a78" modify_external_catalogs: true
I20251027 21:08:06.433456 11065 raft_consensus.cc:515] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:06.433853 9307 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.9.14.65:32849 (request call id 179) took 844 ms (client timeout 1000 ms). Trace:
I20251027 21:08:06.433931 11065 leader_election.cc:290] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:06.434080 11065 raft_consensus.cc:2749] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:06.433969 9307 rpcz_store.cc:269] 1027 21:08:05.588958 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.592181 (+ 3223us) service_pool.cc:225] Handling call
1027 21:08:05.594433 (+ 2252us) spinlock_profiling.cc:243] Waited 1.72 ms on lock 0x616000051154. stack: 00007f956e0a89f6 00007f956e0a87b0 00007f956d0843ad 0000000000983f04 0000000000983ecd 00007f957d3af6bd 00007f957d3af7b3 00007f9579f76441 00007f9581c309dd 00007f9581db7aa8 00007f95788caca6 00007f95788ca9d5 00007f957504727e 00007f957504615e 00007f957504c395 00007f957504ccbf
1027 21:08:06.433795 (+839362us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":3,"spinlock_wait_cycles":3949952}
I20251027 21:08:06.434366 10997 leader_election.cc:290] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
W20251027 21:08:06.434707 9711 leader_election.cc:343] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: e9edb62a8e4b4dfea6a31a2202545a6e
I20251027 21:08:06.434474 10997 raft_consensus.cc:493] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:06.434908 10997 raft_consensus.cc:515] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.435413 9310 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 6600774e4c7b48479a2fdb02614ba505 (table default.Table_d7530773668f42c5a07a921df02a95ab [id=914b728e358e4fdba6d38d6c7beadaca]). Expected version 1 got 0
I20251027 21:08:06.435540 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "420ccfaaa7f24ad0aea9fdfea24fc5dd" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:06.435870 9859 leader_election.cc:343] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:06.436190 9711 leader_election.cc:343] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 32f80614e30d41128f9fb2e0ea165223
I20251027 21:08:06.436620 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e49807b03fc64f669376118d2ee01918" } modify_external_catalogs: true
I20251027 21:08:06.436834 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e49807b03fc64f669376118d2ee01918" } modify_external_catalogs: true
I20251027 21:08:06.436947 10859 raft_consensus.cc:493] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:06.437074 10859 raft_consensus.cc:515] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:06.437532 10859 leader_election.cc:290] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:06.433284 9312 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_203191e192be4e3a9a7477b4313ef4f4 [id=5da6bd4463e24c03accd9985f72b6ebb] alter complete (version 3)
W20251027 21:08:06.441396 9312 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.9.14.67:53829 (request call id 202) took 1163 ms (client timeout 1000 ms). Trace:
W20251027 21:08:06.441505 9312 rpcz_store.cc:269] 1027 21:08:05.277877 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.475638 (+197761us) service_pool.cc:225] Handling call
1027 21:08:06.441336 (+965698us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:08:05.598934 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:08:05.599023 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:08:05.599028 (+ 5us) write_op.cc:435] Acquired schema lock
1027 21:08:05.599030 (+ 2us) tablet.cc:660] Decoding operations
1027 21:08:05.599098 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:08:05.599105 (+ 7us) write_op.cc:641] Partition lock acquired for write op
1027 21:08:05.599108 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
1027 21:08:05.599148 (+ 40us) tablet.cc:699] Row locks acquired
1027 21:08:05.599150 (+ 2us) write_op.cc:260] PREPARE: finished
1027 21:08:05.599173 (+ 23us) write_op.cc:270] Start()
1027 21:08:05.599195 (+ 22us) write_op.cc:276] Timestamp: P: 1761599285593504 usec, L: 0
1027 21:08:05.599198 (+ 3us) op_driver.cc:348] REPLICATION: starting
1027 21:08:05.599322 (+ 124us) log.cc:844] Serialized 356 byte log entry
1027 21:08:06.430654 (+831332us) op_driver.cc:464] REPLICATION: finished
1027 21:08:06.432740 (+ 2086us) write_op.cc:301] APPLY: starting
1027 21:08:06.432757 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1027 21:08:06.432798 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
1027 21:08:06.432847 (+ 49us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:08:06.432916 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:08:06.432948 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:08:06.432955 (+ 7us) write_op.cc:312] APPLY: finished
1027 21:08:06.433032 (+ 77us) log.cc:844] Serialized 23 byte log entry
1027 21:08:06.433094 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:06.433109 (+ 15us) write_op.cc:454] Released schema lock
1027 21:08:06.433115 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1027 21:08:06.433414 (+ 299us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:06.433421 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7661,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":1989,"apply.run_cpu_time_us":469,"apply.run_wall_time_us":670,"num_ops":1,"prepare.queue_time_us":27464,"prepare.run_cpu_time_us":434,"prepare.run_wall_time_us":433,"replication_time_us":831422,"spinlock_wait_cycles":384}]]}
W20251027 21:08:06.445129 9568 leader_election.cc:343] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 01e9db9eaf8e41c6a2ee13b9ed127421
I20251027 21:08:06.447219 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01e9db9eaf8e41c6a2ee13b9ed127421" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:06.447849 9566 leader_election.cc:343] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:06.448045 9566 leader_election.cc:304] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:06.449664 11018 raft_consensus.cc:2749] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:06.450228 10997 leader_election.cc:290] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:06.450685 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "420ccfaaa7f24ad0aea9fdfea24fc5dd" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:06.451696 9860 leader_election.cc:343] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:06.451784 9860 leader_election.cc:304] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:06.452221 11065 raft_consensus.cc:2749] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:06.453356 9735 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:40722 (request call id 150) took 1428 ms. Trace:
I20251027 21:08:06.454206 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3de9fdb7f17a46e0a0560f82afaef828" } modify_external_catalogs: true
I20251027 21:08:06.454424 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3de9fdb7f17a46e0a0560f82afaef828" } modify_external_catalogs: true
I20251027 21:08:06.458344 11097 raft_consensus.cc:515] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.458853 11097 leader_election.cc:290] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:06.460628 9612 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1000) took 1001 ms. Trace:
I20251027 21:08:06.469727 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94f2819dd0394530ae93754bdd1e482f" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:06.461959 9909 raft_consensus.cc:2468] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:06.461597 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c548d2413a5940c8b114ced667b4dc41
I20251027 21:08:06.625265 9303 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:41710 (request call id 1233) took 1129 ms. Trace:
I20251027 21:08:06.625380 9303 rpcz_store.cc:276] 1027 21:08:05.496047 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.594916 (+ 98869us) service_pool.cc:225] Handling call
1027 21:08:05.595379 (+ 463us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1027 21:08:05.595551 (+ 172us) catalog_manager.cc:2815] Modifying in-memory table state
1027 21:08:05.595637 (+ 86us) catalog_manager.cc:2824] Locking tablets
1027 21:08:05.595800 (+ 163us) catalog_manager.cc:2838] Removing table and tablets from system table
1027 21:08:06.453355 (+857555us) catalog_manager.cc:2856] Removing table from by-name map
1027 21:08:06.453471 (+ 116us) catalog_manager.cc:2872] Committing in-memory state
1027 21:08:06.453517 (+ 46us) catalog_manager.cc:2876] Aborting table tasks
1027 21:08:06.625229 (+171712us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:08:05.710403 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:08:05.710513 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:08:05.710520 (+ 7us) write_op.cc:435] Acquired schema lock
1027 21:08:05.710522 (+ 2us) tablet.cc:660] Decoding operations
1027 21:08:05.710625 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:08:05.710634 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1027 21:08:05.710638 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:08:05.710771 (+ 133us) tablet.cc:699] Row locks acquired
1027 21:08:05.710773 (+ 2us) write_op.cc:260] PREPARE: finished
1027 21:08:05.710811 (+ 38us) write_op.cc:270] Start()
1027 21:08:05.710857 (+ 46us) write_op.cc:276] Timestamp: P: 1761599285705133 usec, L: 0
1027 21:08:05.710873 (+ 16us) op_driver.cc:348] REPLICATION: starting
1027 21:08:05.711053 (+ 180us) log.cc:844] Serialized 725 byte log entry
1027 21:08:06.451093 (+740040us) op_driver.cc:464] REPLICATION: finished
1027 21:08:06.451810 (+ 717us) write_op.cc:301] APPLY: starting
1027 21:08:06.451820 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1027 21:08:06.451860 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
1027 21:08:06.451880 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:08:06.451934 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:08:06.451967 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1027 21:08:06.451973 (+ 6us) write_op.cc:312] APPLY: finished
1027 21:08:06.452025 (+ 52us) log.cc:844] Serialized 29 byte log entry
1027 21:08:06.452065 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:06.452080 (+ 15us) write_op.cc:454] Released schema lock
1027 21:08:06.452086 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1027 21:08:06.452162 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:06.452166 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":8004,"child_traces":[["op",{"apply.queue_time_us":688,"apply.run_cpu_time_us":332,"apply.run_wall_time_us":331,"num_ops":2,"prepare.queue_time_us":113988,"prepare.run_cpu_time_us":720,"prepare.run_wall_time_us":973,"replication_time_us":740204}]]}
I20251027 21:08:06.626238 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b02344913e414bada0cc6b5503d42c02" } new_table_name: "default.Table_cdf514bc48a4478da802d1bbfd0f1c71" modify_external_catalogs: true
I20251027 21:08:06.460702 9612 rpcz_store.cc:276] 1027 21:08:05.459334 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.230807 (+771473us) service_pool.cc:225] Handling call
1027 21:08:06.231292 (+ 485us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.231319 (+ 27us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.235135 (+ 3816us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.460439 (+225304us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.460583 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.627700 9612 tablet_service.cc:1505] Processing CreateTablet for tablet 69f1738d936741f792778113b15e57f7 (DEFAULT_TABLE table=default.Table_4a3918c0e2224ae8b75c099bd4332ca0 [id=fde53cb33c144f969adecb272bbb579e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.628038 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f1738d936741f792778113b15e57f7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:06.469830 9710 leader_election.cc:343] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: e9edb62a8e4b4dfea6a31a2202545a6e
I20251027 21:08:06.453433 9735 rpcz_store.cc:276] 1027 21:08:05.024974 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.025023 (+ 49us) service_pool.cc:225] Handling call
1027 21:08:05.082340 (+ 57317us) spinlock_profiling.cc:243] Waited 51.3 ms on lock 0x616000135454. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d8fee1200 00007f8d93d9166f 00007f8d8e1b7c36 00007f8d8e1b7965 00007f8d8afe627e 00007f8d8afe515e 00007f8d8afeb395 00007f8d8afebcbf 00007f8d8afebae8 0000000000a8867c
1027 21:08:05.217903 (+135563us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x6160001b1654. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d8fee1200 00007f8d93d9166f 00007f8d8e1b7c36 00007f8d8e1b7965 00007f8d8afe627e 00007f8d8afe515e 00007f8d8afeb395 00007f8d8afebcbf 00007f8d8afebae8 0000000000a8867c
1027 21:08:05.526076 (+308173us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x6160001b1c54. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d8fee1200 00007f8d93d9166f 00007f8d8e1b7c36 00007f8d8e1b7965 00007f8d8afe627e 00007f8d8afe515e 00007f8d8afeb395 00007f8d8afebcbf 00007f8d8afebae8 0000000000a8867c
1027 21:08:06.089001 (+562925us) spinlock_profiling.cc:243] Waited 541 ms on lock 0x6160000d6f54. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d8fee1200 00007f8d93d9166f 00007f8d8e1b7c36 00007f8d8e1b7965 00007f8d8afe627e 00007f8d8afe515e 00007f8d8afeb395 00007f8d8afebcbf 00007f8d8afebae8 0000000000a8867c
1027 21:08:06.453336 (+364335us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2379192960}
I20251027 21:08:06.630247 9710 leader_election.cc:304] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
W20251027 21:08:06.630563 9710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.65:44531, user_credentials={real_user=slave}} blocked reactor thread for 160751us
I20251027 21:08:06.462426 10041 ts_tablet_manager.cc:1929] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:06.630870 10041 log.cc:1199] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/a033551a591146039b017b4193b221ca
I20251027 21:08:06.631165 10041 ts_tablet_manager.cc:1950] T a033551a591146039b017b4193b221ca P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:06.490916 9736 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 973) took 1042 ms. Trace:
I20251027 21:08:06.631422 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 914) took 3028 ms. Trace:
I20251027 21:08:06.631422 9736 rpcz_store.cc:276] 1027 21:08:05.448464 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.889315 (+440851us) service_pool.cc:225] Handling call
1027 21:08:05.889746 (+ 431us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:05.889771 (+ 25us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.120794 (+231023us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.490746 (+369952us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.490899 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21}
I20251027 21:08:06.631484 10041 rpcz_store.cc:276] 1027 21:08:03.602811 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:03.603560 (+ 749us) service_pool.cc:225] Handling call
1027 21:08:06.307751 (+2704191us) ts_tablet_manager.cc:1151] Deleting tablet a033551a591146039b017b4193b221ca
1027 21:08:06.307759 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:06.360404 (+ 52645us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.462336 (+101932us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.631410 (+169074us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2704054}
I20251027 21:08:06.631798 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 913) took 3029 ms. Trace:
I20251027 21:08:06.631855 10041 rpcz_store.cc:276] 1027 21:08:03.602714 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:03.604998 (+ 2284us) service_pool.cc:225] Handling call
1027 21:08:06.631701 (+3026703us) ts_tablet_manager.cc:1151] Deleting tablet b99232a6018a4b1a850da954c551d38d
1027 21:08:06.631708 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:06.631786 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3026578}
I20251027 21:08:06.631881 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 7057c55a820d4e379fe1d9abfa353821 (DEFAULT_TABLE table=default.Table_f396e1e1ebc44cf9a774f6523942ff0b [id=633dfb2a421c44b2833584fcabc6e475]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.632093 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 918) took 3010 ms. Trace:
I20251027 21:08:06.632192 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7057c55a820d4e379fe1d9abfa353821. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.632144 10041 rpcz_store.cc:276] 1027 21:08:03.621983 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:03.622048 (+ 65us) service_pool.cc:225] Handling call
1027 21:08:06.632022 (+3009974us) ts_tablet_manager.cc:1151] Deleting tablet b748f59a56c54c2b850f5336a1d61c26
1027 21:08:06.632029 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:06.632084 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3009814}
I20251027 21:08:06.632349 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 917) took 3013 ms. Trace:
I20251027 21:08:06.632396 10041 rpcz_store.cc:276] 1027 21:08:03.619252 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:03.624261 (+ 5009us) service_pool.cc:225] Handling call
1027 21:08:06.632286 (+3008025us) ts_tablet_manager.cc:1151] Deleting tablet 00c90ff84d3f4a7f92fda235416eed1d
1027 21:08:06.632293 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:06.632339 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3007419}
I20251027 21:08:06.632567 10041 tablet_replica.cc:333] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:06.632726 10041 raft_consensus.cc:2243] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:06.632841 10041 raft_consensus.cc:2272] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:06.633291 10041 ts_tablet_manager.cc:1916] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:06.461709 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f0f82afba21846a89a2fa83897fde34f" } modify_external_catalogs: true
I20251027 21:08:06.635121 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f0f82afba21846a89a2fa83897fde34f" } modify_external_catalogs: true
I20251027 21:08:06.636749 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet a033551a591146039b017b4193b221ca (table default.Table_7a01697cdee04a929068e3297b598128 [id=84d2047a211e479e89130346cedd9fdf]) successfully deleted
W20251027 21:08:06.637013 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet b99232a6018a4b1a850da954c551d38d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b99232a6018a4b1a850da954c551d38d already in progress: creating tablet
W20251027 21:08:06.637213 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet b748f59a56c54c2b850f5336a1d61c26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b748f59a56c54c2b850f5336a1d61c26 already in progress: creating tablet
W20251027 21:08:06.637401 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 00c90ff84d3f4a7f92fda235416eed1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00c90ff84d3f4a7f92fda235416eed1d already in progress: creating tablet
I20251027 21:08:06.637966 9736 tablet_service.cc:1505] Processing CreateTablet for tablet c76870d7ca1b43308f0ab94d4ab39b5f (DEFAULT_TABLE table=default.Table_645e7862b8ef4097aa7152d5984b7fb7 [id=805d6b3b37ab43358f8d46185f7c9ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.493185 9747 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 978) took 1039 ms. Trace:
I20251027 21:08:06.463205 9610 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 999) took 1006 ms. Trace:
I20251027 21:08:06.493268 9742 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 977) took 1040 ms. Trace:
I20251027 21:08:06.639518 9747 rpcz_store.cc:276] 1027 21:08:05.453404 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.917287 (+463883us) service_pool.cc:225] Handling call
1027 21:08:05.917656 (+ 369us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:05.917678 (+ 22us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.139754 (+222076us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.493046 (+353292us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.493171 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.639551 9610 rpcz_store.cc:276] 1027 21:08:05.457260 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.109036 (+651776us) service_pool.cc:225] Handling call
1027 21:08:06.249968 (+140932us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.250003 (+ 35us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.310999 (+ 60996us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.463083 (+152084us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.463192 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.639889 9747 tablet_service.cc:1505] Processing CreateTablet for tablet 0d0032dd868c4dc79ea87036264434ee (DEFAULT_TABLE table=default.Table_5b5950cb52f34415b3a217c9c772bb3b [id=a6e66d2764ea438188e3d7992f1fd7a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.640026 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 278d8f448ef144148cabfc3a57bd24fd (DEFAULT_TABLE table=default.Table_69ef4f8f04d042a18d248c08958ee9f7 [id=983747413ebc4e3ea8ccb0eac5ffff08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.640161 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0032dd868c4dc79ea87036264434ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.640396 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 278d8f448ef144148cabfc3a57bd24fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.639546 9742 rpcz_store.cc:276] 1027 21:08:05.452524 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.915110 (+462586us) service_pool.cc:225] Handling call
1027 21:08:05.915477 (+ 367us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:05.915496 (+ 19us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.135821 (+220325us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.493046 (+357225us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.493259 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":101}
I20251027 21:08:06.641199 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 6c05b69509414fe8bfc809cd307d5f2e (DEFAULT_TABLE table=default.Table_c9a81e9dabad4a7d9fbd427439a7587c [id=40ce2a0befe54d698759beff3bfdf0f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.641460 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c05b69509414fe8bfc809cd307d5f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.642370 9568 leader_election.cc:304] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:06.642761 11169 raft_consensus.cc:2749] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:06.642992 9566 leader_election.cc:343] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:06.464798 9907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94f2819dd0394530ae93754bdd1e482f" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:06.483460 10032 tablet_bootstrap.cc:492] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:06.643196 9907 raft_consensus.cc:3060] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:06.643244 10032 ts_tablet_manager.cc:1403] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.238s user 0.004s sys 0.000s
I20251027 21:08:06.643939 10032 raft_consensus.cc:359] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.644064 10032 raft_consensus.cc:385] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:06.644102 10032 raft_consensus.cc:740] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:06.644256 10032 consensus_queue.cc:260] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.644632 10032 ts_tablet_manager.cc:1434] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:06.644881 10032 tablet_bootstrap.cc:492] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:06.462736 9306 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41710 (request call id 1230) took 1241 ms. Trace:
I20251027 21:08:06.494112 9753 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 979) took 1039 ms. Trace:
I20251027 21:08:06.500636 9737 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 984) took 1039 ms. Trace:
I20251027 21:08:06.559442 9920 raft_consensus.cc:2468] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:06.645654 9306 rpcz_store.cc:276] 1027 21:08:05.220783 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.567044 (+346261us) service_pool.cc:225] Handling call
1027 21:08:05.567998 (+ 954us) catalog_manager.cc:2101] Acquired catalog manager lock
1027 21:08:05.568767 (+ 769us) catalog_manager.cc:2147] Created new table and tablet info
1027 21:08:06.462521 (+893754us) catalog_manager.cc:2207] Wrote table and tablets to system table
1027 21:08:06.462602 (+ 81us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1027 21:08:06.462672 (+ 70us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:08:05.599420 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:08:05.599498 (+ 78us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:08:05.599502 (+ 4us) write_op.cc:435] Acquired schema lock
1027 21:08:05.599504 (+ 2us) tablet.cc:660] Decoding operations
1027 21:08:05.599571 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:08:05.599579 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1027 21:08:05.599583 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:08:05.599652 (+ 69us) tablet.cc:699] Row locks acquired
1027 21:08:05.599655 (+ 3us) write_op.cc:260] PREPARE: finished
1027 21:08:05.599679 (+ 24us) write_op.cc:270] Start()
1027 21:08:05.599703 (+ 24us) write_op.cc:276] Timestamp: P: 1761599285594009 usec, L: 0
1027 21:08:05.599707 (+ 4us) op_driver.cc:348] REPLICATION: starting
1027 21:08:05.709903 (+110196us) log.cc:844] Serialized 433 byte log entry
1027 21:08:06.451031 (+741128us) op_driver.cc:464] REPLICATION: finished
1027 21:08:06.451259 (+ 228us) write_op.cc:301] APPLY: starting
1027 21:08:06.451273 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1027 21:08:06.451313 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
1027 21:08:06.451352 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:08:06.451458 (+ 106us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:08:06.451487 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1027 21:08:06.451495 (+ 8us) write_op.cc:312] APPLY: finished
1027 21:08:06.451586 (+ 91us) log.cc:844] Serialized 29 byte log entry
1027 21:08:06.451662 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:06.451680 (+ 18us) write_op.cc:454] Released schema lock
1027 21:08:06.451687 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1027 21:08:06.451748 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:06.451752 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":184,"apply.run_cpu_time_us":481,"apply.run_wall_time_us":481,"num_ops":2,"prepare.queue_time_us":27553,"prepare.run_cpu_time_us":674,"prepare.run_wall_time_us":110934,"replication_time_us":851298,"spinlock_wait_cycles":1428864}]]}
I20251027 21:08:06.646090 9753 rpcz_store.cc:276] 1027 21:08:05.454376 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.918965 (+464589us) service_pool.cc:225] Handling call
1027 21:08:05.919379 (+ 414us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:05.919397 (+ 18us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.139303 (+219906us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.490746 (+351443us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.494090 (+ 3344us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3147}
I20251027 21:08:06.502400 9740 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 983) took 1042 ms. Trace:
I20251027 21:08:06.646328 9740 rpcz_store.cc:276] 1027 21:08:05.459839 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.014847 (+555008us) service_pool.cc:225] Handling call
1027 21:08:06.015181 (+ 334us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.015198 (+ 17us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.127552 (+112354us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.502279 (+374727us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.502387 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.646409 10032 tablet_bootstrap.cc:654] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:06.646484 9753 tablet_service.cc:1505] Processing CreateTablet for tablet eef7d8549ecc4546bc03c8e600c54c49 (DEFAULT_TABLE table=default.Table_9193c70ec7c84ce4829d242e054520a4 [id=9f6872a0c18a461e9d82f1735fd8cf60]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.646481 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f7947222dc5043359f1891191346329b" } new_table_name: "default.Table_daa7cbb7c0864fa1aeca3cbe911f96c2" modify_external_catalogs: true
I20251027 21:08:06.646653 9740 tablet_service.cc:1505] Processing CreateTablet for tablet eefbbf73da40432ba2ad54764a8ace91 (DEFAULT_TABLE table=default.Table_3fa040a0070246308ad0028af0f86ae2 [id=8da0bb8b94fc49b9b1f475fbcf672994]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.646765 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef7d8549ecc4546bc03c8e600c54c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.646921 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eefbbf73da40432ba2ad54764a8ace91. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.504441 9751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 987) took 1038 ms. Trace:
I20251027 21:08:06.646994 9751 rpcz_store.cc:276] 1027 21:08:05.466291 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.080673 (+614382us) service_pool.cc:225] Handling call
1027 21:08:06.120166 (+ 39493us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.120189 (+ 23us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.141666 (+ 21477us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.504303 (+362637us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.504416 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.504194 9309 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41710 (request call id 1236) took 1007 ms. Trace:
I20251027 21:08:06.647296 9751 tablet_service.cc:1505] Processing CreateTablet for tablet 278d8f448ef144148cabfc3a57bd24fd (DEFAULT_TABLE table=default.Table_69ef4f8f04d042a18d248c08958ee9f7 [id=983747413ebc4e3ea8ccb0eac5ffff08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.647567 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 278d8f448ef144148cabfc3a57bd24fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.647238 9309 rpcz_store.cc:276] 1027 21:08:05.496258 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.044271 (+548013us) service_pool.cc:225] Handling call
1027 21:08:06.431222 (+386951us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x616000051154. stack: 00007f956e0a89f6 00007f956e0a87b0 00007f956d0843ad 0000000000983f04 0000000000983ecd 00007f957d3af6bd 00007f957d3af7b3 00007f9579f76441 00007f9581c309dd 00007f9581dbd263 00007f95788cd7c6 00007f95788cd4f5 00007f957504727e 00007f957504615e 00007f957504c395 00007f957504ccbf
1027 21:08:06.431981 (+ 759us) catalog_manager.cc:2101] Acquired catalog manager lock
1027 21:08:06.432607 (+ 626us) catalog_manager.cc:2147] Created new table and tablet info
1027 21:08:06.503889 (+ 71282us) catalog_manager.cc:2207] Wrote table and tablets to system table
1027 21:08:06.504018 (+ 129us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1027 21:08:06.504124 (+ 106us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:08:06.467228 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:08:06.467370 (+ 142us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:08:06.467375 (+ 5us) write_op.cc:435] Acquired schema lock
1027 21:08:06.467378 (+ 3us) tablet.cc:660] Decoding operations
1027 21:08:06.467468 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:08:06.467479 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1027 21:08:06.467483 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:08:06.467549 (+ 66us) tablet.cc:699] Row locks acquired
1027 21:08:06.467551 (+ 2us) write_op.cc:260] PREPARE: finished
1027 21:08:06.467601 (+ 50us) write_op.cc:270] Start()
1027 21:08:06.467630 (+ 29us) write_op.cc:276] Timestamp: P: 1761599286467549 usec, L: 0
1027 21:08:06.467634 (+ 4us) op_driver.cc:348] REPLICATION: starting
1027 21:08:06.467815 (+ 181us) log.cc:844] Serialized 433 byte log entry
1027 21:08:06.503002 (+ 35187us) op_driver.cc:464] REPLICATION: finished
1027 21:08:06.503193 (+ 191us) write_op.cc:301] APPLY: starting
1027 21:08:06.503207 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1027 21:08:06.503250 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
1027 21:08:06.503275 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:08:06.503386 (+ 111us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:08:06.503419 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1027 21:08:06.503426 (+ 7us) write_op.cc:312] APPLY: finished
1027 21:08:06.503500 (+ 74us) log.cc:844] Serialized 29 byte log entry
1027 21:08:06.503567 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:06.503587 (+ 20us) write_op.cc:454] Released schema lock
1027 21:08:06.503593 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1027 21:08:06.503656 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:06.503659 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":889241856,"child_traces":[["op",{"apply.queue_time_us":144,"apply.run_cpu_time_us":454,"apply.run_wall_time_us":454,"num_ops":2,"prepare.queue_time_us":34113,"prepare.run_cpu_time_us":664,"prepare.run_wall_time_us":1016,"replication_time_us":35337,"thread_start_us":35077,"threads_started":1}]]}
I20251027 21:08:06.647929 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b98328c296ab4b6ab486c3477c221544" } new_table_name: "default.Table_c7d5f748f61a4201ad3c06b98ab1a958" modify_external_catalogs: true
I20251027 21:08:06.463672 9600 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 997) took 1009 ms. Trace:
I20251027 21:08:06.648260 9600 rpcz_store.cc:276] 1027 21:08:05.454696 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.107807 (+653111us) service_pool.cc:225] Handling call
1027 21:08:06.241796 (+133989us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.241831 (+ 35us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.314604 (+ 72773us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.463502 (+148898us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.463655 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.648695 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 11d5abcbef6c4762adbdbe0f78f3ca53 (DEFAULT_TABLE table=default.Table_401e3aad77324534a4e47f63fe64de89 [id=553ba2e0be3a4fdd8bcc7460994dcf47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.649041 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11d5abcbef6c4762adbdbe0f78f3ca53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.561858 9902 tablet_service.cc:1505] Processing CreateTablet for tablet a24dbceaa95e4f43a5ea416a20e7041e (DEFAULT_TABLE table=default.Table_5a6211c8465040dbad1cc86377a370ea [id=91fb9f95fe9343fca1d8cb0d0f781afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.508126 9741 tablet_service.cc:1505] Processing CreateTablet for tablet 824ff6913b0142d5889c75aa8e3cba3e (DEFAULT_TABLE table=default.Table_152a2bed589f47858b8a99ab7df9984c [id=c1fcc453aee04cbca24ccb8bdfa3ba00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.650413 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 824ff6913b0142d5889c75aa8e3cba3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.650442 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a24dbceaa95e4f43a5ea416a20e7041e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:06.508474 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8b1e5ab297da49058e5f8d27f2fba65d consensus state has no leader
I20251027 21:08:06.653527 9304 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:41710 (request call id 1235) took 1157 ms. Trace:
I20251027 21:08:06.653640 9304 rpcz_store.cc:276] 1027 21:08:05.496182 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.041397 (+545215us) service_pool.cc:225] Handling call
1027 21:08:06.433321 (+391924us) spinlock_profiling.cc:243] Waited 389 ms on lock 0x616000051154. stack: 00007f956e0a89f6 00007f956e0a87b0 00007f956d0843ad 0000000000983f04 0000000000983ecd 00007f957d3af6bd 00007f957d3af7b3 00007f9579f76441 00007f9581c309dd 00007f9581dbf4f6 00007f95788d1146 00007f95788d0e75 00007f957504727e 00007f957504615e 00007f957504c395 00007f957504ccbf
1027 21:08:06.433540 (+ 219us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1027 21:08:06.433735 (+ 195us) catalog_manager.cc:3708] Apply alter schema
1027 21:08:06.433899 (+ 164us) catalog_manager.cc:3740] Acquired catalog manager lock
1027 21:08:06.440540 (+ 6641us) catalog_manager.cc:3898] Updating metadata on disk
1027 21:08:06.508274 (+ 67734us) catalog_manager.cc:3929] Committing alterations to in-memory state
1027 21:08:06.653483 (+145209us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:08:06.468271 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:08:06.468363 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:08:06.468368 (+ 5us) write_op.cc:435] Acquired schema lock
1027 21:08:06.468370 (+ 2us) tablet.cc:660] Decoding operations
1027 21:08:06.468447 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:08:06.468454 (+ 7us) write_op.cc:641] Partition lock acquired for write op
1027 21:08:06.468458 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1027 21:08:06.468513 (+ 55us) tablet.cc:699] Row locks acquired
1027 21:08:06.468517 (+ 4us) write_op.cc:260] PREPARE: finished
1027 21:08:06.468548 (+ 31us) write_op.cc:270] Start()
1027 21:08:06.468576 (+ 28us) write_op.cc:276] Timestamp: P: 1761599286468515 usec, L: 0
1027 21:08:06.468580 (+ 4us) op_driver.cc:348] REPLICATION: starting
1027 21:08:06.468728 (+ 148us) log.cc:844] Serialized 405 byte log entry
1027 21:08:06.503080 (+ 34352us) op_driver.cc:464] REPLICATION: finished
1027 21:08:06.503719 (+ 639us) write_op.cc:301] APPLY: starting
1027 21:08:06.503747 (+ 28us) tablet.cc:1365] starting BulkCheckPresence
1027 21:08:06.503776 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
1027 21:08:06.503799 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:08:06.503852 (+ 53us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:08:06.503875 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:08:06.503882 (+ 7us) write_op.cc:312] APPLY: finished
1027 21:08:06.503932 (+ 50us) log.cc:844] Serialized 23 byte log entry
1027 21:08:06.503974 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:06.503984 (+ 10us) write_op.cc:454] Released schema lock
1027 21:08:06.503989 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1027 21:08:06.504039 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:06.504043 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":895739264,"child_traces":[["op",{"apply.queue_time_us":611,"apply.run_cpu_time_us":313,"apply.run_wall_time_us":314,"num_ops":1,"prepare.queue_time_us":27277,"prepare.run_cpu_time_us":518,"prepare.run_wall_time_us":790,"replication_time_us":34474}]]}
I20251027 21:08:06.654311 11097 raft_consensus.cc:2804] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:06.654343 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f4d6cc291d944761aae3234c8ebdd6f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:06.654472 11097 raft_consensus.cc:697] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
W20251027 21:08:06.654845 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4d6cc291d944761aae3234c8ebdd6f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:06.654778 11097 consensus_queue.cc:237] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.466781 9595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 995) took 1013 ms. Trace:
I20251027 21:08:06.655947 9595 rpcz_store.cc:276] 1027 21:08:05.452996 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.104513 (+651517us) service_pool.cc:225] Handling call
1027 21:08:06.238761 (+134248us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.264581 (+ 25820us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.274381 (+ 9800us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.464349 (+189968us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.466768 (+ 2419us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.646104 9737 rpcz_store.cc:276] 1027 21:08:05.461411 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.071240 (+609829us) service_pool.cc:225] Handling call
1027 21:08:06.071722 (+ 482us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.071755 (+ 33us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.140620 (+ 68865us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.500479 (+359859us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.500605 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.656320 9595 tablet_service.cc:1505] Processing CreateTablet for tablet eefbbf73da40432ba2ad54764a8ace91 (DEFAULT_TABLE table=default.Table_3fa040a0070246308ad0028af0f86ae2 [id=8da0bb8b94fc49b9b1f475fbcf672994]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.656320 9737 tablet_service.cc:1505] Processing CreateTablet for tablet d332d8e8f0c949589d1fd29b4bd7f076 (DEFAULT_TABLE table=default.Table_a8719fbbe9b6463bb1dd10f1b8f92cb8 [id=f53c29285cba45b88e6942c1e1d2f26e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.656642 9595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eefbbf73da40432ba2ad54764a8ace91. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.658869 11018 raft_consensus.cc:493] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:06.658984 11018 raft_consensus.cc:515] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:06.659462 11018 leader_election.cc:290] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:06.516850 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 8ab9310bba1f425fbee6595e0182aae9 (DEFAULT_TABLE table=default.Table_7ff9f51c77a1418faf1e35ad182c91d7 [id=51b8c67b97dd461791faaa2cc6d9e5a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.520470 9743 tablet_service.cc:1505] Processing CreateTablet for tablet a24dbceaa95e4f43a5ea416a20e7041e (DEFAULT_TABLE table=default.Table_5a6211c8465040dbad1cc86377a370ea [id=91fb9f95fe9343fca1d8cb0d0f781afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.525259 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 793aebd92b65423796d6ad2d845519c2 (DEFAULT_TABLE table=default.Table_55f3c5dc04064eb7b371b99b5e64287a [id=d05bc2048ea84afc9b1c28340af4febb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.527550 9754 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 974) took 1078 ms. Trace:
I20251027 21:08:06.530692 9739 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 971) took 1084 ms. Trace:
I20251027 21:08:06.662456 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ab9310bba1f425fbee6595e0182aae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.662484 9739 rpcz_store.cc:276] 1027 21:08:05.446616 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.657342 (+210726us) service_pool.cc:225] Handling call
1027 21:08:05.698555 (+ 41213us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:05.698586 (+ 31us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.146831 (+448245us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.500479 (+353648us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.530678 (+ 30199us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30039}
I20251027 21:08:06.662866 9739 tablet_service.cc:1505] Processing CreateTablet for tablet 1778f4a7f81541bc8932201ead4189b4 (DEFAULT_TABLE table=default.Table_d5398f9a1dca4617be8062e32b583441 [id=704c1f1c3b964a55b95d7858445dd7e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.533036 9755 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 972) took 1085 ms. Trace:
I20251027 21:08:06.514659 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 6600774e4c7b48479a2fdb02614ba505 (table default.Table_d7530773668f42c5a07a921df02a95ab [id=914b728e358e4fdba6d38d6c7beadaca]). Expected version 1 got 0
I20251027 21:08:06.663069 9755 rpcz_store.cc:276] 1027 21:08:05.447472 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.660056 (+212584us) service_pool.cc:225] Handling call
1027 21:08:05.977832 (+317776us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:05.977867 (+ 35us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.316506 (+338639us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.532869 (+216363us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.533017 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.528380 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 4c67c5cb3b244feb8478f9594b834fbf
I20251027 21:08:06.663334 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet ca477937278f45b4b115fec715f32a92 (table default.Table_69627cab65c2407b9226a05106b7bdf4 [id=fc5a0b5a96e6400da438307d26f73614]). Expected version 3 got 0
I20251027 21:08:06.545389 9752 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 975) took 1095 ms. Trace:
I20251027 21:08:06.663597 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet d56d323ef0e74344a479940b93fea11a (table default.Table_cf11a8c344c544c5ba54672fee38e4aa [id=71e02b21c9654e45a41d3e450ac3fbe1]). Expected version 2 got 0
I20251027 21:08:06.663538 9752 rpcz_store.cc:276] 1027 21:08:05.450923 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.895204 (+444281us) service_pool.cc:225] Handling call
1027 21:08:05.901816 (+ 6612us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:05.901851 (+ 35us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.204959 (+303108us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.545250 (+340291us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.545358 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.548183 9749 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 986) took 1083 ms. Trace:
I20251027 21:08:06.663784 9749 rpcz_store.cc:276] 1027 21:08:05.465131 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.076676 (+611545us) service_pool.cc:225] Handling call
1027 21:08:06.120411 (+ 43735us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.120429 (+ 18us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.322692 (+202263us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.548036 (+225344us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.548166 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.663859 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 45568792d7954b498581ce69fb611948 (table default.Table_3e3e0c17a3bf452fbb217c6e3ee8bdcb [id=2eb093cd58e643a58b14e564d5914fbc]). Expected version 1 got 0
I20251027 21:08:06.550374 9750 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 970) took 1104 ms. Trace:
I20251027 21:08:06.664001 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_58a7455c41ba48e9bd85661c6f1f91bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:06.565186 9888 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 992) took 1111 ms. Trace:
I20251027 21:08:06.668748 9888 rpcz_store.cc:276] 1027 21:08:05.454188 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.247923 (+793735us) service_pool.cc:225] Handling call
1027 21:08:06.248361 (+ 438us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.248382 (+ 21us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.464645 (+216263us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.565029 (+100384us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.565167 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.669219 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 1778f4a7f81541bc8932201ead4189b4 (DEFAULT_TABLE table=default.Table_d5398f9a1dca4617be8062e32b583441 [id=704c1f1c3b964a55b95d7858445dd7e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.669536 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1778f4a7f81541bc8932201ead4189b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.469089 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 793aebd92b65423796d6ad2d845519c2 (DEFAULT_TABLE table=default.Table_55f3c5dc04064eb7b371b99b5e64287a [id=d05bc2048ea84afc9b1c28340af4febb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.670279 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 793aebd92b65423796d6ad2d845519c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.571031 9901 tablet_service.cc:1505] Processing CreateTablet for tablet 793aebd92b65423796d6ad2d845519c2 (DEFAULT_TABLE table=default.Table_55f3c5dc04064eb7b371b99b5e64287a [id=d05bc2048ea84afc9b1c28340af4febb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.672966 9901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 793aebd92b65423796d6ad2d845519c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.573554 9890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 993) took 1118 ms. Trace:
I20251027 21:08:06.472198 9606 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1002) took 1011 ms. Trace:
I20251027 21:08:06.674196 9890 rpcz_store.cc:276] 1027 21:08:05.455042 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.251472 (+796430us) service_pool.cc:225] Handling call
1027 21:08:06.251840 (+ 368us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.251871 (+ 31us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.268134 (+ 16263us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.573429 (+305295us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.573541 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.674196 9606 rpcz_store.cc:276] 1027 21:08:05.461209 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.259887 (+798678us) service_pool.cc:225] Handling call
1027 21:08:06.260252 (+ 365us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.260271 (+ 19us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.278290 (+ 18019us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.472008 (+193718us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.472179 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.674439 9595 tablet_service.cc:1505] Processing CreateTablet for tablet ea02ab2fed634dbcba3ca8b04808b694 (DEFAULT_TABLE table=default.Table_0ce16ec9dc8b44eab80cedc074577cba [id=c22fee46f319480e88ab7745f6a95126]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.674547 9890 tablet_service.cc:1505] Processing CreateTablet for tablet dfc461f51c704e63a67946df02ca90d4 (DEFAULT_TABLE table=default.Table_7cde4cdee7294ac8ab1c8dfcbb93abcf [id=8d165461112a482398be4516ae81d130]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.674603 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 3ecfcff3d55147a2a29e65820de730f5 (DEFAULT_TABLE table=default.Table_2374f13424064567a276cca554d2aff8 [id=a528343b93b245fcb5a8019226c61546]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.674816 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc461f51c704e63a67946df02ca90d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.475632 9601 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 991) took 1026 ms. Trace:
I20251027 21:08:06.476172 10029 tablet_bootstrap.cc:492] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:06.675002 10029 ts_tablet_manager.cc:1403] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.572s user 0.018s sys 0.023s
I20251027 21:08:06.675524 11177 raft_consensus.cc:493] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:06.675616 9595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea02ab2fed634dbcba3ca8b04808b694. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.675662 11177 raft_consensus.cc:515] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.676313 11177 leader_election.cc:290] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:06.674904 9601 rpcz_store.cc:276] 1027 21:08:05.448796 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.558922 (+110126us) service_pool.cc:225] Handling call
1027 21:08:05.560069 (+ 1147us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:05.560088 (+ 19us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.308236 (+748148us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.475504 (+167268us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.475619 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.677213 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 94de56b0a42b4b98b666f0fde1771e17 (DEFAULT_TABLE table=default.Table_cfb487c77526400b84830a2d722437a7 [id=4920a6ea8aec4b7cb85cfe7cf228f481]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.576576 9893 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 995) took 1117 ms. Trace:
I20251027 21:08:06.677645 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94de56b0a42b4b98b666f0fde1771e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.677659 9893 rpcz_store.cc:276] 1027 21:08:05.458727 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.262192 (+803465us) service_pool.cc:225] Handling call
1027 21:08:06.262504 (+ 312us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.262523 (+ 19us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.265902 (+ 3379us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.576439 (+310537us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.576552 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.678074 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 4c3d07b891a248efb0860cd000e44432 (DEFAULT_TABLE table=default.Table_6bb3152125264bc6a28eda3f88cee649 [id=8dd05f4b131145e78b51c4eb56305cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.678404 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3d07b891a248efb0860cd000e44432. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.476241 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 21ead81fd55d49cda3f031cdf8a47a8c (DEFAULT_TABLE table=default.Table_bfc1511d230f4df297686bac2acbeac2 [id=444b53029bd74970b8278fab308c5769]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.476177 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32f80614e30d41128f9fb2e0ea165223" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:06.681758 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21ead81fd55d49cda3f031cdf8a47a8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.579461 9889 tablet_service.cc:1505] Processing CreateTablet for tablet 21ead81fd55d49cda3f031cdf8a47a8c (DEFAULT_TABLE table=default.Table_bfc1511d230f4df297686bac2acbeac2 [id=444b53029bd74970b8278fab308c5769]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.582995 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 7057c55a820d4e379fe1d9abfa353821 (DEFAULT_TABLE table=default.Table_f396e1e1ebc44cf9a774f6523942ff0b [id=633dfb2a421c44b2833584fcabc6e475]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.585049 9898 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 994) took 1128 ms. Trace:
I20251027 21:08:06.479023 9614 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 998) took 1022 ms. Trace:
I20251027 21:08:06.682348 9898 rpcz_store.cc:276] 1027 21:08:05.456601 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.254296 (+797695us) service_pool.cc:225] Handling call
1027 21:08:06.254619 (+ 323us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.254639 (+ 20us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.257756 (+ 3117us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.584892 (+327136us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.585032 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.682359 9614 rpcz_store.cc:276] 1027 21:08:05.456153 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.108136 (+651983us) service_pool.cc:225] Handling call
1027 21:08:06.248273 (+140137us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.248313 (+ 40us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.311687 (+ 63374us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.478906 (+167219us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.479010 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.682703 9898 tablet_service.cc:1505] Processing CreateTablet for tablet e5a4118310fd45609f7f0b7d12d253ca (DEFAULT_TABLE table=default.Table_904989a77e13498f8fa57464da33ecfe [id=35b252ef3700450b87d9202333f2f1c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.682703 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 322337041c954838bc07c43241abf06d (DEFAULT_TABLE table=default.Table_1812ea722de14512bad256f85c0596ab [id=93ac6bededf24c18be9e281c678a7699]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.585677 9887 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 999) took 1006 ms. Trace:
I20251027 21:08:06.683034 9887 rpcz_store.cc:276] 1027 21:08:05.578907 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.273163 (+694256us) service_pool.cc:225] Handling call
1027 21:08:06.293705 (+ 20542us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.293725 (+ 20us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.307277 (+ 13552us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.585533 (+278256us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.585663 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.683231 9889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21ead81fd55d49cda3f031cdf8a47a8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.683403 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 69f1738d936741f792778113b15e57f7 (DEFAULT_TABLE table=default.Table_4a3918c0e2224ae8b75c099bd4332ca0 [id=fde53cb33c144f969adecb272bbb579e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.684028 10029 raft_consensus.cc:359] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:06.684185 10029 raft_consensus.cc:385] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:06.684234 10029 raft_consensus.cc:740] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:06.684427 10029 consensus_queue.cc:260] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:06.684965 10029 ts_tablet_manager.cc:1434] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251027 21:08:06.685240 10029 tablet_bootstrap.cc:492] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:06.479195 9604 tablet_service.cc:1505] Processing CreateTablet for tablet 7057c55a820d4e379fe1d9abfa353821 (DEFAULT_TABLE table=default.Table_f396e1e1ebc44cf9a774f6523942ff0b [id=633dfb2a421c44b2833584fcabc6e475]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.585728 9899 tablet_service.cc:1505] Processing CreateTablet for tablet c76870d7ca1b43308f0ab94d4ab39b5f (DEFAULT_TABLE table=default.Table_645e7862b8ef4097aa7152d5984b7fb7 [id=805d6b3b37ab43358f8d46185f7c9ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.481307 9603 tablet_service.cc:1505] Processing CreateTablet for tablet c76870d7ca1b43308f0ab94d4ab39b5f (DEFAULT_TABLE table=default.Table_645e7862b8ef4097aa7152d5984b7fb7 [id=805d6b3b37ab43358f8d46185f7c9ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.686399 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ecfcff3d55147a2a29e65820de730f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.588268 9886 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 1004) took 1008 ms. Trace:
I20251027 21:08:06.686515 9886 rpcz_store.cc:276] 1027 21:08:05.579338 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.274205 (+694867us) service_pool.cc:225] Handling call
1027 21:08:06.310256 (+ 36051us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.310287 (+ 31us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.313679 (+ 3392us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.588154 (+274475us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.588256 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.687007 9886 tablet_service.cc:1505] Processing CreateTablet for tablet 11d5abcbef6c4762adbdbe0f78f3ca53 (DEFAULT_TABLE table=default.Table_401e3aad77324534a4e47f63fe64de89 [id=553ba2e0be3a4fdd8bcc7460994dcf47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.482151 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 942) took 1716 ms. Trace:
I20251027 21:08:06.687186 10043 rpcz_store.cc:276] 1027 21:08:04.765195 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.765368 (+ 173us) service_pool.cc:225] Handling call
1027 21:08:05.541151 (+775783us) ts_tablet_manager.cc:1151] Deleting tablet 32f80614e30d41128f9fb2e0ea165223
1027 21:08:05.541157 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:06.106033 (+564876us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.393811 (+287778us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.482121 (+ 88310us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":775630}
I20251027 21:08:06.687472 10043 tablet_replica.cc:333] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:06.687433 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7292bc6c22ca4ab5b8f53a69520833ba" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:06.687610 10043 raft_consensus.cc:2243] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:06.687906 10043 raft_consensus.cc:2272] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:06.687922 9860 leader_election.cc:343] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:06.589972 9891 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 968) took 1253 ms. Trace:
I20251027 21:08:06.688066 9891 rpcz_store.cc:276] 1027 21:08:05.336093 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.336148 (+ 55us) service_pool.cc:225] Handling call
1027 21:08:05.540756 (+204608us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:05.540779 (+ 23us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.551096 (+1010317us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.589846 (+ 38750us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.589959 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.688392 9891 tablet_service.cc:1505] Processing CreateTablet for tablet eefbbf73da40432ba2ad54764a8ace91 (DEFAULT_TABLE table=default.Table_3fa040a0070246308ad0028af0f86ae2 [id=8da0bb8b94fc49b9b1f475fbcf672994]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.483484 9613 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1001) took 1023 ms. Trace:
I20251027 21:08:06.664101 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 87d5af4607f04f92abcc7292b4176cbd (table default.Table_c78ef9ebe86b4da08443e37c83d4e80d [id=cdb023218cda4e9ea04a226b37a220d9]). Expected version 2 got 0
I20251027 21:08:06.529853 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 17a19ce92da8448bba471f94e3ee4647
I20251027 21:08:06.688665 9613 rpcz_store.cc:276] 1027 21:08:05.460303 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.231103 (+770800us) service_pool.cc:225] Handling call
1027 21:08:06.231451 (+ 348us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.231471 (+ 20us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.234647 (+ 3176us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.483354 (+248707us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.483465 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.689035 9613 tablet_service.cc:1505] Processing CreateTablet for tablet 4c60797e8cd5442b9eef86ee51d4b9c6 (DEFAULT_TABLE table=default.Table_44f9303ecfd745f99df2ffa39d8a0499 [id=344f0961965d4b379ac0ca204ff88500]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:06.664467 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58a7455c41ba48e9bd85661c6f1f91bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:06.688961 9312 catalog_manager.cc:5649] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:06.689460 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 9fde4fd91943498d8888d1783005e492 (table default.Table_87ac731a20bf4f6bbce8d82c1ad3bd97 [id=3a0c8b19df934d1580349b061fe0e8f5]). Expected version 1 got 0
I20251027 21:08:06.590540 9905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 989) took 1138 ms. Trace:
I20251027 21:08:06.689569 9905 rpcz_store.cc:276] 1027 21:08:05.451587 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.024812 (+573225us) service_pool.cc:225] Handling call
1027 21:08:06.025449 (+ 637us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.025470 (+ 21us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.198795 (+173325us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.590431 (+391636us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.590528 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.689541 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a611f21a506d4b8badac8f08f6a4ad55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:06.689905 9905 tablet_service.cc:1505] Processing CreateTablet for tablet ea02ab2fed634dbcba3ca8b04808b694 (DEFAULT_TABLE table=default.Table_0ce16ec9dc8b44eab80cedc074577cba [id=c22fee46f319480e88ab7745f6a95126]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.488663 9682 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:08:06.529548 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 35b99f8c57e847b1848cbdea9e50da11
I20251027 21:08:06.590809 9900 tablet_service.cc:1505] Processing CreateTablet for tablet 0d0032dd868c4dc79ea87036264434ee (DEFAULT_TABLE table=default.Table_5b5950cb52f34415b3a217c9c772bb3b [id=a6e66d2764ea438188e3d7992f1fd7a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.512280 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 0d0032dd868c4dc79ea87036264434ee (DEFAULT_TABLE table=default.Table_5b5950cb52f34415b3a217c9c772bb3b [id=a6e66d2764ea438188e3d7992f1fd7a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.593663 9903 tablet_service.cc:1505] Processing CreateTablet for tablet 6c05b69509414fe8bfc809cd307d5f2e (DEFAULT_TABLE table=default.Table_c9a81e9dabad4a7d9fbd427439a7587c [id=40ce2a0befe54d698759beff3bfdf0f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.517135 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 6c05b69509414fe8bfc809cd307d5f2e (DEFAULT_TABLE table=default.Table_c9a81e9dabad4a7d9fbd427439a7587c [id=40ce2a0befe54d698759beff3bfdf0f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.597069 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 278d8f448ef144148cabfc3a57bd24fd (DEFAULT_TABLE table=default.Table_69ef4f8f04d042a18d248c08958ee9f7 [id=983747413ebc4e3ea8ccb0eac5ffff08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.519919 9608 tablet_service.cc:1505] Processing CreateTablet for tablet eef7d8549ecc4546bc03c8e600c54c49 (DEFAULT_TABLE table=default.Table_9193c70ec7c84ce4829d242e054520a4 [id=9f6872a0c18a461e9d82f1735fd8cf60]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.600601 9895 tablet_service.cc:1505] Processing CreateTablet for tablet eef7d8549ecc4546bc03c8e600c54c49 (DEFAULT_TABLE table=default.Table_9193c70ec7c84ce4829d242e054520a4 [id=9f6872a0c18a461e9d82f1735fd8cf60]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.523334 9599 tablet_service.cc:1505] Processing CreateTablet for tablet d332d8e8f0c949589d1fd29b4bd7f076 (DEFAULT_TABLE table=default.Table_a8719fbbe9b6463bb1dd10f1b8f92cb8 [id=f53c29285cba45b88e6942c1e1d2f26e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.690733 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5b230fb9992d42798e2da304f545376c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:06.603977 9894 tablet_service.cc:1505] Processing CreateTablet for tablet d332d8e8f0c949589d1fd29b4bd7f076 (DEFAULT_TABLE table=default.Table_a8719fbbe9b6463bb1dd10f1b8f92cb8 [id=f53c29285cba45b88e6942c1e1d2f26e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.534435 9598 tablet_service.cc:1505] Processing CreateTablet for tablet 1778f4a7f81541bc8932201ead4189b4 (DEFAULT_TABLE table=default.Table_d5398f9a1dca4617be8062e32b583441 [id=704c1f1c3b964a55b95d7858445dd7e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.607420 9897 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 996) took 1147 ms. Trace:
W20251027 21:08:06.691193 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b230fb9992d42798e2da304f545376c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:06.691138 9897 rpcz_store.cc:276] 1027 21:08:05.459661 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.268683 (+809022us) service_pool.cc:225] Handling call
1027 21:08:06.289337 (+ 20654us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.289358 (+ 21us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.292276 (+ 2918us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.607302 (+315026us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.607407 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.691514 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 3ecfcff3d55147a2a29e65820de730f5 (DEFAULT_TABLE table=default.Table_2374f13424064567a276cca554d2aff8 [id=a528343b93b245fcb5a8019226c61546]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.537658 9594 tablet_service.cc:1505] Processing CreateTablet for tablet dfc461f51c704e63a67946df02ca90d4 (DEFAULT_TABLE table=default.Table_7cde4cdee7294ac8ab1c8dfcbb93abcf [id=8d165461112a482398be4516ae81d130]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.607671 9896 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 997) took 1146 ms. Trace:
I20251027 21:08:06.691957 9896 rpcz_store.cc:276] 1027 21:08:05.460896 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.272594 (+811698us) service_pool.cc:225] Handling call
1027 21:08:06.293649 (+ 21055us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.293679 (+ 30us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.296736 (+ 3057us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.607493 (+310757us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.607652 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251027 21:08:06.690088 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a611f21a506d4b8badac8f08f6a4ad55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:06.692385 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 94de56b0a42b4b98b666f0fde1771e17 (DEFAULT_TABLE table=default.Table_cfb487c77526400b84830a2d722437a7 [id=4920a6ea8aec4b7cb85cfe7cf228f481]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.540659 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 4c3d07b891a248efb0860cd000e44432 (DEFAULT_TABLE table=default.Table_6bb3152125264bc6a28eda3f88cee649 [id=8dd05f4b131145e78b51c4eb56305cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.695003 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a4118310fd45609f7f0b7d12d253ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.695199 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c76870d7ca1b43308f0ab94d4ab39b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.664007 9750 rpcz_store.cc:276] 1027 21:08:05.445478 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.652183 (+206705us) service_pool.cc:225] Handling call
1027 21:08:05.690307 (+ 38124us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:05.690336 (+ 29us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.145083 (+454747us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.550231 (+405148us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.550343 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.554520 10040 ts_tablet_manager.cc:1929] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:06.696928 10040 log.cc:1199] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/ccf10ef559f74e2db28023fd520c8de9
I20251027 21:08:06.560143 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 21ead81fd55d49cda3f031cdf8a47a8c (DEFAULT_TABLE table=default.Table_bfc1511d230f4df297686bac2acbeac2 [id=444b53029bd74970b8278fab308c5769]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.567086 9744 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 985) took 1103 ms. Trace:
I20251027 21:08:06.697198 11177 raft_consensus.cc:493] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:06.697178 9744 rpcz_store.cc:276] 1027 21:08:05.464106 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.072483 (+608377us) service_pool.cc:225] Handling call
1027 21:08:06.119763 (+ 47280us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.119814 (+ 51us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.354803 (+234989us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.566888 (+212085us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.567069 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.697307 11177 raft_consensus.cc:515] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.598407 10829 raft_consensus.cc:697] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:06.697682 10829 consensus_queue.cc:237] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.697832 11177 leader_election.cc:290] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:06.608742 10031 tablet_bootstrap.cc:492] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:06.698220 10031 ts_tablet_manager.cc:1403] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.532s user 0.034s sys 0.105s
I20251027 21:08:06.551599 10997 raft_consensus.cc:493] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:06.698421 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3abc5f78c4e41e29128e2d790319c75" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:06.698411 10997 raft_consensus.cc:515] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.664736 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 793aebd92b65423796d6ad2d845519c2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:06.698864 9860 leader_election.cc:343] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:06.662415 9754 rpcz_store.cc:276] 1027 21:08:05.449381 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.891426 (+442045us) service_pool.cc:225] Handling call
1027 21:08:05.891775 (+ 349us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:05.891811 (+ 36us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.227045 (+335234us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.527400 (+300355us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.527534 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.699298 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c76870d7ca1b43308f0ab94d4ab39b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.700425 9907 raft_consensus.cc:2468] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:06.701532 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1778f4a7f81541bc8932201ead4189b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.701891 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a24dbceaa95e4f43a5ea416a20e7041e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.702085 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f1738d936741f792778113b15e57f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.705322 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c05b69509414fe8bfc809cd307d5f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.705775 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef7d8549ecc4546bc03c8e600c54c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.706450 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c76870d7ca1b43308f0ab94d4ab39b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.706462 9886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11d5abcbef6c4762adbdbe0f78f3ca53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.706811 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d332d8e8f0c949589d1fd29b4bd7f076. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.707144 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0032dd868c4dc79ea87036264434ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.707468 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3d07b891a248efb0860cd000e44432. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.707810 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc461f51c704e63a67946df02ca90d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.708156 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1778f4a7f81541bc8932201ead4189b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.708523 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7057c55a820d4e379fe1d9abfa353821. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:06.708809 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 90f96c21da9944f1ade05d35787d47e2 consensus state has no leader
I20251027 21:08:06.708913 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 322337041c954838bc07c43241abf06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.709311 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c60797e8cd5442b9eef86ee51d4b9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.709393 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9dfdbd16e4f34bc4ba44d300083c774d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:06.709930 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9dfdbd16e4f34bc4ba44d300083c774d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:06.713006 10043 ts_tablet_manager.cc:1916] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:06.715744 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 0d75e6a4286e428eb85b24638382d23b (DEFAULT_TABLE table=default.Table_bde50fa24ef54301a7b55d76cf5a9bda [id=25ffc58bf0a54adfa648f55c76f3df44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.716190 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d332d8e8f0c949589d1fd29b4bd7f076. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.717319 10032 tablet_bootstrap.cc:492] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:06.717423 10032 ts_tablet_manager.cc:1403] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.073s user 0.013s sys 0.000s
I20251027 21:08:06.717864 10029 tablet_bootstrap.cc:654] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:06.718237 10032 raft_consensus.cc:359] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.718377 10032 raft_consensus.cc:385] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:06.718425 10032 raft_consensus.cc:740] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:06.718616 10032 consensus_queue.cc:260] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.719111 10032 ts_tablet_manager.cc:1434] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:06.724078 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 933fb952d66d4cb1ade70c334fbbb3e8
I20251027 21:08:06.724725 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4ba4c823d97045959f6926a92a489794" } new_table_name: "default.Table_b2a1b49649c54e5ab7a23fad7a157502" modify_external_catalogs: true
I20251027 21:08:06.729993 9601 tablet_service.cc:1505] Processing CreateTablet for tablet febc011f42244fdfa1954dbb630c3d0c (DEFAULT_TABLE table=default.Table_0e8bd764e80e46a5b665da0e3e2b677e [id=288ecea80da64655ae6246da518b77b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.740625 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4d37173734c0410292144b74c11b4d9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:06.740957 9310 master_service.cc:954] ReplaceTablet: received request to replace tablet 4a621d5246f14997a2f2952ab640efc8 from {username='slave'} at 127.0.0.1:41710
W20251027 21:08:06.741173 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d37173734c0410292144b74c11b4d9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:06.741815 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 83d6f067aef343ddaaa29b1a85e7f328 consensus state has no leader
I20251027 21:08:06.742756 10032 tablet_bootstrap.cc:492] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:06.742110 9309 master_service.cc:954] ReplaceTablet: received request to replace tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:06.743606 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2430eb08210e4f16ad19473f4682c43c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:06.744102 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2430eb08210e4f16ad19473f4682c43c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:06.744297 10032 tablet_bootstrap.cc:654] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:06.742522 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet 8dc358445526415d8cd4f1ac6779126d from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:06.742811 9305 master_service.cc:954] ReplaceTablet: received request to replace tablet aabaa34787cf4d179822269b7f323314 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:06.746706 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_56eaa1a50d974ff2bbef789f8054226a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:06.666051 9568 leader_election.cc:304] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:06.747092 9568 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 81075.4us
W20251027 21:08:06.747191 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56eaa1a50d974ff2bbef789f8054226a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:06.747884 9568 leader_election.cc:343] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: f775a7e42bdc4910bdb604b26a28a7f4
I20251027 21:08:06.748029 9568 leader_election.cc:304] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:06.748349 11057 raft_consensus.cc:2804] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:06.748463 11057 raft_consensus.cc:697] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:06.748507 11142 raft_consensus.cc:2804] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:06.748589 11142 raft_consensus.cc:697] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:06.748749 11142 consensus_queue.cc:237] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:06.749204 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d0562f16c30449a8c761320a80f0ae7 consensus state has no leader
I20251027 21:08:06.749369 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 32f80614e30d41128f9fb2e0ea165223 (table default.Table_a4ae12c512d645e58eadc2daa1bb7b0a [id=e1f4a77dfc0a497080c3c0ad4fe2e370]) successfully deleted
I20251027 21:08:06.749403 9306 master_service.cc:954] ReplaceTablet: received request to replace tablet db2735604dc14039810df4ecd330c671 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:06.602458 9611 tablet_service.cc:1505] Processing CreateTablet for tablet e5a4118310fd45609f7f0b7d12d253ca (DEFAULT_TABLE table=default.Table_904989a77e13498f8fa57464da33ecfe [id=35b252ef3700450b87d9202333f2f1c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.735777 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 01e9db9eaf8e41c6a2ee13b9ed127421 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:06.742935 10859 raft_consensus.cc:493] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:06.750265 10859 raft_consensus.cc:515] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:06.750945 10859 leader_election.cc:290] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:06.748704 11057 consensus_queue.cc:237] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:06.752856 9568 leader_election.cc:343] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 76e4fb2bea7f4a4696fe9f6bb892bccc
I20251027 21:08:06.737624 9605 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1019) took 1030 ms. Trace:
I20251027 21:08:06.756961 9605 rpcz_store.cc:276] 1027 21:08:05.706687 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.468881 (+762194us) service_pool.cc:225] Handling call
1027 21:08:06.670084 (+201203us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.670113 (+ 29us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.673531 (+ 3418us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.737464 (+ 63933us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.737611 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.757586 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 6d0562f16c30449a8c761320a80f0ae7 (DEFAULT_TABLE table=default.Table_f7947222dc5043359f1891191346329b [id=f541f8840fd341ba80ff1d46c59e3eed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.758003 11177 raft_consensus.cc:493] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:06.758162 11177 raft_consensus.cc:515] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.758855 11177 leader_election.cc:290] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:06.759567 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f96c21da9944f1ade05d35787d47e2" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:06.760200 9860 leader_election.cc:343] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:06.761636 9600 tablet_service.cc:1505] Processing CreateTablet for tablet b57c38c7a97f45feb1b7f90e85577a40 (DEFAULT_TABLE table=default.Table_f433bb2ad7be45fc82c323420fe9808f [id=37247477866142efab09f520c780f952]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.770990 9899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 1017) took 1072 ms. Trace:
I20251027 21:08:06.771075 9899 rpcz_store.cc:276] 1027 21:08:05.698052 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.585532 (+887480us) service_pool.cc:225] Handling call
1027 21:08:06.695037 (+109505us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.695084 (+ 47us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.697932 (+ 2848us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.770814 (+ 72882us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.770973 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.771476 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 322337041c954838bc07c43241abf06d (DEFAULT_TABLE table=default.Table_1812ea722de14512bad256f85c0596ab [id=93ac6bededf24c18be9e281c678a7699]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.771958 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 4c60797e8cd5442b9eef86ee51d4b9c6 (DEFAULT_TABLE table=default.Table_44f9303ecfd745f99df2ffa39d8a0499 [id=344f0961965d4b379ac0ca204ff88500]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.772294 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 0d75e6a4286e428eb85b24638382d23b (DEFAULT_TABLE table=default.Table_bde50fa24ef54301a7b55d76cf5a9bda [id=25ffc58bf0a54adfa648f55c76f3df44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.775663 9595 tablet_service.cc:1505] Processing CreateTablet for tablet f2e5b6912c484517afd48f8070455da8 (DEFAULT_TABLE table=default.Table_557acd3dc3344fff87c5b92798fcfa66 [id=307c97840b09416c91f5e3fc0368a600]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.776011 9595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2e5b6912c484517afd48f8070455da8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:06.776597 9710 leader_election.cc:343] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: Tablet not RUNNING: SHUTDOWN
I20251027 21:08:06.776703 9710 leader_election.cc:304] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:06.777079 11169 raft_consensus.cc:2749] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:06.777235 10997 leader_election.cc:290] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:06.780885 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21ead81fd55d49cda3f031cdf8a47a8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.781533 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 69f1738d936741f792778113b15e57f7 (DEFAULT_TABLE table=default.Table_4a3918c0e2224ae8b75c099bd4332ca0 [id=fde53cb33c144f969adecb272bbb579e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.781756 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f775a7e42bdc4910bdb604b26a28a7f4" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:06.782922 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f1738d936741f792778113b15e57f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.783970 9744 tablet_service.cc:1505] Processing CreateTablet for tablet 11d5abcbef6c4762adbdbe0f78f3ca53 (DEFAULT_TABLE table=default.Table_401e3aad77324534a4e47f63fe64de89 [id=553ba2e0be3a4fdd8bcc7460994dcf47]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.784260 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11d5abcbef6c4762adbdbe0f78f3ca53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.785194 10040 ts_tablet_manager.cc:1950] T ccf10ef559f74e2db28023fd520c8de9 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:06.785475 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 938) took 1983 ms. Trace:
I20251027 21:08:06.785538 10040 rpcz_store.cc:276] 1027 21:08:04.801992 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.802216 (+ 224us) service_pool.cc:225] Handling call
1027 21:08:06.004434 (+1202218us) ts_tablet_manager.cc:1151] Deleting tablet ccf10ef559f74e2db28023fd520c8de9
1027 21:08:06.004441 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:06.115787 (+111346us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.554402 (+438615us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.785462 (+231060us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1202049}
I20251027 21:08:06.785935 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 936) took 1998 ms. Trace:
I20251027 21:08:06.786000 10040 rpcz_store.cc:276] 1027 21:08:04.787206 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.850456 (+ 63250us) service_pool.cc:225] Handling call
1027 21:08:06.785828 (+1935372us) ts_tablet_manager.cc:1151] Deleting tablet 1d1d4031a14e448ea5a60327bd37d003
1027 21:08:06.785836 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:06.785920 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1935212}
I20251027 21:08:06.786245 10040 tablet_replica.cc:333] T 4debc3f13f3a42939b6b83f6523aab7c P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:06.786461 10040 raft_consensus.cc:2243] T 4debc3f13f3a42939b6b83f6523aab7c P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:06.786653 10040 raft_consensus.cc:2272] T 4debc3f13f3a42939b6b83f6523aab7c P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:06.787287 9755 tablet_service.cc:1505] Processing CreateTablet for tablet e5a4118310fd45609f7f0b7d12d253ca (DEFAULT_TABLE table=default.Table_904989a77e13498f8fa57464da33ecfe [id=35b252ef3700450b87d9202333f2f1c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.787655 9754 tablet_service.cc:1505] Processing CreateTablet for tablet ea02ab2fed634dbcba3ca8b04808b694 (DEFAULT_TABLE table=default.Table_0ce16ec9dc8b44eab80cedc074577cba [id=c22fee46f319480e88ab7745f6a95126]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.787971 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 4c3d07b891a248efb0860cd000e44432 (DEFAULT_TABLE table=default.Table_6bb3152125264bc6a28eda3f88cee649 [id=8dd05f4b131145e78b51c4eb56305cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.788307 9752 tablet_service.cc:1505] Processing CreateTablet for tablet dfc461f51c704e63a67946df02ca90d4 (DEFAULT_TABLE table=default.Table_7cde4cdee7294ac8ab1c8dfcbb93abcf [id=8d165461112a482398be4516ae81d130]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.788396 9902 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 1013) took 1090 ms. Trace:
I20251027 21:08:06.788489 9902 rpcz_store.cc:276] 1027 21:08:05.697708 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.561637 (+863929us) service_pool.cc:225] Handling call
1027 21:08:06.650294 (+ 88657us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.650314 (+ 20us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.653347 (+ 3033us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.788250 (+134903us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.788379 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.791567 10040 ts_tablet_manager.cc:1916] T 4debc3f13f3a42939b6b83f6523aab7c P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:06.791811 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet c548d2413a5940c8b114ced667b4dc41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:05 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:06.791996 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c67c5cb3b244feb8478f9594b834fbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:06.792120 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet 17a19ce92da8448bba471f94e3ee4647 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:06.792255 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet 35b99f8c57e847b1848cbdea9e50da11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:06.792359 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet 933fb952d66d4cb1ade70c334fbbb3e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:06.792695 9890 tablet_service.cc:1505] Processing CreateTablet for tablet febc011f42244fdfa1954dbb630c3d0c (DEFAULT_TABLE table=default.Table_0e8bd764e80e46a5b665da0e3e2b677e [id=288ecea80da64655ae6246da518b77b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.793053 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a4118310fd45609f7f0b7d12d253ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.794523 9898 tablet_service.cc:1505] Processing CreateTablet for tablet b57c38c7a97f45feb1b7f90e85577a40 (DEFAULT_TABLE table=default.Table_f433bb2ad7be45fc82c323420fe9808f [id=37247477866142efab09f520c780f952]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.794986 9902 tablet_service.cc:1505] Processing CreateTablet for tablet f2e5b6912c484517afd48f8070455da8 (DEFAULT_TABLE table=default.Table_557acd3dc3344fff87c5b92798fcfa66 [id=307c97840b09416c91f5e3fc0368a600]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.798808 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet ccf10ef559f74e2db28023fd520c8de9 (table default.Table_17e67d6acbdd470d9b351522b7908579 [id=bbdedeb45c464c47b96d078308141f9c]) successfully deleted
W20251027 21:08:06.799047 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 1d1d4031a14e448ea5a60327bd37d003 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d1d4031a14e448ea5a60327bd37d003 already in progress: creating tablet
I20251027 21:08:06.800596 9746 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 994) took 1197 ms. Trace:
I20251027 21:08:06.800670 9746 rpcz_store.cc:276] 1027 21:08:05.603531 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.168594 (+565063us) service_pool.cc:225] Handling call
1027 21:08:06.168974 (+ 380us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.169006 (+ 32us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.795756 (+626750us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.800458 (+ 4702us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.800573 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.800899 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 6d0562f16c30449a8c761320a80f0ae7 (DEFAULT_TABLE table=default.Table_f7947222dc5043359f1891191346329b [id=f541f8840fd341ba80ff1d46c59e3eed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.801007 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 3ecfcff3d55147a2a29e65820de730f5 (DEFAULT_TABLE table=default.Table_2374f13424064567a276cca554d2aff8 [id=a528343b93b245fcb5a8019226c61546]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.801280 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ecfcff3d55147a2a29e65820de730f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.810830 10032 tablet_bootstrap.cc:492] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:06.810963 10032 ts_tablet_manager.cc:1403] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.068s user 0.007s sys 0.003s
I20251027 21:08:06.811619 10032 raft_consensus.cc:359] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.814096 9307 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 6600774e4c7b48479a2fdb02614ba505 (table default.Table_d7530773668f42c5a07a921df02a95ab [id=914b728e358e4fdba6d38d6c7beadaca]). Expected version 1 got 0
I20251027 21:08:06.814417 9307 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 87d5af4607f04f92abcc7292b4176cbd (table default.Table_c78ef9ebe86b4da08443e37c83d4e80d [id=cdb023218cda4e9ea04a226b37a220d9]). Expected version 2 got 0
I20251027 21:08:06.811769 10032 raft_consensus.cc:385] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:06.815294 10032 raft_consensus.cc:740] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:06.815495 10032 consensus_queue.cc:260] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.815954 10032 ts_tablet_manager.cc:1434] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251027 21:08:06.816195 10032 tablet_bootstrap.cc:492] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:06.817723 10032 tablet_bootstrap.cc:654] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:06.825172 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 4a621d5246f14997a2f2952ab640efc8
I20251027 21:08:06.825783 9310 catalog_manager.cc:6719] ReplaceTablet: tablet 4a621d5246f14997a2f2952ab640efc8 deleted and replaced by tablet 01cfe828d88646438ef96f8a7d08f6db
I20251027 21:08:06.826632 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 8dc358445526415d8cd4f1ac6779126d
I20251027 21:08:06.827126 9311 catalog_manager.cc:6719] ReplaceTablet: tablet 8dc358445526415d8cd4f1ac6779126d deleted and replaced by tablet dbf29208d34b4e6f92f5394b14dcc1cc
I20251027 21:08:06.827642 9747 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1004) took 1146 ms. Trace:
I20251027 21:08:06.827719 9747 rpcz_store.cc:276] 1027 21:08:05.681012 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.639705 (+958693us) service_pool.cc:225] Handling call
1027 21:08:06.640027 (+ 322us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.640048 (+ 21us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.667637 (+ 27589us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.827469 (+159832us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.827607 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.830474 9741 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 997) took 1179 ms. Trace:
I20251027 21:08:06.830574 9741 rpcz_store.cc:276] 1027 21:08:05.650778 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.507938 (+857160us) service_pool.cc:225] Handling call
1027 21:08:06.650281 (+142343us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.650301 (+ 20us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.653163 (+ 2862us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.830284 (+177121us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.830452 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26222}
I20251027 21:08:06.831063 9741 tablet_service.cc:1505] Processing CreateTablet for tablet 322337041c954838bc07c43241abf06d (DEFAULT_TABLE table=default.Table_1812ea722de14512bad256f85c0596ab [id=93ac6bededf24c18be9e281c678a7699]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.831475 9736 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1003) took 1161 ms. Trace:
I20251027 21:08:06.831535 9736 rpcz_store.cc:276] 1027 21:08:05.670183 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.637794 (+967611us) service_pool.cc:225] Handling call
1027 21:08:06.638123 (+ 329us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.638142 (+ 19us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.807524 (+169382us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.831350 (+ 23826us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.831464 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21305}
I20251027 21:08:06.831877 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 4c60797e8cd5442b9eef86ee51d4b9c6 (DEFAULT_TABLE table=default.Table_44f9303ecfd745f99df2ffa39d8a0499 [id=344f0961965d4b379ac0ca204ff88500]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.836428 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet db2735604dc14039810df4ecd330c671
I20251027 21:08:06.836845 9306 catalog_manager.cc:6719] ReplaceTablet: tablet db2735604dc14039810df4ecd330c671 deleted and replaced by tablet 1dc45f9d5219461b9fca93a7719d3f6c
I20251027 21:08:06.832235 9751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1013) took 1034 ms. Trace:
I20251027 21:08:06.837353 9751 rpcz_store.cc:276] 1027 21:08:05.798216 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.647144 (+848928us) service_pool.cc:225] Handling call
1027 21:08:06.647435 (+ 291us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.647454 (+ 19us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.718566 (+ 71112us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.832131 (+113565us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.832225 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8734}
W20251027 21:08:06.837697 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fa39697360c46138ef4cd21e8c7a950 consensus state has no leader
I20251027 21:08:06.838119 10032 tablet_bootstrap.cc:492] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:06.838276 10032 ts_tablet_manager.cc:1403] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.004s
I20251027 21:08:06.839071 10032 raft_consensus.cc:359] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.839233 10032 raft_consensus.cc:385] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:06.839274 10032 raft_consensus.cc:740] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:06.839438 10032 consensus_queue.cc:260] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.839866 10032 ts_tablet_manager.cc:1434] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:06.840104 10032 tablet_bootstrap.cc:492] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:06.840550 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet 8dc358445526415d8cd4f1ac6779126d with delete_type TABLET_DATA_DELETED (replaced by tablet dbf29208d34b4e6f92f5394b14dcc1cc) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:06.840721 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet db2735604dc14039810df4ecd330c671 with delete_type TABLET_DATA_DELETED (replaced by tablet 1dc45f9d5219461b9fca93a7719d3f6c) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:06.840396 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a621d5246f14997a2f2952ab640efc8 with delete_type TABLET_DATA_DELETED (replaced by tablet 01cfe828d88646438ef96f8a7d08f6db) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:06.844360 9742 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1005) took 1160 ms. Trace:
I20251027 21:08:06.844439 9742 rpcz_store.cc:276] 1027 21:08:05.683697 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.641041 (+957344us) service_pool.cc:225] Handling call
1027 21:08:06.641334 (+ 293us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.641355 (+ 21us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.644058 (+ 2703us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.789187 (+145129us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.844347 (+ 55160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.844756 9742 tablet_service.cc:1505] Processing CreateTablet for tablet febc011f42244fdfa1954dbb630c3d0c (DEFAULT_TABLE table=default.Table_0e8bd764e80e46a5b665da0e3e2b677e [id=288ecea80da64655ae6246da518b77b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.845657 10031 raft_consensus.cc:359] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.845801 10031 raft_consensus.cc:385] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:06.845841 10031 raft_consensus.cc:740] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:06.846007 10031 consensus_queue.cc:260] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.846341 9312 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_85f8ddead4f94d889f32e6c27e524f21 [id=186c9cce53d74040b9799ea386fdffd7] alter complete (version 5)
I20251027 21:08:06.846905 10032 tablet_bootstrap.cc:654] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:06.850811 10031 ts_tablet_manager.cc:1434] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.153s user 0.001s sys 0.000s
I20251027 21:08:06.851058 10031 tablet_bootstrap.cc:492] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:06.854099 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet f775a7e42bdc4910bdb604b26a28a7f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 11627a84ff9b478c87c323617a09570b) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:06.854467 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76e4fb2bea7f4a4696fe9f6bb892bccc" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:06.854897 9566 leader_election.cc:343] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:06.855020 9566 leader_election.cc:304] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:06.855234 11142 raft_consensus.cc:2749] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:06.855916 10032 tablet_bootstrap.cc:492] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:06.856026 10032 ts_tablet_manager.cc:1403] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
I20251027 21:08:06.856098 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3abc5f78c4e41e29128e2d790319c75" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:06.856731 10032 raft_consensus.cc:359] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.856877 10032 raft_consensus.cc:385] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:06.856933 10032 raft_consensus.cc:740] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
W20251027 21:08:06.857168 9859 leader_election.cc:343] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:06.857139 10032 consensus_queue.cc:260] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.857251 9859 leader_election.cc:304] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:06.857493 11162 raft_consensus.cc:2749] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:06.857693 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f96c21da9944f1ade05d35787d47e2" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:06.857933 10032 ts_tablet_manager.cc:1434] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:06.858194 10032 tablet_bootstrap.cc:492] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:08:06.858326 9711 leader_election.cc:343] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: a26ded6012174348b7295fc7c0e05e6e
W20251027 21:08:06.858664 9859 leader_election.cc:343] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:06.858781 9859 leader_election.cc:304] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:06.859472 11065 raft_consensus.cc:2749] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:06.859668 10032 tablet_bootstrap.cc:654] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:06.860260 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0a6d8856e582400da9228702189fc7a9" } modify_external_catalogs: true
I20251027 21:08:06.860431 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0a6d8856e582400da9228702189fc7a9" } modify_external_catalogs: true
I20251027 21:08:06.860924 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7292bc6c22ca4ab5b8f53a69520833ba" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:06.861423 9710 leader_election.cc:343] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: a26ded6012174348b7295fc7c0e05e6e
I20251027 21:08:06.861507 9710 leader_election.cc:304] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:06.861814 10997 raft_consensus.cc:2749] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:06.862031 9566 leader_election.cc:343] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:06.862115 9566 leader_election.cc:304] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:06.862505 11142 raft_consensus.cc:2749] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:06.863806 10031 tablet_bootstrap.cc:654] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:06.866627 9859 leader_election.cc:343] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:06.866606 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b81999a26dfe4176a646bd057c455f92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:06.866729 9859 leader_election.cc:304] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:06.867040 11177 raft_consensus.cc:2749] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:06.867188 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b81999a26dfe4176a646bd057c455f92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:06.868288 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_1155ecb9419544b7b940e6699cb4464c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:06.868760 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1155ecb9419544b7b940e6699cb4464c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:06.870838 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3bdec7059cee4e748b8e00c5ebdb800a" } new_table_name: "default.Table_0b4e42c5dcdb46209d2970fb037bfba8" modify_external_catalogs: true
I20251027 21:08:06.872879 10043 ts_tablet_manager.cc:1929] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:06.873032 10043 log.cc:1199] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/4debc3f13f3a42939b6b83f6523aab7c
I20251027 21:08:06.873418 10043 ts_tablet_manager.cc:1950] T 4debc3f13f3a42939b6b83f6523aab7c P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:06.875486 9747 tablet_service.cc:1505] Processing CreateTablet for tablet 94de56b0a42b4b98b666f0fde1771e17 (DEFAULT_TABLE table=default.Table_cfb487c77526400b84830a2d722437a7 [id=4920a6ea8aec4b7cb85cfe7cf228f481]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.875746 9751 tablet_service.cc:1505] Processing CreateTablet for tablet 0d75e6a4286e428eb85b24638382d23b (DEFAULT_TABLE table=default.Table_bde50fa24ef54301a7b55d76cf5a9bda [id=25ffc58bf0a54adfa648f55c76f3df44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.876513 10031 tablet_bootstrap.cc:492] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:06.876672 10031 ts_tablet_manager.cc:1403] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20251027 21:08:06.877483 10031 raft_consensus.cc:359] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.877704 10031 raft_consensus.cc:385] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:06.877759 10031 raft_consensus.cc:740] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:06.877951 10031 consensus_queue.cc:260] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.878396 10031 ts_tablet_manager.cc:1434] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:06.878624 10031 tablet_bootstrap.cc:492] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:06.879199 9740 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1015) took 1080 ms. Trace:
I20251027 21:08:06.879289 9740 rpcz_store.cc:276] 1027 21:08:05.798412 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.646505 (+848093us) service_pool.cc:225] Handling call
1027 21:08:06.646785 (+ 280us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.646805 (+ 20us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.650006 (+ 3201us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.879042 (+229036us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.879183 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51878}
I20251027 21:08:06.879923 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet e9edb62a8e4b4dfea6a31a2202545a6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:06.880179 10031 tablet_bootstrap.cc:654] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:06.880151 9602 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1020) took 1173 ms. Trace:
I20251027 21:08:06.880247 9740 tablet_service.cc:1505] Processing CreateTablet for tablet b57c38c7a97f45feb1b7f90e85577a40 (DEFAULT_TABLE table=default.Table_f433bb2ad7be45fc82c323420fe9808f [id=37247477866142efab09f520c780f952]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:06.880266 9602 rpcz_store.cc:276] 1027 21:08:05.706740 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.476045 (+769305us) service_pool.cc:225] Handling call
1027 21:08:06.681604 (+205559us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.681631 (+ 27us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.877012 (+195381us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.880011 (+ 2999us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.880134 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.888377 9886 tablet_service.cc:1552] Processing DeleteTablet for tablet c548d2413a5940c8b114ced667b4dc41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:05 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:06.888541 9886 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c67c5cb3b244feb8478f9594b834fbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:06.888649 9886 tablet_service.cc:1552] Processing DeleteTablet for tablet 17a19ce92da8448bba471f94e3ee4647 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:06.888738 9886 tablet_service.cc:1552] Processing DeleteTablet for tablet 35b99f8c57e847b1848cbdea9e50da11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:06.888823 9886 tablet_service.cc:1552] Processing DeleteTablet for tablet 933fb952d66d4cb1ade70c334fbbb3e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:06.888909 9886 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a621d5246f14997a2f2952ab640efc8 with delete_type TABLET_DATA_DELETED (replaced by tablet 01cfe828d88646438ef96f8a7d08f6db) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:06.888990 9886 tablet_service.cc:1552] Processing DeleteTablet for tablet 8dc358445526415d8cd4f1ac6779126d with delete_type TABLET_DATA_DELETED (replaced by tablet dbf29208d34b4e6f92f5394b14dcc1cc) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:06.889073 9886 tablet_service.cc:1552] Processing DeleteTablet for tablet db2735604dc14039810df4ecd330c671 with delete_type TABLET_DATA_DELETED (replaced by tablet 1dc45f9d5219461b9fca93a7719d3f6c) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:06.889161 9886 tablet_service.cc:1552] Processing DeleteTablet for tablet f6789dc2f7cb4a3c826ac507d5c9a239 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:06.889249 9886 tablet_service.cc:1552] Processing DeleteTablet for tablet 7292bc6c22ca4ab5b8f53a69520833ba with delete_type TABLET_DATA_DELETED (replaced by tablet 37ae3b6bf590473e8e09092fc5f7df1b) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:06.889338 9886 tablet_service.cc:1552] Processing DeleteTablet for tablet 07a9c3b7ef1548ec9d5ae9f20545f263 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:06.889429 9886 tablet_service.cc:1552] Processing DeleteTablet for tablet b3abc5f78c4e41e29128e2d790319c75 with delete_type TABLET_DATA_DELETED (replaced by tablet ccd419ff40a94f9587b024ec87a2517e) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:06.889508 9886 tablet_service.cc:1552] Processing DeleteTablet for tablet a2d26784bbb24ae3893103802c41affb with delete_type TABLET_DATA_DELETED (replaced by tablet 9ace4188d3494a05a5678381f8d8fd17) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:06.897621 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 322337041c954838bc07c43241abf06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.897855 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d75e6a4286e428eb85b24638382d23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.898217 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d332d8e8f0c949589d1fd29b4bd7f076. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.898568 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef7d8549ecc4546bc03c8e600c54c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.898923 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c05b69509414fe8bfc809cd307d5f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.899238 9900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0032dd868c4dc79ea87036264434ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.899538 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7057c55a820d4e379fe1d9abfa353821. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.899840 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 278d8f448ef144148cabfc3a57bd24fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.900183 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eefbbf73da40432ba2ad54764a8ace91. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.900529 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea02ab2fed634dbcba3ca8b04808b694. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.900840 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ecfcff3d55147a2a29e65820de730f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.901166 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94de56b0a42b4b98b666f0fde1771e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.901783 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet febc011f42244fdfa1954dbb630c3d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.903158 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c60797e8cd5442b9eef86ee51d4b9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.906351 9901 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 1014) took 1208 ms. Trace:
I20251027 21:08:06.906445 9901 rpcz_store.cc:276] 1027 21:08:05.697805 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.570761 (+872956us) service_pool.cc:225] Handling call
1027 21:08:06.672808 (+102047us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.672843 (+ 35us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.901445 (+228602us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.906116 (+ 4671us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.906328 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.907377 10041 ts_tablet_manager.cc:1929] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:06.907483 10041 log.cc:1199] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/67bb12de9f044c73ae9ab26d10498c90
I20251027 21:08:06.908097 10041 ts_tablet_manager.cc:1950] T 67bb12de9f044c73ae9ab26d10498c90 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:06.919134 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a4118310fd45609f7f0b7d12d253ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.919601 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d0562f16c30449a8c761320a80f0ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.921794 9604 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1021) took 1215 ms. Trace:
I20251027 21:08:06.921876 9604 rpcz_store.cc:276] 1027 21:08:05.706796 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.479017 (+772221us) service_pool.cc:225] Handling call
1027 21:08:06.686236 (+207219us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.686259 (+ 23us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.916552 (+230293us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.921626 (+ 5074us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.921777 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.923516 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b94af6879cd940829a7f531277d5544c" } modify_external_catalogs: true
I20251027 21:08:06.923723 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b94af6879cd940829a7f531277d5544c" } modify_external_catalogs: true
I20251027 21:08:06.924253 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e4b781970438453eb7f7f4b885331bfb" } modify_external_catalogs: true
I20251027 21:08:06.924422 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e4b781970438453eb7f7f4b885331bfb" } modify_external_catalogs: true
I20251027 21:08:06.924872 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5a6211c8465040dbad1cc86377a370ea" } modify_external_catalogs: true
I20251027 21:08:06.924993 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5a6211c8465040dbad1cc86377a370ea" } modify_external_catalogs: true
I20251027 21:08:06.925422 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e8162d3f2ed747b3802569f1e9ab2115" } new_table_name: "default.Table_b4ce658a48f944e19495e91e2b302308" modify_external_catalogs: true
I20251027 21:08:06.929597 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_feecf5aa70d7443293e8b3af119d90e4" } modify_external_catalogs: true
I20251027 21:08:06.929759 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_feecf5aa70d7443293e8b3af119d90e4" } modify_external_catalogs: true
I20251027 21:08:06.930258 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b57c38c7a97f45feb1b7f90e85577a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.930761 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2e5b6912c484517afd48f8070455da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.933432 9608 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1025) took 1203 ms. Trace:
I20251027 21:08:06.933511 9608 rpcz_store.cc:276] 1027 21:08:05.730428 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.519740 (+789312us) service_pool.cc:225] Handling call
1027 21:08:06.690675 (+170935us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.690693 (+ 18us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.889612 (+198919us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.933266 (+ 43654us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.933414 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.936093 9598 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1027) took 1189 ms. Trace:
I20251027 21:08:06.936168 9598 rpcz_store.cc:276] 1027 21:08:05.746225 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.534276 (+788051us) service_pool.cc:225] Handling call
1027 21:08:06.691036 (+156760us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.691056 (+ 20us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.913474 (+222418us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.935949 (+ 22475us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.936076 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.940282 9889 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 1015) took 1242 ms. Trace:
I20251027 21:08:06.940366 9889 rpcz_store.cc:276] 1027 21:08:05.697893 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.579271 (+881378us) service_pool.cc:225] Handling call
1027 21:08:06.682293 (+103022us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.682343 (+ 50us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.937322 (+254979us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.940116 (+ 2794us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.940264 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1408}
I20251027 21:08:06.959314 11177 raft_consensus.cc:493] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:06.959441 11177 raft_consensus.cc:515] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.960016 11177 leader_election.cc:290] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:06.960659 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "588e2f58f07b4a32a78dffd5b92ceb12" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:06.961980 9859 leader_election.cc:343] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:06.962195 9820 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:08:06.987505 10032 tablet_bootstrap.cc:492] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:06.987644 10032 ts_tablet_manager.cc:1403] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.130s user 0.011s sys 0.019s
I20251027 21:08:06.988340 10032 raft_consensus.cc:359] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.988480 10032 raft_consensus.cc:385] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:06.988531 10032 raft_consensus.cc:740] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:06.988736 10032 consensus_queue.cc:260] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:06.989203 10032 ts_tablet_manager.cc:1434] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251027 21:08:06.989450 10032 tablet_bootstrap.cc:492] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:06.989574 9900 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 1018) took 1291 ms. Trace:
I20251027 21:08:06.989665 9900 rpcz_store.cc:276] 1027 21:08:05.698122 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.590649 (+892527us) service_pool.cc:225] Handling call
1027 21:08:06.742058 (+151409us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.742080 (+ 22us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.964597 (+222517us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.989410 (+ 24813us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:06.989556 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:06.994542 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d0562f16c30449a8c761320a80f0ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:06.994535 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "588e2f58f07b4a32a78dffd5b92ceb12" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:06.995177 10032 tablet_bootstrap.cc:654] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:06.998616 9860 leader_election.cc:343] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:06.998773 9860 leader_election.cc:304] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:06.999248 11177 raft_consensus.cc:2749] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:07.002920 9615 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1023) took 1296 ms. Trace:
I20251027 21:08:07.003005 9615 rpcz_store.cc:276] 1027 21:08:05.706938 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.512110 (+805172us) service_pool.cc:225] Handling call
1027 21:08:06.690311 (+178201us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.690331 (+ 20us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.900924 (+210593us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.002744 (+101820us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.002902 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.005192 11177 raft_consensus.cc:493] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.005306 11177 raft_consensus.cc:515] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.006085 11177 leader_election.cc:290] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:07.006624 9599 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1026) took 1274 ms. Trace:
I20251027 21:08:07.006723 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b8156d9ab2e4dfba383ddc72137aab4" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:07.006705 9599 rpcz_store.cc:276] 1027 21:08:05.731658 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.523174 (+791516us) service_pool.cc:225] Handling call
1027 21:08:06.690860 (+167686us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.690877 (+ 17us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.896813 (+205936us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.004968 (+108155us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.006591 (+ 1623us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251027 21:08:07.007300 9859 leader_election.cc:343] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.007558 11177 raft_consensus.cc:493] T 882c58a45d46402eabe96591a7c1de71 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.007583 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b8156d9ab2e4dfba383ddc72137aab4" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:07.007673 11177 raft_consensus.cc:515] T 882c58a45d46402eabe96591a7c1de71 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.008345 11177 leader_election.cc:290] T 882c58a45d46402eabe96591a7c1de71 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.008896 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882c58a45d46402eabe96591a7c1de71" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:07.009341 9859 leader_election.cc:343] T 882c58a45d46402eabe96591a7c1de71 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:07.009825 9860 leader_election.cc:343] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.009840 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882c58a45d46402eabe96591a7c1de71" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:07.009922 9860 leader_election.cc:304] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:07.010387 9860 leader_election.cc:343] T 882c58a45d46402eabe96591a7c1de71 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.010469 9860 leader_election.cc:304] T 882c58a45d46402eabe96591a7c1de71 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.010766 11162 raft_consensus.cc:2749] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:07.010929 11162 raft_consensus.cc:2749] T 882c58a45d46402eabe96591a7c1de71 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:07.014406 9609 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1029) took 1262 ms. Trace:
I20251027 21:08:07.014494 9609 rpcz_store.cc:276] 1027 21:08:05.751897 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.540508 (+788611us) service_pool.cc:225] Handling call
1027 21:08:06.692626 (+152118us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.692649 (+ 23us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.904495 (+211846us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.014274 (+109779us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.014391 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.021144 9594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1028) took 1272 ms. Trace:
I20251027 21:08:07.021217 9594 rpcz_store.cc:276] 1027 21:08:05.748886 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.537464 (+788578us) service_pool.cc:225] Handling call
1027 21:08:06.691736 (+154272us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.691757 (+ 21us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.908519 (+216762us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.021006 (+112487us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.021129 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.023357 9603 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1022) took 1316 ms. Trace:
I20251027 21:08:07.023430 9603 rpcz_store.cc:276] 1027 21:08:05.706861 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.481118 (+774257us) service_pool.cc:225] Handling call
1027 21:08:06.686436 (+205318us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.686455 (+ 19us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.892983 (+206528us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.023234 (+130251us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.023341 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.025213 9903 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 1019) took 1326 ms. Trace:
I20251027 21:08:07.025291 9903 rpcz_store.cc:276] 1027 21:08:05.698253 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.593483 (+895230us) service_pool.cc:225] Handling call
1027 21:08:06.742114 (+148631us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.742143 (+ 29us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.960639 (+218496us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.025052 (+ 64413us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.025196 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.028946 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 916) took 3409 ms. Trace:
I20251027 21:08:07.029165 9892 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34644 (request call id 1016) took 1331 ms. Trace:
I20251027 21:08:07.030315 9892 rpcz_store.cc:276] 1027 21:08:05.697979 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.582783 (+884804us) service_pool.cc:225] Handling call
1027 21:08:06.742513 (+159730us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.742547 (+ 34us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.968318 (+225771us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.029021 (+ 60703us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.029150 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.031013 10041 rpcz_store.cc:276] 1027 21:08:03.619157 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:03.627094 (+ 7937us) service_pool.cc:225] Handling call
1027 21:08:06.632536 (+3005442us) ts_tablet_manager.cc:1151] Deleting tablet 67bb12de9f044c73ae9ab26d10498c90
1027 21:08:06.632542 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:06.761961 (+129419us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.907278 (+145317us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.028926 (+121648us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3005184}
I20251027 21:08:07.031649 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 67bb12de9f044c73ae9ab26d10498c90 (table default.Table_0ce16ec9dc8b44eab80cedc074577cba [id=c22fee46f319480e88ab7745f6a95126]) successfully deleted
I20251027 21:08:07.031740 9607 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1024) took 1324 ms. Trace:
I20251027 21:08:07.031809 9607 rpcz_store.cc:276] 1027 21:08:05.706989 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.516938 (+809949us) service_pool.cc:225] Handling call
1027 21:08:06.690502 (+173564us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.690519 (+ 17us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:06.883475 (+192956us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.031605 (+148130us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.031725 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.032006 10041 tablet_replica.cc:333] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:07.032159 10041 raft_consensus.cc:2243] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:07.032274 10041 raft_consensus.cc:2272] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:07.032745 10041 ts_tablet_manager.cc:1916] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:07.037294 10029 tablet_bootstrap.cc:492] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:07.037443 10029 ts_tablet_manager.cc:1403] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.352s user 0.003s sys 0.007s
I20251027 21:08:07.038092 10029 raft_consensus.cc:359] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:07.038254 10029 raft_consensus.cc:385] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.038296 10029 raft_consensus.cc:740] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.044713 10029 consensus_queue.cc:260] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:07.045213 10029 ts_tablet_manager.cc:1434] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251027 21:08:07.046288 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc461f51c704e63a67946df02ca90d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.046802 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3d07b891a248efb0860cd000e44432. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.047185 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea02ab2fed634dbcba3ca8b04808b694. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.051214 9750 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1012) took 1253 ms. Trace:
I20251027 21:08:07.051291 9750 rpcz_store.cc:276] 1027 21:08:05.798057 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.696852 (+898795us) service_pool.cc:225] Handling call
1027 21:08:06.782773 (+ 85921us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.782802 (+ 29us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.041622 (+258820us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.048058 (+ 6436us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.051187 (+ 3129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.051734 9750 tablet_service.cc:1505] Processing CreateTablet for tablet f2e5b6912c484517afd48f8070455da8 (DEFAULT_TABLE table=default.Table_557acd3dc3344fff87c5b92798fcfa66 [id=307c97840b09416c91f5e3fc0368a600]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.054411 10829 raft_consensus.cc:493] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.054538 10829 raft_consensus.cc:515] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.055128 10029 tablet_bootstrap.cc:492] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:07.055163 10829 leader_election.cc:290] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
W20251027 21:08:07.056224 9711 leader_election.cc:343] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 8b30052ff19744a9bf4bfaaa1bc77b36
I20251027 21:08:07.056651 9739 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1008) took 1259 ms. Trace:
I20251027 21:08:07.056643 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b30052ff19744a9bf4bfaaa1bc77b36" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:07.056799 9635 raft_consensus.cc:2468] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:08:07.056743 9739 rpcz_store.cc:276] 1027 21:08:05.797555 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.662686 (+865131us) service_pool.cc:225] Handling call
1027 21:08:06.664880 (+ 2194us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.664914 (+ 34us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.034423 (+369509us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.050043 (+ 15620us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.056635 (+ 6592us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.057129 9739 tablet_service.cc:1505] Processing CreateTablet for tablet 6d0562f16c30449a8c761320a80f0ae7 (DEFAULT_TABLE table=default.Table_f7947222dc5043359f1891191346329b [id=f541f8840fd341ba80ff1d46c59e3eed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.057894 9710 leader_election.cc:304] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:07.058391 10829 raft_consensus.cc:2804] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:07.058476 10829 raft_consensus.cc:493] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:07.058545 10829 raft_consensus.cc:3060] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:07.061192 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c60797e8cd5442b9eef86ee51d4b9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.061877 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet febc011f42244fdfa1954dbb630c3d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.062378 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94de56b0a42b4b98b666f0fde1771e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.062636 10029 tablet_bootstrap.cc:654] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.063591 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d75e6a4286e428eb85b24638382d23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.064368 10032 tablet_bootstrap.cc:492] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:07.064484 10032 ts_tablet_manager.cc:1403] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.075s user 0.007s sys 0.003s
I20251027 21:08:07.065165 10032 raft_consensus.cc:359] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.065290 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 322337041c954838bc07c43241abf06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.065307 10032 raft_consensus.cc:385] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.065351 10032 raft_consensus.cc:740] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.065429 10997 raft_consensus.cc:493] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.065531 10997 raft_consensus.cc:515] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.065536 10032 consensus_queue.cc:260] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.066162 10997 leader_election.cc:290] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
W20251027 21:08:07.067406 9711 leader_election.cc:343] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: c3b9a461567a48fcb4af514ecdf618b4
I20251027 21:08:07.067430 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3b9a461567a48fcb4af514ecdf618b4" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:07.067660 9635 raft_consensus.cc:2468] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:08:07.070398 9710 leader_election.cc:304] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:07.070801 10997 raft_consensus.cc:2804] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:07.070881 10997 raft_consensus.cc:493] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:07.070945 10997 raft_consensus.cc:3060] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:07.074672 10032 ts_tablet_manager.cc:1434] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251027 21:08:07.075033 10032 tablet_bootstrap.cc:492] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:07.077077 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b57c38c7a97f45feb1b7f90e85577a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.078375 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2e5b6912c484517afd48f8070455da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.093876 9308 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 6600774e4c7b48479a2fdb02614ba505 (table default.Table_d7530773668f42c5a07a921df02a95ab [id=914b728e358e4fdba6d38d6c7beadaca]). Expected version 1 got 0
I20251027 21:08:07.095430 9894 tablet_service.cc:1505] Processing CreateTablet for tablet e4b84d8c0ee74dcbaee8aeb49aa30c57 (DEFAULT_TABLE table=default.Table_877147b58eff438190609ce0142c08f5 [id=05da7a8eb8ea441d90763f0bf6abd69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.096084 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 53301ba720544c8cbfb075bfc5a7dbb8 (DEFAULT_TABLE table=default.Table_66beca448bc04e81aecbc35fd309fe12 [id=f126c17de97148838027047872fb8aea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.096941 9898 tablet_service.cc:1505] Processing CreateTablet for tablet f1a931c36fec45029715b01754d0b161 (DEFAULT_TABLE table=default.Table_2aad78a009654cb88858d59b51ee78cb [id=c0406adb27444330b8e387a2d1fb7e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.097573 9902 tablet_service.cc:1505] Processing CreateTablet for tablet a794b2a2586c483c95b3f85c20499de0 (DEFAULT_TABLE table=default.Table_3b772db224e14e8993234e232491c8c9 [id=51bee83a7dae40e68b5bf226cf23128f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.099009 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 58c9cd6828a1497babb9c91bee938041 (DEFAULT_TABLE table=default.Table_b6554b3e96a841fb975ff74ff767069a [id=b6464fe950c5487fb13f6763626d8b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.099769 9903 tablet_service.cc:1505] Processing CreateTablet for tablet b070d77b63ff4557883770df9ff4c55e (DEFAULT_TABLE table=default.Table_653e76a8cae24ff38c2eee3d9322aa6e [id=0d70035dcbd64d83b303f36b7306efc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.100822 10032 tablet_bootstrap.cc:654] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.105007 9602 tablet_service.cc:1505] Processing CreateTablet for tablet b070d77b63ff4557883770df9ff4c55e (DEFAULT_TABLE table=default.Table_653e76a8cae24ff38c2eee3d9322aa6e [id=0d70035dcbd64d83b303f36b7306efc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.107580 10032 tablet_bootstrap.cc:492] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:07.107750 10032 ts_tablet_manager.cc:1403] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
I20251027 21:08:07.107925 10829 raft_consensus.cc:515] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.108616 10032 raft_consensus.cc:359] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.108814 10032 raft_consensus.cc:385] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.108788 10829 leader_election.cc:290] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:07.108873 10032 raft_consensus.cc:740] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.109109 10032 consensus_queue.cc:260] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.109733 10032 ts_tablet_manager.cc:1434] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:07.110057 10032 tablet_bootstrap.cc:492] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:07.113153 9744 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1014) took 1314 ms. Trace:
I20251027 21:08:07.113230 9744 rpcz_store.cc:276] 1027 21:08:05.798310 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.697371 (+899061us) service_pool.cc:225] Handling call
1027 21:08:06.784124 (+ 86753us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.784155 (+ 31us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.075707 (+291552us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.113003 (+ 37296us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.113137 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.113500 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet c548d2413a5940c8b114ced667b4dc41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:05 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.113638 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c67c5cb3b244feb8478f9594b834fbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.113766 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet 17a19ce92da8448bba471f94e3ee4647 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.113888 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet 35b99f8c57e847b1848cbdea9e50da11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.113991 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet 933fb952d66d4cb1ade70c334fbbb3e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.114089 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a621d5246f14997a2f2952ab640efc8 with delete_type TABLET_DATA_DELETED (replaced by tablet 01cfe828d88646438ef96f8a7d08f6db) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.116319 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b30052ff19744a9bf4bfaaa1bc77b36" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:07.116573 9635 raft_consensus.cc:3060] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:07.118582 11160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4b8b31d47da48aaa386a8e64a5749c2. Latest consensus state: current_term: 3 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:08:07.118813 11160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 21:08:07.120479 11160 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a4b8b31d47da48aaa386a8e64a5749c2)
I20251027 21:08:07.120682 11160 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } }
I20251027 21:08:07.121250 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d0562f16c30449a8c761320a80f0ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.127030 10029 tablet_bootstrap.cc:492] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:07.127170 10029 ts_tablet_manager.cc:1403] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.082s user 0.002s sys 0.009s
I20251027 21:08:07.128021 10029 raft_consensus.cc:359] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.128201 10029 raft_consensus.cc:385] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.128255 10029 raft_consensus.cc:740] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
W20251027 21:08:07.128955 9711 leader_election.cc:343] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 8b30052ff19744a9bf4bfaaa1bc77b36
I20251027 21:08:07.129133 11177 raft_consensus.cc:493] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.129249 11177 raft_consensus.cc:515] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:07.130060 11177 leader_election.cc:290] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.130589 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96a680e8f16343b1908829a67e960b60" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:07.130739 9634 raft_consensus.cc:2468] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 0.
I20251027 21:08:07.131147 9860 leader_election.cc:304] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:07.131428 11177 raft_consensus.cc:2804] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:07.131498 11177 raft_consensus.cc:493] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:07.131582 11177 raft_consensus.cc:3060] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:07.132329 11160 leader_election.cc:290] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a4b8b31d47da48aaa386a8e64a5749c2 (127.9.14.126:34057), 18b15927ce5744ad8ba81dda52725c11 (127.9.14.125:46313)
I20251027 21:08:07.138502 10029 consensus_queue.cc:260] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.139973 10032 tablet_bootstrap.cc:654] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.140345 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96a680e8f16343b1908829a67e960b60" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:07.140545 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 57b8f69381904f96a0bfa96c985e8aa9 (DEFAULT_TABLE table=default.Table_3b8f6db604d34f05b8dde4b29bd4cd3f [id=80d1ab0a30c64b5db860a6082823043b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.141544 10029 ts_tablet_manager.cc:1434] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20251027 21:08:07.141860 10029 tablet_bootstrap.cc:492] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:07.142436 9609 tablet_service.cc:1505] Processing CreateTablet for tablet a794b2a2586c483c95b3f85c20499de0 (DEFAULT_TABLE table=default.Table_3b772db224e14e8993234e232491c8c9 [id=51bee83a7dae40e68b5bf226cf23128f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.142868 9594 tablet_service.cc:1505] Processing CreateTablet for tablet f1a931c36fec45029715b01754d0b161 (DEFAULT_TABLE table=default.Table_2aad78a009654cb88858d59b51ee78cb [id=c0406adb27444330b8e387a2d1fb7e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.143551 9308 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet ca477937278f45b4b115fec715f32a92 (table default.Table_69627cab65c2407b9226a05106b7bdf4 [id=fc5a0b5a96e6400da438307d26f73614]). Expected version 3 got 0
I20251027 21:08:07.143817 9308 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet d56d323ef0e74344a479940b93fea11a (table default.Table_cf11a8c344c544c5ba54672fee38e4aa [id=71e02b21c9654e45a41d3e450ac3fbe1]). Expected version 2 got 0
I20251027 21:08:07.143860 10029 tablet_bootstrap.cc:654] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:07.144449 9859 leader_election.cc:343] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.145077 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 58c9cd6828a1497babb9c91bee938041 (DEFAULT_TABLE table=default.Table_b6554b3e96a841fb975ff74ff767069a [id=b6464fe950c5487fb13f6763626d8b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.145710 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 57b8f69381904f96a0bfa96c985e8aa9 (DEFAULT_TABLE table=default.Table_3b8f6db604d34f05b8dde4b29bd4cd3f [id=80d1ab0a30c64b5db860a6082823043b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.151089 10032 tablet_bootstrap.cc:492] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:07.151216 10032 ts_tablet_manager.cc:1403] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.003s
I20251027 21:08:07.151948 9603 tablet_service.cc:1505] Processing CreateTablet for tablet 53301ba720544c8cbfb075bfc5a7dbb8 (DEFAULT_TABLE table=default.Table_66beca448bc04e81aecbc35fd309fe12 [id=f126c17de97148838027047872fb8aea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.152063 10032 raft_consensus.cc:359] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.152243 10032 raft_consensus.cc:385] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.152302 10032 raft_consensus.cc:740] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.152516 10032 consensus_queue.cc:260] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.153038 10032 ts_tablet_manager.cc:1434] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:07.153295 10032 tablet_bootstrap.cc:492] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:07.155433 9307 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 90f96c21da9944f1ade05d35787d47e2 (table default.Table_cdf514bc48a4478da802d1bbfd0f1c71 [id=ec9aa8523bf44f2f9486b14f4cf7639c]). Expected version 2 got 0
I20251027 21:08:07.155642 9307 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet d56d323ef0e74344a479940b93fea11a (table default.Table_cf11a8c344c544c5ba54672fee38e4aa [id=71e02b21c9654e45a41d3e450ac3fbe1]). Expected version 2 got 0
I20251027 21:08:07.155788 9307 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet ca477937278f45b4b115fec715f32a92 (table default.Table_69627cab65c2407b9226a05106b7bdf4 [id=fc5a0b5a96e6400da438307d26f73614]). Expected version 3 got 0
I20251027 21:08:07.155995 9307 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 67cb71ce43244d8ab9e71ead6bb315c9 (table default.Table_6327cacf4ed54d56b522291ad9594784 [id=3eb54d245ec64ee48022c541c2e49b17]). Expected version 2 got 0
I20251027 21:08:07.159459 10029 tablet_bootstrap.cc:492] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:07.159596 10029 ts_tablet_manager.cc:1403] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20251027 21:08:07.160408 10029 raft_consensus.cc:359] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.160594 10029 raft_consensus.cc:385] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.160650 10029 raft_consensus.cc:740] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.164678 9307 catalog_manager.cc:5649] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:07.165606 9307 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet aaa8910d9fdb4146945621ec4ebf969e (table default.Table_d8da07f0725347338012d992737c5d04 [id=bb676562b97f48088767941239cfbf33]). Expected version 2 got 0
I20251027 21:08:07.166868 9970 heartbeater.cc:344] Connected to a master server at 127.9.14.124:45071
I20251027 21:08:07.167830 11162 raft_consensus.cc:493] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.167945 11162 raft_consensus.cc:515] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.168529 11162 leader_election.cc:290] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.168507 10029 consensus_queue.cc:260] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.169076 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ed4f9c61ad54e1292281b30bcd19533" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:07.169395 10029 ts_tablet_manager.cc:1434] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
W20251027 21:08:07.169523 9860 leader_election.cc:343] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.169679 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d75e6a4286e428eb85b24638382d23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.169688 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ed4f9c61ad54e1292281b30bcd19533" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:07.171168 10032 tablet_bootstrap.cc:654] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.171689 10970 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45376 (request call id 0) took 2799 ms. Trace:
I20251027 21:08:07.173306 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet febc011f42244fdfa1954dbb630c3d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.168388 9607 tablet_service.cc:1505] Processing CreateTablet for tablet e4b84d8c0ee74dcbaee8aeb49aa30c57 (DEFAULT_TABLE table=default.Table_877147b58eff438190609ce0142c08f5 [id=05da7a8eb8ea441d90763f0bf6abd69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.176435 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b57c38c7a97f45feb1b7f90e85577a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.178972 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53301ba720544c8cbfb075bfc5a7dbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.182332 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet 8dc358445526415d8cd4f1ac6779126d with delete_type TABLET_DATA_DELETED (replaced by tablet dbf29208d34b4e6f92f5394b14dcc1cc) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.182525 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet db2735604dc14039810df4ecd330c671 with delete_type TABLET_DATA_DELETED (replaced by tablet 1dc45f9d5219461b9fca93a7719d3f6c) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.182835 9744 tablet_service.cc:1505] Processing CreateTablet for tablet e4b84d8c0ee74dcbaee8aeb49aa30c57 (DEFAULT_TABLE table=default.Table_877147b58eff438190609ce0142c08f5 [id=05da7a8eb8ea441d90763f0bf6abd69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.184996 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b070d77b63ff4557883770df9ff4c55e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.185611 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a931c36fec45029715b01754d0b161. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.186077 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a794b2a2586c483c95b3f85c20499de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.186599 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58c9cd6828a1497babb9c91bee938041. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.187088 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57b8f69381904f96a0bfa96c985e8aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.187841 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4b84d8c0ee74dcbaee8aeb49aa30c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.188619 9635 raft_consensus.cc:2468] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:07.189059 9710 leader_election.cc:304] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:07.189415 10829 raft_consensus.cc:2804] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:07.189520 10829 raft_consensus.cc:697] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:07.189759 10829 consensus_queue.cc:237] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.192237 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 943) took 2426 ms. Trace:
I20251027 21:08:07.192338 10043 rpcz_store.cc:276] 1027 21:08:04.765317 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.765551 (+ 234us) service_pool.cc:225] Handling call
1027 21:08:06.687433 (+1921882us) ts_tablet_manager.cc:1151] Deleting tablet 4debc3f13f3a42939b6b83f6523aab7c
1027 21:08:06.687443 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:06.776448 (+ 89005us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:06.872713 (+ 96265us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.192198 (+319485us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1921747}
I20251027 21:08:07.192581 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b070d77b63ff4557883770df9ff4c55e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.192965 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 945) took 2406 ms. Trace:
I20251027 21:08:07.193073 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a931c36fec45029715b01754d0b161. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.193048 10043 rpcz_store.cc:276] 1027 21:08:04.786066 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.786468 (+ 402us) service_pool.cc:225] Handling call
1027 21:08:07.192809 (+2406341us) ts_tablet_manager.cc:1151] Deleting tablet 1d1d4031a14e448ea5a60327bd37d003
1027 21:08:07.192819 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:07.192949 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2406130}
I20251027 21:08:07.193414 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a794b2a2586c483c95b3f85c20499de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.193755 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58c9cd6828a1497babb9c91bee938041. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.193733 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 946) took 2399 ms. Trace:
I20251027 21:08:07.193810 10043 rpcz_store.cc:276] 1027 21:08:04.794553 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.794602 (+ 49us) service_pool.cc:225] Handling call
1027 21:08:07.193600 (+2398998us) ts_tablet_manager.cc:1151] Deleting tablet ee0c598904004dd18df058da483aa45c
1027 21:08:07.193612 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:07.193717 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2398802}
I20251027 21:08:07.194108 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53301ba720544c8cbfb075bfc5a7dbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.194233 10043 tablet_replica.cc:333] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:07.194478 10043 raft_consensus.cc:2243] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:07.194612 11177 raft_consensus.cc:515] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:07.195034 10043 raft_consensus.cc:2272] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:07.195677 10043 ts_tablet_manager.cc:1916] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:07.196012 11177 leader_election.cc:290] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.196699 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96a680e8f16343b1908829a67e960b60" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:07.196751 10997 raft_consensus.cc:515] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.196873 9635 raft_consensus.cc:3060] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:07.200209 9737 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1007) took 1402 ms. Trace:
I20251027 21:08:07.200310 9737 rpcz_store.cc:276] 1027 21:08:05.797438 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.656183 (+858745us) service_pool.cc:225] Handling call
1027 21:08:06.656471 (+ 288us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.656493 (+ 22us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.076265 (+419772us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.200059 (+123794us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.200194 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.200781 10029 tablet_bootstrap.cc:492] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:07.200714 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 53301ba720544c8cbfb075bfc5a7dbb8 (DEFAULT_TABLE table=default.Table_66beca448bc04e81aecbc35fd309fe12 [id=f126c17de97148838027047872fb8aea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.203431 9745 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1000) took 1545 ms. Trace:
I20251027 21:08:07.203500 9745 rpcz_store.cc:276] 1027 21:08:05.658045 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.525027 (+866982us) service_pool.cc:225] Handling call
1027 21:08:06.664578 (+139551us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.664610 (+ 32us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.076009 (+411399us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.203297 (+127288us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.203417 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.203899 9745 tablet_service.cc:1505] Processing CreateTablet for tablet f1a931c36fec45029715b01754d0b161 (DEFAULT_TABLE table=default.Table_2aad78a009654cb88858d59b51ee78cb [id=c0406adb27444330b8e387a2d1fb7e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.205662 10029 tablet_bootstrap.cc:654] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.206552 9748 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1001) took 1542 ms. Trace:
I20251027 21:08:07.206632 9748 rpcz_store.cc:276] 1027 21:08:05.664205 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.559940 (+895735us) service_pool.cc:225] Handling call
1027 21:08:06.697078 (+137138us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.780755 (+ 83677us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.076527 (+295772us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.206433 (+129906us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.206540 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.206948 9748 tablet_service.cc:1505] Processing CreateTablet for tablet a794b2a2586c483c95b3f85c20499de0 (DEFAULT_TABLE table=default.Table_3b772db224e14e8993234e232491c8c9 [id=51bee83a7dae40e68b5bf226cf23128f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.210005 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3b9a461567a48fcb4af514ecdf618b4" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:07.210193 9634 raft_consensus.cc:3060] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:07.213889 10829 consensus_queue.cc:1048] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:07.215456 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 4debc3f13f3a42939b6b83f6523aab7c (table default.Table_4bd57498972a413ca2bb7de2cd588e8b [id=5ebbc228b61046f8aef2d6fe13f12e57]) successfully deleted
W20251027 21:08:07.215776 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 1d1d4031a14e448ea5a60327bd37d003 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d1d4031a14e448ea5a60327bd37d003 already in progress: creating tablet
W20251027 21:08:07.215983 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet ee0c598904004dd18df058da483aa45c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee0c598904004dd18df058da483aa45c already in progress: creating tablet
I20251027 21:08:07.216148 10997 leader_election.cc:290] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:07.217757 10029 tablet_bootstrap.cc:492] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:07.217866 10029 ts_tablet_manager.cc:1403] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20251027 21:08:07.223011 10032 tablet_bootstrap.cc:492] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:07.223033 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57b8f69381904f96a0bfa96c985e8aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.223136 10032 ts_tablet_manager.cc:1403] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.070s user 0.005s sys 0.001s
I20251027 21:08:07.223827 10032 raft_consensus.cc:359] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.223960 10032 raft_consensus.cc:385] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.223999 10032 raft_consensus.cc:740] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.224169 10032 consensus_queue.cc:260] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.224577 10032 ts_tablet_manager.cc:1434] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:07.224812 10032 tablet_bootstrap.cc:492] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:07.226881 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96a680e8f16343b1908829a67e960b60" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:07.219748 11097 raft_consensus.cc:493] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.227540 11097 raft_consensus.cc:515] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.228094 11097 leader_election.cc:290] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.228825 9907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6b58b1f4f63461aa96e416cae879be8" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:07.228989 9907 raft_consensus.cc:2468] T e6b58b1f4f63461aa96e416cae879be8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:07.229367 9568 leader_election.cc:304] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
W20251027 21:08:07.229681 9859 leader_election.cc:343] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.229866 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6b58b1f4f63461aa96e416cae879be8" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:07.230000 9711 leader_election.cc:343] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: c3b9a461567a48fcb4af514ecdf618b4
W20251027 21:08:07.230311 9566 leader_election.cc:343] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.230613 11097 raft_consensus.cc:2804] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:07.230681 11097 raft_consensus.cc:493] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:07.230741 11097 raft_consensus.cc:3060] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:07.218627 10029 raft_consensus.cc:359] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.231444 10029 raft_consensus.cc:385] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.231604 10029 raft_consensus.cc:740] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.231990 10029 consensus_queue.cc:260] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.232826 10029 ts_tablet_manager.cc:1434] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251027 21:08:07.233467 10029 tablet_bootstrap.cc:492] T 29693cf06bc24930870dc60457126519 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:07.235160 10029 tablet_bootstrap.cc:654] T 29693cf06bc24930870dc60457126519 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.241510 10032 tablet_bootstrap.cc:654] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.242259 11162 raft_consensus.cc:493] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.242381 11162 raft_consensus.cc:515] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.242995 11162 leader_election.cc:290] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.243137 11162 raft_consensus.cc:493] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.243239 11162 raft_consensus.cc:515] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.243722 11162 leader_election.cc:290] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:07.248188 10029 tablet_bootstrap.cc:492] T 29693cf06bc24930870dc60457126519 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:07.248287 10029 ts_tablet_manager.cc:1403] T 29693cf06bc24930870dc60457126519 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20251027 21:08:07.248926 10029 raft_consensus.cc:359] T 29693cf06bc24930870dc60457126519 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.249051 10029 raft_consensus.cc:385] T 29693cf06bc24930870dc60457126519 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.249087 10029 raft_consensus.cc:740] T 29693cf06bc24930870dc60457126519 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.250510 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "952583fd7fde4b4baccdb98c01bdf9e1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:07.250847 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29693cf06bc24930870dc60457126519" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:07.251758 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "952583fd7fde4b4baccdb98c01bdf9e1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:07.252152 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29693cf06bc24930870dc60457126519" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:07.249967 10032 tablet_bootstrap.cc:492] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
W20251027 21:08:07.251242 9860 leader_election.cc:343] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.252422 10032 ts_tablet_manager.cc:1403] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
W20251027 21:08:07.252532 9859 leader_election.cc:343] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.252640 9859 leader_election.cc:304] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:07.252983 9859 leader_election.cc:343] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.253122 10032 raft_consensus.cc:359] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.253242 10032 raft_consensus.cc:385] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.253279 10032 raft_consensus.cc:740] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.253443 10032 consensus_queue.cc:260] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.253857 10032 ts_tablet_manager.cc:1434] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:07.254089 10032 tablet_bootstrap.cc:492] T 986bf148b3e9482fa2fe53bbac54750d P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:07.255035 11162 raft_consensus.cc:2749] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:07.260704 9859 leader_election.cc:343] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.260805 9859 leader_election.cc:304] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.261121 11162 raft_consensus.cc:2749] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:07.261454 10029 consensus_queue.cc:260] T 29693cf06bc24930870dc60457126519 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.262712 10029 ts_tablet_manager.cc:1434] T 29693cf06bc24930870dc60457126519 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251027 21:08:07.262955 10029 tablet_bootstrap.cc:492] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:07.263365 9633 raft_consensus.cc:2468] T 29693cf06bc24930870dc60457126519 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 0.
I20251027 21:08:07.264078 9860 leader_election.cc:304] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.264415 11162 raft_consensus.cc:2804] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:07.264483 10029 tablet_bootstrap.cc:654] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.264485 11162 raft_consensus.cc:493] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:07.264541 11162 raft_consensus.cc:3060] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:07.269794 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet 7425b6b0184e4d0b871777b25d904ac4 with delete_type TABLET_DATA_DELETED (replaced by tablet 070167be7d8d4ddf8da561df14448360) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:07.270833 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 588e2f58f07b4a32a78dffd5b92ceb12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:07.271073 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet f4187b2f66f24055b1df170f1062cc88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:07.271785 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet 30ce703650c94c06b93bce976cb5d1fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:07.272281 9743 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 999) took 1614 ms. Trace:
I20251027 21:08:07.272360 9743 rpcz_store.cc:276] 1027 21:08:05.657950 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.520190 (+862240us) service_pool.cc:225] Handling call
1027 21:08:06.665004 (+144814us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.665022 (+ 18us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.071878 (+406856us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.272136 (+200258us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.272265 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.272768 9743 tablet_service.cc:1505] Processing CreateTablet for tablet 57b8f69381904f96a0bfa96c985e8aa9 (DEFAULT_TABLE table=default.Table_3b8f6db604d34f05b8dde4b29bd4cd3f [id=80d1ab0a30c64b5db860a6082823043b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.275712 9738 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 998) took 1624 ms. Trace:
I20251027 21:08:07.275789 9738 rpcz_store.cc:276] 1027 21:08:05.651684 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.516639 (+864955us) service_pool.cc:225] Handling call
1027 21:08:06.662316 (+145677us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.662343 (+ 27us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.130256 (+467913us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.275576 (+145320us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.275696 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.276160 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 58c9cd6828a1497babb9c91bee938041 (DEFAULT_TABLE table=default.Table_b6554b3e96a841fb975ff74ff767069a [id=b6464fe950c5487fb13f6763626d8b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.278998 9753 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1006) took 1595 ms. Trace:
I20251027 21:08:07.279063 9753 rpcz_store.cc:276] 1027 21:08:05.683824 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.646316 (+962492us) service_pool.cc:225] Handling call
1027 21:08:06.646640 (+ 324us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.646658 (+ 18us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.076927 (+430269us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.278856 (+201929us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.278986 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.279361 9753 tablet_service.cc:1505] Processing CreateTablet for tablet b070d77b63ff4557883770df9ff4c55e (DEFAULT_TABLE table=default.Table_653e76a8cae24ff38c2eee3d9322aa6e [id=0d70035dcbd64d83b303f36b7306efc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:07.281965 9755 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1011) took 1484 ms. Trace:
I20251027 21:08:07.282034 9755 rpcz_store.cc:276] 1027 21:08:05.797954 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.687296 (+889342us) service_pool.cc:225] Handling call
1027 21:08:06.792914 (+105618us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.792937 (+ 23us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.077260 (+284323us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.281832 (+204572us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.281951 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.285704 11065 raft_consensus.cc:493] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.285842 11065 raft_consensus.cc:515] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.286554 11065 leader_election.cc:290] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.286674 11065 raft_consensus.cc:493] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.286747 11065 raft_consensus.cc:515] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.287227 11065 leader_election.cc:290] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:07.287933 10032 tablet_bootstrap.cc:654] T 986bf148b3e9482fa2fe53bbac54750d P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.288812 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2710f265ae9485dafec389168328067" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:07.289264 9860 leader_election.cc:343] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.294759 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 32f80614e30d41128f9fb2e0ea165223 with delete_type TABLET_DATA_DELETED (replaced by tablet a7d8d060916941a5b7e944e8b08a2882) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.294886 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet c9efffa9766e46aca8eb286fc2ad48d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.295001 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 67bb12de9f044c73ae9ab26d10498c90 with delete_type TABLET_DATA_DELETED (replaced by tablet ea02ab2fed634dbcba3ca8b04808b694) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.295107 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b30052ff19744a9bf4bfaaa1bc77b36 with delete_type TABLET_DATA_DELETED (replaced by tablet 283b72f626644ac2b9b61186387b3839) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.284547 9746 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1017) took 1485 ms. Trace:
I20251027 21:08:07.295194 9746 rpcz_store.cc:276] 1027 21:08:05.798635 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.800838 (+1002203us) service_pool.cc:225] Handling call
1027 21:08:06.801157 (+ 319us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.801174 (+ 17us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.132910 (+331736us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.284435 (+151525us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.284535 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.287060 9749 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1010) took 1489 ms. Trace:
I20251027 21:08:07.295362 9749 rpcz_store.cc:276] 1027 21:08:05.797849 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.663959 (+866110us) service_pool.cc:225] Handling call
1027 21:08:06.846696 (+182737us) spinlock_profiling.cc:243] Waited 5.39 ms on lock 0x61800000d518. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d840302fd 00007f8d8402ff1e 00007f8d97c2155d 00007f8d97c214fc 00007f8d93e3aa04 00007f8d93e12fcd 00007f8d93d62482 00007f8d8dbff6a6 00007f8d8dbff3d5 00007f8d8afe627e
1027 21:08:06.846717 (+ 21us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.846736 (+ 19us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.138139 (+291403us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.286949 (+148810us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.287047 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12397952}
I20251027 21:08:07.289300 9754 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1016) took 1490 ms. Trace:
I20251027 21:08:07.295538 9754 rpcz_store.cc:276] 1027 21:08:05.798537 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.699049 (+900512us) service_pool.cc:225] Handling call
1027 21:08:06.846814 (+147765us) spinlock_profiling.cc:243] Waited 5.65 ms on lock 0x61800000d518. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d840302fd 00007f8d8402ff1e 00007f8d97c2155d 00007f8d97c214fc 00007f8d93e3aa04 00007f8d93e12fcd 00007f8d93d62482 00007f8d8dbff6a6 00007f8d8dbff3d5 00007f8d8afe627e
1027 21:08:06.846837 (+ 23us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.846867 (+ 30us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.135579 (+288712us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.289192 (+153613us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.289289 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12994944}
I20251027 21:08:07.297798 10029 tablet_bootstrap.cc:492] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:07.297935 10029 ts_tablet_manager.cc:1403] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20251027 21:08:07.291783 9752 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1009) took 1494 ms. Trace:
I20251027 21:08:07.298179 9752 rpcz_store.cc:276] 1027 21:08:05.797723 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.663730 (+866007us) service_pool.cc:225] Handling call
1027 21:08:06.846541 (+182811us) spinlock_profiling.cc:243] Waited 5.11 ms on lock 0x61800000d518. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d840302fd 00007f8d8402ff1e 00007f8d97c2155d 00007f8d97c214fc 00007f8d93e3aa04 00007f8d93e12fcd 00007f8d93d62482 00007f8d8dbff6a6 00007f8d8dbff3d5 00007f8d8afe627e
1027 21:08:06.846565 (+ 24us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.846586 (+ 21us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.140291 (+293705us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.291656 (+151365us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.291771 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11752832}
I20251027 21:08:07.294615 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet a26ded6012174348b7295fc7c0e05e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:51 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.298655 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7425b6b0184e4d0b871777b25d904ac4" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:07.298823 10029 raft_consensus.cc:359] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:07.298970 10029 raft_consensus.cc:385] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.299041 10029 raft_consensus.cc:740] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
W20251027 21:08:07.299124 9860 leader_election.cc:343] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.300181 10032 tablet_bootstrap.cc:492] T 986bf148b3e9482fa2fe53bbac54750d P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:07.300307 10032 ts_tablet_manager.cc:1403] T 986bf148b3e9482fa2fe53bbac54750d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.000s
I20251027 21:08:07.301090 10032 raft_consensus.cc:359] T 986bf148b3e9482fa2fe53bbac54750d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.301258 10032 raft_consensus.cc:385] T 986bf148b3e9482fa2fe53bbac54750d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.301311 10032 raft_consensus.cc:740] T 986bf148b3e9482fa2fe53bbac54750d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.308109 9747 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1018) took 1509 ms. Trace:
I20251027 21:08:07.308177 9747 rpcz_store.cc:276] 1027 21:08:05.798740 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.827903 (+1029163us) service_pool.cc:225] Handling call
1027 21:08:06.875660 (+ 47757us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.875683 (+ 23us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.134399 (+258716us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.307993 (+173594us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.308096 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.311368 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7425b6b0184e4d0b871777b25d904ac4" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:07.311748 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2710f265ae9485dafec389168328067" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:07.312176 9859 leader_election.cc:343] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.312266 9859 leader_election.cc:304] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:07.312602 9859 leader_election.cc:343] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.312680 9859 leader_election.cc:304] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.313078 11065 raft_consensus.cc:2749] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:07.313210 11065 raft_consensus.cc:2749] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:07.322319 10040 ts_tablet_manager.cc:1929] T 4debc3f13f3a42939b6b83f6523aab7c P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:07.322412 10040 log.cc:1199] T 4debc3f13f3a42939b6b83f6523aab7c P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/4debc3f13f3a42939b6b83f6523aab7c
I20251027 21:08:07.326447 10029 consensus_queue.cc:260] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:07.326810 10032 consensus_queue.cc:260] T 986bf148b3e9482fa2fe53bbac54750d P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.327279 10032 ts_tablet_manager.cc:1434] T 986bf148b3e9482fa2fe53bbac54750d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20251027 21:08:07.327533 10032 tablet_bootstrap.cc:492] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:07.331744 11162 raft_consensus.cc:515] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.332993 11162 leader_election.cc:290] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:07.333240 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29693cf06bc24930870dc60457126519" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:07.333390 9632 raft_consensus.cc:3060] T 29693cf06bc24930870dc60457126519 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:07.334403 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53301ba720544c8cbfb075bfc5a7dbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.334688 10029 ts_tablet_manager.cc:1434] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20251027 21:08:07.334903 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b070d77b63ff4557883770df9ff4c55e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.334946 10029 tablet_bootstrap.cc:492] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:07.335644 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a794b2a2586c483c95b3f85c20499de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.336009 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58c9cd6828a1497babb9c91bee938041. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.336521 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a931c36fec45029715b01754d0b161. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.336910 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57b8f69381904f96a0bfa96c985e8aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.337041 10029 tablet_bootstrap.cc:654] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:07.337328 9711 consensus_peers.cc:597] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9efffa9766e46aca8eb286fc2ad48d2. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:07.339166 9296 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 83f5fbf5757a49df93ab949b7657c61f (table default.Table_50b757fa69504e57872e1ecaee13583c [id=2719caa93b0945c59e6ad4d6f1e9e717], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 83f5fbf5757a49df93ab949b7657c61f consensus state has no leader
I20251027 21:08:07.340097 10032 tablet_bootstrap.cc:654] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.345463 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29693cf06bc24930870dc60457126519" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:07.345597 9741 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1019) took 1546 ms. Trace:
I20251027 21:08:07.345679 9741 rpcz_store.cc:276] 1027 21:08:05.798869 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.830831 (+1031962us) service_pool.cc:225] Handling call
1027 21:08:06.831222 (+ 391us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.831244 (+ 22us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.339667 (+508423us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.345430 (+ 5763us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.345567 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251027 21:08:07.346367 9859 leader_election.cc:343] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.346820 11234 raft_consensus.cc:493] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.346985 11234 raft_consensus.cc:515] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.348016 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb95618d9fd54a4f8121b43e717d5616" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:07.348424 9860 leader_election.cc:343] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.350386 11234 leader_election.cc:290] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.351233 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb95618d9fd54a4f8121b43e717d5616" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:07.351704 9859 leader_election.cc:343] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.351819 9859 leader_election.cc:304] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.352262 11234 raft_consensus.cc:2749] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:07.359032 11057 consensus_queue.cc:1048] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251027 21:08:07.359673 9566 consensus_peers.cc:597] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:07.377153 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4b84d8c0ee74dcbaee8aeb49aa30c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.397912 9321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" candidate_term: 4 candidate_status { last_received { term: 3 index: 1191 } } ignore_live_leader: false dest_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" is_pre_election: true
I20251027 21:08:07.401557 11162 raft_consensus.cc:493] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.401739 11162 raft_consensus.cc:515] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.402427 9307 catalog_manager.cc:5649] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:07.402827 9307 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 94f2819dd0394530ae93754bdd1e482f (table default.Table_090fa7c40bdb40d09777fd8458acd6b2 [id=63147f61013c4d6dbc96fd25769b6136]). Expected version 2 got 0
I20251027 21:08:07.402937 11162 leader_election.cc:290] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.403122 9307 catalog_manager.cc:5649] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:07.403443 9307 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 980eb6755aee4bcfb2ea44d84009021f (table default.Table_bb56319b6fe0405991a4d171477621e7 [id=42c8fedcabe0435dbaa11427141228ed]). Expected version 1 got 0
I20251027 21:08:07.404588 10040 ts_tablet_manager.cc:1950] T 4debc3f13f3a42939b6b83f6523aab7c P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:07.404810 10032 tablet_bootstrap.cc:492] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:07.408145 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "842c1a1298234d9ea2719bab29c54ae3" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:07.409835 9612 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1031) took 1649 ms. Trace:
I20251027 21:08:07.409916 9612 rpcz_store.cc:276] 1027 21:08:05.760712 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.627475 (+866763us) service_pool.cc:225] Handling call
1027 21:08:06.627869 (+ 394us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.627894 (+ 25us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.176482 (+548588us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.409722 (+233240us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.409819 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.412710 9614 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1038) took 1595 ms. Trace:
I20251027 21:08:07.412796 9614 rpcz_store.cc:276] 1027 21:08:05.817486 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.682541 (+865055us) service_pool.cc:225] Handling call
1027 21:08:06.682852 (+ 311us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.682882 (+ 30us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.184689 (+501807us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.412543 (+227854us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.412692 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.415753 9611 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1030) took 1656 ms. Trace:
I20251027 21:08:07.415820 9611 rpcz_store.cc:276] 1027 21:08:05.759427 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.602234 (+842807us) service_pool.cc:225] Handling call
1027 21:08:06.749976 (+147742us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.749996 (+ 20us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.180419 (+430423us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.415630 (+235211us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.415740 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.421418 9610 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1040) took 1599 ms. Trace:
I20251027 21:08:07.421504 9610 rpcz_store.cc:276] 1027 21:08:05.821496 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.715552 (+894056us) service_pool.cc:225] Handling call
1027 21:08:06.715918 (+ 366us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.715940 (+ 22us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.330686 (+614746us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.421263 (+ 90577us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.421400 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.422493 9736 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1020) took 1603 ms. Trace:
I20251027 21:08:07.422573 9736 rpcz_store.cc:276] 1027 21:08:05.818811 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.831716 (+1012905us) service_pool.cc:225] Handling call
1027 21:08:06.832005 (+ 289us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.832024 (+ 19us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.128104 (+296080us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.301264 (+173160us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.422476 (+121212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.424031 10829 raft_consensus.cc:493] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.424176 10829 raft_consensus.cc:515] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.425091 9635 raft_consensus.cc:2468] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 1.
W20251027 21:08:07.425352 9711 leader_election.cc:343] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 2a4aa1fe05114c1287ca580c1f411d3f
I20251027 21:08:07.426079 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4aa1fe05114c1287ca580c1f411d3f" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:07.426249 9635 raft_consensus.cc:2468] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:08:07.426549 9710 leader_election.cc:304] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:07.426822 10997 raft_consensus.cc:2804] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:07.426890 10997 raft_consensus.cc:493] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:07.426946 10997 raft_consensus.cc:3060] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:07.427206 10829 leader_election.cc:290] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:07.428117 9634 raft_consensus.cc:2468] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:07.428510 9710 leader_election.cc:304] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:07.428838 10829 raft_consensus.cc:2804] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:07.429289 10043 ts_tablet_manager.cc:1929] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:07.429411 10043 log.cc:1199] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/ccf10ef559f74e2db28023fd520c8de9
I20251027 21:08:07.429728 10829 raft_consensus.cc:697] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:07.429962 10829 consensus_queue.cc:237] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.430711 10043 ts_tablet_manager.cc:1950] T ccf10ef559f74e2db28023fd520c8de9 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:07.435706 9308 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 952583fd7fde4b4baccdb98c01bdf9e1 (table default.Table_df5f4de40f7f46728f38580d23124544 [id=7527117eef9d46a694977cbf2c073974]). Expected version 1 got 0
I20251027 21:08:07.445070 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4b84d8c0ee74dcbaee8aeb49aa30c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:07.171954 10970 rpcz_store.cc:276] 1027 21:08:04.376030 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.378789 (+ 2759us) service_pool.cc:225] Handling call
1027 21:08:06.401688 (+2022899us) raft_consensus.cc:1471] Updating replica for 409 ops
1027 21:08:06.619771 (+218083us) raft_consensus.cc:1534] Early marking committed up to index 0
1027 21:08:06.619780 (+ 9us) raft_consensus.cc:1539] Triggering prepare for 409 ops
1027 21:08:06.620558 (+ 778us) write_op.cc:270] Start()
1027 21:08:06.620630 (+ 72us) write_op.cc:276] Timestamp: P: 1761599276401066 usec, L: 0
1027 21:08:06.707188 (+ 86558us) write_op.cc:270] Start()
1027 21:08:06.707236 (+ 48us) write_op.cc:276] Timestamp: P: 1761599276412043 usec, L: 0
1027 21:08:06.707484 (+ 248us) write_op.cc:270] Start()
1027 21:08:06.707522 (+ 38us) write_op.cc:276] Timestamp: P: 1761599276425129 usec, L: 0
1027 21:08:06.707715 (+ 193us) write_op.cc:270] Start()
1027 21:08:06.707740 (+ 25us) write_op.cc:276] Timestamp: P: 1761599276429237 usec, L: 0
1027 21:08:06.707897 (+ 157us) write_op.cc:270] Start()
1027 21:08:06.707918 (+ 21us) write_op.cc:276] Timestamp: P: 1761599276434390 usec, L: 0
1027 21:08:06.708057 (+ 139us) write_op.cc:270] Start()
1027 21:08:06.708079 (+ 22us) write_op.cc:276] Timestamp: P: 1761599276440979 usec, L: 0
1027 21:08:06.708218 (+ 139us) write_op.cc:270] Start()
1027 21:08:06.708266 (+ 48us) write_op.cc:276] Timestamp: P: 1761599276445506 usec, L: 0
1027 21:08:06.708406 (+ 140us) write_op.cc:270] Start()
1027 21:08:06.708437 (+ 31us) write_op.cc:276] Timestamp: P: 1761599276459630 usec, L: 0
1027 21:08:06.708603 (+ 166us) write_op.cc:270] Start()
1027 21:08:06.708624 (+ 21us) write_op.cc:276] Timestamp: P: 1761599276460277 usec, L: 0
1027 21:08:06.708780 (+ 156us) write_op.cc:270] Start()
1027 21:08:06.708801 (+ 21us) write_op.cc:276] Timestamp: P: 1761599276461358 usec, L: 0
1027 21:08:06.708979 (+ 178us) write_op.cc:270] Start()
1027 21:08:06.708998 (+ 19us) write_op.cc:276] Timestamp: P: 1761599276496821 usec, L: 0
1027 21:08:06.709161 (+ 163us) write_op.cc:270] Start()
1027 21:08:06.709180 (+ 19us) write_op.cc:276] Timestamp: P: 1761599276505521 usec, L: 0
1027 21:08:06.709338 (+ 158us) write_op.cc:270] Start()
1027 21:08:06.709375 (+ 37us) write_op.cc:276] Timestamp: P: 1761599276508348 usec, L: 0
1027 21:08:06.709510 (+ 135us) write_op.cc:270] Start()
1027 21:08:06.709530 (+ 20us) write_op.cc:276] Timestamp: P: 1761599276518607 usec, L: 0
1027 21:08:06.709684 (+ 154us) write_op.cc:270] Start()
1027 21:08:06.709705 (+ 21us) write_op.cc:276] Timestamp: P: 1761599276527774 usec, L: 0
1027 21:08:06.709828 (+ 123us) write_op.cc:270] Start()
1027 21:08:06.709845 (+ 17us) write_op.cc:276] Timestamp: P: 1761599276555418 usec, L: 0
1027 21:08:06.709967 (+ 122us) write_op.cc:270] Start()
1027 21:08:06.709989 (+ 22us) write_op.cc:276] Timestamp: P: 1761599276556005 usec, L: 0
1027 21:08:06.710609 (+ 620us) write_op.cc:270] Start()
1027 21:08:06.710635 (+ 26us) write_op.cc:276] Timestamp: P: 1761599276558930 usec, L: 0
1027 21:08:06.710798 (+ 163us) write_op.cc:270] Start()
1027 21:08:06.710815 (+ 17us) write_op.cc:276] Timestamp: P: 1761599276563069 usec, L: 0
1027 21:08:06.710925 (+ 110us) write_op.cc:270] Start()
1027 21:08:06.710950 (+ 25us) write_op.cc:276] Timestamp: P: 1761599276573870 usec, L: 0
1027 21:08:06.711061 (+ 111us) write_op.cc:270] Start()
1027 21:08:06.711079 (+ 18us) write_op.cc:276] Timestamp: P: 1761599276595801 usec, L: 0
1027 21:08:06.711206 (+ 127us) write_op.cc:270] Start()
1027 21:08:06.711222 (+ 16us) write_op.cc:276] Timestamp: P: 1761599276598510 usec, L: 0
1027 21:08:06.711324 (+ 102us) write_op.cc:270] Start()
1027 21:08:06.711339 (+ 15us) write_op.cc:276] Timestamp: P: 1761599276600603 usec, L: 0
1027 21:08:06.711461 (+ 122us) write_op.cc:270] Start()
1027 21:08:06.711475 (+ 14us) write_op.cc:276] Timestamp: P: 1761599276635119 usec, L: 0
1027 21:08:06.711606 (+ 131us) write_op.cc:270] Start()
1027 21:08:06.711623 (+ 17us) write_op.cc:276] Timestamp: P: 1761599276669756 usec, L: 0
1027 21:08:06.711733 (+ 110us) write_op.cc:270] Start()
1027 21:08:06.711748 (+ 15us) write_op.cc:276] Timestamp: P: 1761599276670351 usec, L: 0
1027 21:08:06.711864 (+ 116us) write_op.cc:270] Start()
1027 21:08:06.711880 (+ 16us) write_op.cc:276] Timestamp: P: 1761599276675673 usec, L: 0
1027 21:08:06.712008 (+ 128us) write_op.cc:270] Start()
1027 21:08:06.712031 (+ 23us) write_op.cc:276] Timestamp: P: 1761599276702599 usec, L: 0
1027 21:08:06.712192 (+ 161us) write_op.cc:270] Start()
1027 21:08:06.712211 (+ 19us) write_op.cc:276] Timestamp: P: 1761599276705849 usec, L: 0
1027 21:08:06.712364 (+ 153us) write_op.cc:270] Start()
1027 21:08:06.712387 (+ 23us) write_op.cc:276] Timestamp: P: 1761599276710916 usec, L: 0
1027 21:08:06.712536 (+ 149us) write_op.cc:270] Start()
1027 21:08:06.712558 (+ 22us) write_op.cc:276] Timestamp: P: 1761599276713361 usec, L: 0
1027 21:08:06.712710 (+ 152us) write_op.cc:270] Start()
1027 21:08:06.712735 (+ 25us) write_op.cc:276] Timestamp: P: 1761599276715191 usec, L: 0
1027 21:08:06.712894 (+ 159us) write_op.cc:270] Start()
1027 21:08:06.712914 (+ 20us) write_op.cc:276] Timestamp: P: 1761599276717697 usec, L: 0
1027 21:08:06.713030 (+ 116us) write_op.cc:270] Start()
1027 21:08:06.713048 (+ 18us) write_op.cc:276] Timestamp: P: 1761599276718322 usec, L: 0
1027 21:08:06.713179 (+ 131us) write_op.cc:270] Start()
1027 21:08:06.713197 (+ 18us) write_op.cc:276] Timestamp: P: 1761599276719877 usec, L: 0
1027 21:08:06.713334 (+ 137us) write_op.cc:270] Start()
1027 21:08:06.713352 (+ 18us) write_op.cc:276] Timestamp: P: 1761599276724103 usec, L: 0
1027 21:08:06.713463 (+ 111us) write_op.cc:270] Start()
1027 21:08:06.713487 (+ 24us) write_op.cc:276] Timestamp: P: 1761599276739772 usec, L: 0
1027 21:08:06.713646 (+ 159us) write_op.cc:270] Start()
1027 21:08:06.713662 (+ 16us) write_op.cc:276] Timestamp: P: 1761599276744378 usec, L: 0
1027 21:08:06.713755 (+ 93us) write_op.cc:270] Start()
1027 21:08:06.713769 (+ 14us) write_op.cc:276] Timestamp: P: 1761599277371354 usec, L: 0
1027 21:08:06.713881 (+ 112us) write_op.cc:270] Start()
1027 21:08:06.713896 (+ 15us) write_op.cc:276] Timestamp: P: 1761599277380132 usec, L: 0
1027 21:08:06.713994 (+ 98us) write_op.cc:270] Start()
1027 21:08:06.714009 (+ 15us) write_op.cc:276] Timestamp: P: 1761599277381630 usec, L: 0
1027 21:08:06.724805 (+ 10796us) write_op.cc:270] Start()
1027 21:08:06.724829 (+ 24us) write_op.cc:276] Timestamp: P: 1761599277382214 usec, L: 0
1027 21:08:06.724990 (+ 161us) write_op.cc:270] Start()
1027 21:08:06.725014 (+ 24us) write_op.cc:276] Timestamp: P: 1761599277382772 usec, L: 0
1027 21:08:06.725150 (+ 136us) write_op.cc:270] Start()
1027 21:08:06.725170 (+ 20us) write_op.cc:276] Timestamp: P: 1761599277383375 usec, L: 0
1027 21:08:06.725300 (+ 130us) write_op.cc:270] Start()
1027 21:08:06.725320 (+ 20us) write_op.cc:276] Timestamp: P: 1761599277383900 usec, L: 0
1027 21:08:06.725476 (+ 156us) write_op.cc:270] Start()
1027 21:08:06.725495 (+ 19us) write_op.cc:276] Timestamp: P: 1761599277384470 usec, L: 0
1027 21:08:06.725626 (+ 131us) write_op.cc:270] Start()
1027 21:08:06.725645 (+ 19us) write_op.cc:276] Timestamp: P: 1761599277394243 usec, L: 0
1027 21:08:06.725784 (+ 139us) write_op.cc:270] Start()
1027 21:08:06.725807 (+ 23us) write_op.cc:276] Timestamp: P: 1761599277457715 usec, L: 0
1027 21:08:06.725975 (+ 168us) write_op.cc:270] Start()
1027 21:08:06.725997 (+ 22us) write_op.cc:276] Timestamp: P: 1761599277468528 usec, L: 0
1027 21:08:06.726184 (+ 187us) write_op.cc:270] Start()
1027 21:08:06.726207 (+ 23us) write_op.cc:276] Timestamp: P: 1761599277469051 usec, L: 0
1027 21:08:06.726433 (+ 226us) write_op.cc:270] Start()
1027 21:08:06.726454 (+ 21us) write_op.cc:276] Timestamp: P: 1761599277469526 usec, L: 0
1027 21:08:06.726621 (+ 167us) write_op.cc:270] Start()
1027 21:08:06.726643 (+ 22us) write_op.cc:276] Timestamp: P: 1761599277511085 usec, L: 0
1027 21:08:06.726772 (+ 129us) write_op.cc:270] Start()
1027 21:08:06.726790 (+ 18us) write_op.cc:276] Timestamp: P: 1761599277531271 usec, L: 0
1027 21:08:06.726916 (+ 126us) write_op.cc:270] Start()
1027 21:08:06.726934 (+ 18us) write_op.cc:276] Timestamp: P: 1761599277531853 usec, L: 0
1027 21:08:06.727084 (+ 150us) write_op.cc:270] Start()
1027 21:08:06.727105 (+ 21us) write_op.cc:276] Timestamp: P: 1761599277532411 usec, L: 0
1027 21:08:06.727249 (+ 144us) write_op.cc:270] Start()
1027 21:08:06.727269 (+ 20us) write_op.cc:276] Timestamp: P: 1761599277532919 usec, L: 0
1027 21:08:06.727425 (+ 156us) write_op.cc:270] Start()
1027 21:08:06.727444 (+ 19us) write_op.cc:276] Timestamp: P: 1761599277564837 usec, L: 0
1027 21:08:06.727578 (+ 134us) write_op.cc:270] Start()
1027 21:08:06.727608 (+ 30us) write_op.cc:276] Timestamp: P: 1761599277855754 usec, L: 0
1027 21:08:06.727744 (+ 136us) write_op.cc:270] Start()
1027 21:08:06.727762 (+ 18us) write_op.cc:276] Timestamp: P: 1761599277882388 usec, L: 0
1027 21:08:06.727883 (+ 121us) write_op.cc:270] Start()
1027 21:08:06.727923 (+ 40us) write_op.cc:276] Timestamp: P: 1761599277883022 usec, L: 0
1027 21:08:06.728041 (+ 118us) write_op.cc:270] Start()
1027 21:08:06.728057 (+ 16us) write_op.cc:276] Timestamp: P: 1761599277883581 usec, L: 0
1027 21:08:06.728193 (+ 136us) write_op.cc:270] Start()
1027 21:08:06.728210 (+ 17us) write_op.cc:276] Timestamp: P: 1761599277894371 usec, L: 0
1027 21:08:06.728340 (+ 130us) write_op.cc:270] Start()
1027 21:08:06.728358 (+ 18us) write_op.cc:276] Timestamp: P: 1761599277894993 usec, L: 0
1027 21:08:06.728482 (+ 124us) write_op.cc:270] Start()
1027 21:08:06.728500 (+ 18us) write_op.cc:276] Timestamp: P: 1761599277895624 usec, L: 0
1027 21:08:06.728652 (+ 152us) write_op.cc:270] Start()
1027 21:08:06.734517 (+ 5865us) write_op.cc:276] Timestamp: P: 1761599277896290 usec, L: 0
1027 21:08:06.734697 (+ 180us) write_op.cc:270] Start()
1027 21:08:06.734719 (+ 22us) write_op.cc:276] Timestamp: P: 1761599277896831 usec, L: 0
1027 21:08:06.734866 (+ 147us) write_op.cc:270] Start()
1027 21:08:06.734885 (+ 19us) write_op.cc:276] Timestamp: P: 1761599277897749 usec, L: 0
1027 21:08:06.735027 (+ 142us) write_op.cc:270] Start()
1027 21:08:06.735046 (+ 19us) write_op.cc:276] Timestamp: P: 1761599278704862 usec, L: 0
1027 21:08:06.735163 (+ 117us) write_op.cc:270] Start()
1027 21:08:06.735181 (+ 18us) write_op.cc:276] Timestamp: P: 1761599278761790 usec, L: 0
1027 21:08:06.735305 (+ 124us) write_op.cc:270] Start()
1027 21:08:06.735321 (+ 16us) write_op.cc:276] Timestamp: P: 1761599278762459 usec, L: 0
1027 21:08:06.735441 (+ 120us) write_op.cc:270] Start()
1027 21:08:06.735457 (+ 16us) write_op.cc:276] Timestamp: P: 1761599278793896 usec, L: 0
1027 21:08:06.736124 (+ 667us) write_op.cc:270] Start()
1027 21:08:06.736149 (+ 25us) write_op.cc:276] Timestamp: P: 1761599278804816 usec, L: 0
1027 21:08:06.736312 (+ 163us) write_op.cc:270] Start()
1027 21:08:06.736330 (+ 18us) write_op.cc:276] Timestamp: P: 1761599278809957 usec, L: 0
1027 21:08:06.736502 (+ 172us) write_op.cc:270] Start()
1027 21:08:06.736520 (+ 18us) write_op.cc:276] Timestamp: P: 1761599278814840 usec, L: 0
1027 21:08:06.743928 (+ 7408us) write_op.cc:270] Start()
1027 21:08:06.743952 (+ 24us) write_op.cc:276] Timestamp: P: 1761599278821888 usec, L: 0
1027 21:08:06.744108 (+ 156us) write_op.cc:270] Start()
1027 21:08:06.744133 (+ 25us) write_op.cc:276] Timestamp: P: 1761599278837036 usec, L: 0
1027 21:08:06.744273 (+ 140us) write_op.cc:270] Start()
1027 21:08:06.744299 (+ 26us) write_op.cc:276] Timestamp: P: 1761599278841981 usec, L: 0
1027 21:08:06.744429 (+ 130us) write_op.cc:270] Start()
1027 21:08:06.744447 (+ 18us) write_op.cc:276] Timestamp: P: 1761599278848622 usec, L: 0
1027 21:08:06.744629 (+ 182us) write_op.cc:270] Start()
1027 21:08:06.744674 (+ 45us) write_op.cc:276] Timestamp: P: 1761599278856082 usec, L: 0
1027 21:08:06.744837 (+ 163us) write_op.cc:270] Start()
1027 21:08:06.744856 (+ 19us) write_op.cc:276] Timestamp: P: 1761599278859452 usec, L: 0
1027 21:08:06.745006 (+ 150us) write_op.cc:270] Start()
1027 21:08:06.745031 (+ 25us) write_op.cc:276] Timestamp: P: 1761599278885773 usec, L: 0
1027 21:08:06.745162 (+ 131us) write_op.cc:270] Start()
1027 21:08:06.745180 (+ 18us) write_op.cc:276] Timestamp: P: 1761599278895175 usec, L: 0
1027 21:08:06.745302 (+ 122us) write_op.cc:270] Start()
1027 21:08:06.745322 (+ 20us) write_op.cc:276] Timestamp: P: 1761599278902228 usec, L: 0
1027 21:08:06.745482 (+ 160us) write_op.cc:270] Start()
1027 21:08:06.745503 (+ 21us) write_op.cc:276] Timestamp: P: 1761599278914203 usec, L: 0
1027 21:08:06.745664 (+ 161us) write_op.cc:270] Start()
1027 21:08:06.745684 (+ 20us) write_op.cc:276] Timestamp: P: 1761599278916382 usec, L: 0
1027 21:08:06.745819 (+ 135us) write_op.cc:270] Start()
1027 21:08:06.745836 (+ 17us) write_op.cc:276] Timestamp: P: 1761599278923763 usec, L: 0
1027 21:08:06.745962 (+ 126us) write_op.cc:270] Start()
1027 21:08:06.745981 (+ 19us) write_op.cc:276] Timestamp: P: 1761599278951570 usec, L: 0
1027 21:08:06.746110 (+ 129us) write_op.cc:270] Start()
1027 21:08:06.746129 (+ 19us) write_op.cc:276] Timestamp: P: 1761599278953694 usec, L: 0
1027 21:08:06.746317 (+ 188us) write_op.cc:270] Start()
1027 21:08:06.746341 (+ 24us) write_op.cc:276] Timestamp: P: 1761599278954357 usec, L: 0
1027 21:08:06.746494 (+ 153us) write_op.cc:270] Start()
1027 21:08:06.746513 (+ 19us) write_op.cc:276] Timestamp: P: 1761599278985977 usec, L: 0
1027 21:08:06.746680 (+ 167us) write_op.cc:270] Start()
1027 21:08:06.746705 (+ 25us) write_op.cc:276] Timestamp: P: 1761599278993565 usec, L: 0
1027 21:08:06.761499 (+ 14794us) write_op.cc:270] Start()
1027 21:08:06.761534 (+ 35us) write_op.cc:276] Timestamp: P: 1761599279002625 usec, L: 0
1027 21:08:06.761716 (+ 182us) write_op.cc:270] Start()
1027 21:08:06.761743 (+ 27us) write_op.cc:276] Timestamp: P: 1761599279015746 usec, L: 0
1027 21:08:06.761869 (+ 126us) write_op.cc:270] Start()
1027 21:08:06.761885 (+ 16us) write_op.cc:276] Timestamp: P: 1761599279037921 usec, L: 0
1027 21:08:06.761984 (+ 99us) write_op.cc:270] Start()
1027 21:08:06.761998 (+ 14us) write_op.cc:276] Timestamp: P: 1761599279046283 usec, L: 0
1027 21:08:06.762126 (+ 128us) write_op.cc:270] Start()
1027 21:08:06.762163 (+ 37us) write_op.cc:276] Timestamp: P: 1761599279055424 usec, L: 0
1027 21:08:06.762294 (+ 131us) write_op.cc:270] Start()
1027 21:08:06.762310 (+ 16us) write_op.cc:276] Timestamp: P: 1761599279057705 usec, L: 0
1027 21:08:06.762405 (+ 95us) write_op.cc:270] Start()
1027 21:08:06.762420 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279058349 usec, L: 0
1027 21:08:06.762513 (+ 93us) write_op.cc:270] Start()
1027 21:08:06.762528 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279107967 usec, L: 0
1027 21:08:06.762652 (+ 124us) write_op.cc:270] Start()
1027 21:08:06.762667 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279116046 usec, L: 0
1027 21:08:06.762766 (+ 99us) write_op.cc:270] Start()
1027 21:08:06.762781 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279126787 usec, L: 0
1027 21:08:06.762878 (+ 97us) write_op.cc:270] Start()
1027 21:08:06.762893 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279135616 usec, L: 0
1027 21:08:06.762986 (+ 93us) write_op.cc:270] Start()
1027 21:08:06.763001 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279143558 usec, L: 0
1027 21:08:06.763105 (+ 104us) write_op.cc:270] Start()
1027 21:08:06.763119 (+ 14us) write_op.cc:276] Timestamp: P: 1761599279150476 usec, L: 0
1027 21:08:06.764094 (+ 975us) write_op.cc:270] Start()
1027 21:08:06.764121 (+ 27us) write_op.cc:276] Timestamp: P: 1761599279153214 usec, L: 0
1027 21:08:06.764244 (+ 123us) write_op.cc:270] Start()
1027 21:08:06.764260 (+ 16us) write_op.cc:276] Timestamp: P: 1761599279154493 usec, L: 0
1027 21:08:06.764361 (+ 101us) write_op.cc:270] Start()
1027 21:08:06.764376 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279203100 usec, L: 0
1027 21:08:06.764486 (+ 110us) write_op.cc:270] Start()
1027 21:08:06.764500 (+ 14us) write_op.cc:276] Timestamp: P: 1761599279216417 usec, L: 0
1027 21:08:06.764602 (+ 102us) write_op.cc:270] Start()
1027 21:08:06.764617 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279224742 usec, L: 0
1027 21:08:06.764732 (+ 115us) write_op.cc:270] Start()
1027 21:08:06.764747 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279261383 usec, L: 0
1027 21:08:06.764839 (+ 92us) write_op.cc:270] Start()
1027 21:08:06.764855 (+ 16us) write_op.cc:276] Timestamp: P: 1761599279270378 usec, L: 0
1027 21:08:06.764966 (+ 111us) write_op.cc:270] Start()
1027 21:08:06.764982 (+ 16us) write_op.cc:276] Timestamp: P: 1761599279279680 usec, L: 0
1027 21:08:06.765079 (+ 97us) write_op.cc:270] Start()
1027 21:08:06.765093 (+ 14us) write_op.cc:276] Timestamp: P: 1761599279296231 usec, L: 0
1027 21:08:06.765211 (+ 118us) write_op.cc:270] Start()
1027 21:08:06.765226 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279317494 usec, L: 0
1027 21:08:06.765326 (+ 100us) write_op.cc:270] Start()
1027 21:08:06.765339 (+ 13us) write_op.cc:276] Timestamp: P: 1761599279333495 usec, L: 0
1027 21:08:06.765436 (+ 97us) write_op.cc:270] Start()
1027 21:08:06.765451 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279340609 usec, L: 0
1027 21:08:06.765539 (+ 88us) write_op.cc:270] Start()
1027 21:08:06.765563 (+ 24us) write_op.cc:276] Timestamp: P: 1761599279344459 usec, L: 0
1027 21:08:06.765683 (+ 120us) write_op.cc:270] Start()
1027 21:08:06.765698 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279358043 usec, L: 0
1027 21:08:06.765790 (+ 92us) write_op.cc:270] Start()
1027 21:08:06.765805 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279384728 usec, L: 0
1027 21:08:06.765913 (+ 108us) write_op.cc:270] Start()
1027 21:08:06.765931 (+ 18us) write_op.cc:276] Timestamp: P: 1761599279401239 usec, L: 0
1027 21:08:06.766066 (+ 135us) write_op.cc:270] Start()
1027 21:08:06.766084 (+ 18us) write_op.cc:276] Timestamp: P: 1761599279407369 usec, L: 0
1027 21:08:06.766255 (+ 171us) write_op.cc:270] Start()
1027 21:08:06.766277 (+ 22us) write_op.cc:276] Timestamp: P: 1761599279417166 usec, L: 0
1027 21:08:06.766402 (+ 125us) write_op.cc:270] Start()
1027 21:08:06.766419 (+ 17us) write_op.cc:276] Timestamp: P: 1761599279428209 usec, L: 0
1027 21:08:06.766547 (+ 128us) write_op.cc:270] Start()
1027 21:08:06.766564 (+ 17us) write_op.cc:276] Timestamp: P: 1761599279441486 usec, L: 0
1027 21:08:06.766699 (+ 135us) write_op.cc:270] Start()
1027 21:08:06.766715 (+ 16us) write_op.cc:276] Timestamp: P: 1761599279462471 usec, L: 0
1027 21:08:06.766857 (+ 142us) write_op.cc:270] Start()
1027 21:08:06.766875 (+ 18us) write_op.cc:276] Timestamp: P: 1761599279464946 usec, L: 0
1027 21:08:06.766979 (+ 104us) write_op.cc:270] Start()
1027 21:08:06.766994 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279488928 usec, L: 0
1027 21:08:06.767112 (+ 118us) write_op.cc:270] Start()
1027 21:08:06.767128 (+ 16us) write_op.cc:276] Timestamp: P: 1761599279505444 usec, L: 0
1027 21:08:06.767256 (+ 128us) write_op.cc:270] Start()
1027 21:08:06.767272 (+ 16us) write_op.cc:276] Timestamp: P: 1761599279506070 usec, L: 0
1027 21:08:06.767408 (+ 136us) write_op.cc:270] Start()
1027 21:08:06.767425 (+ 17us) write_op.cc:276] Timestamp: P: 1761599279506696 usec, L: 0
1027 21:08:06.767571 (+ 146us) write_op.cc:270] Start()
1027 21:08:06.767630 (+ 59us) write_op.cc:276] Timestamp: P: 1761599279507208 usec, L: 0
1027 21:08:06.767745 (+ 115us) write_op.cc:270] Start()
1027 21:08:06.767762 (+ 17us) write_op.cc:276] Timestamp: P: 1761599279575327 usec, L: 0
1027 21:08:06.767865 (+ 103us) write_op.cc:270] Start()
1027 21:08:06.767880 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279586680 usec, L: 0
1027 21:08:06.767979 (+ 99us) write_op.cc:270] Start()
1027 21:08:06.768010 (+ 31us) write_op.cc:276] Timestamp: P: 1761599279587855 usec, L: 0
1027 21:08:06.768126 (+ 116us) write_op.cc:270] Start()
1027 21:08:06.768141 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279588390 usec, L: 0
1027 21:08:06.768241 (+ 100us) write_op.cc:270] Start()
1027 21:08:06.768262 (+ 21us) write_op.cc:276] Timestamp: P: 1761599279642921 usec, L: 0
1027 21:08:06.782085 (+ 13823us) write_op.cc:270] Start()
1027 21:08:06.782117 (+ 32us) write_op.cc:276] Timestamp: P: 1761599279643873 usec, L: 0
1027 21:08:06.782269 (+ 152us) write_op.cc:270] Start()
1027 21:08:06.782287 (+ 18us) write_op.cc:276] Timestamp: P: 1761599279644447 usec, L: 0
1027 21:08:06.782424 (+ 137us) write_op.cc:270] Start()
1027 21:08:06.782447 (+ 23us) write_op.cc:276] Timestamp: P: 1761599279645003 usec, L: 0
1027 21:08:06.782608 (+ 161us) write_op.cc:270] Start()
1027 21:08:06.782628 (+ 20us) write_op.cc:276] Timestamp: P: 1761599279645583 usec, L: 0
1027 21:08:06.782767 (+ 139us) write_op.cc:270] Start()
1027 21:08:06.782790 (+ 23us) write_op.cc:276] Timestamp: P: 1761599279646201 usec, L: 0
1027 21:08:06.782944 (+ 154us) write_op.cc:270] Start()
1027 21:08:06.782960 (+ 16us) write_op.cc:276] Timestamp: P: 1761599279647099 usec, L: 0
1027 21:08:06.783064 (+ 104us) write_op.cc:270] Start()
1027 21:08:06.783078 (+ 14us) write_op.cc:276] Timestamp: P: 1761599279667239 usec, L: 0
1027 21:08:06.783180 (+ 102us) write_op.cc:270] Start()
1027 21:08:06.783195 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279669087 usec, L: 0
1027 21:08:06.783295 (+ 100us) write_op.cc:270] Start()
1027 21:08:06.783309 (+ 14us) write_op.cc:276] Timestamp: P: 1761599279672936 usec, L: 0
1027 21:08:06.783407 (+ 98us) write_op.cc:270] Start()
1027 21:08:06.783421 (+ 14us) write_op.cc:276] Timestamp: P: 1761599279697749 usec, L: 0
1027 21:08:06.783523 (+ 102us) write_op.cc:270] Start()
1027 21:08:06.783537 (+ 14us) write_op.cc:276] Timestamp: P: 1761599279701645 usec, L: 0
1027 21:08:06.783650 (+ 113us) write_op.cc:270] Start()
1027 21:08:06.783664 (+ 14us) write_op.cc:276] Timestamp: P: 1761599279718212 usec, L: 0
1027 21:08:06.783758 (+ 94us) write_op.cc:270] Start()
1027 21:08:06.783772 (+ 14us) write_op.cc:276] Timestamp: P: 1761599279816068 usec, L: 0
1027 21:08:06.783875 (+ 103us) write_op.cc:270] Start()
1027 21:08:06.783892 (+ 17us) write_op.cc:276] Timestamp: P: 1761599279826028 usec, L: 0
1027 21:08:06.784042 (+ 150us) write_op.cc:270] Start()
1027 21:08:06.784062 (+ 20us) write_op.cc:276] Timestamp: P: 1761599279826762 usec, L: 0
1027 21:08:06.784201 (+ 139us) write_op.cc:270] Start()
1027 21:08:06.784216 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279827343 usec, L: 0
1027 21:08:06.784325 (+ 109us) write_op.cc:270] Start()
1027 21:08:06.784339 (+ 14us) write_op.cc:276] Timestamp: P: 1761599279828591 usec, L: 0
1027 21:08:06.784441 (+ 102us) write_op.cc:270] Start()
1027 21:08:06.784456 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279848811 usec, L: 0
1027 21:08:06.784545 (+ 89us) write_op.cc:270] Start()
1027 21:08:06.784560 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279851259 usec, L: 0
1027 21:08:06.784675 (+ 115us) write_op.cc:270] Start()
1027 21:08:06.784690 (+ 15us) write_op.cc:276] Timestamp: P: 1761599279851993 usec, L: 0
1027 21:08:06.784791 (+ 101us) write_op.cc:270] Start()
1027 21:08:06.784805 (+ 14us) write_op.cc:276] Timestamp: P: 1761599279969718 usec, L: 0
1027 21:08:06.784920 (+ 115us) write_op.cc:270] Start()
1027 21:08:06.784941 (+ 21us) write_op.cc:276] Timestamp: P: 1761599279971663 usec, L: 0
1027 21:08:06.785073 (+ 132us) write_op.cc:270] Start()
1027 21:08:06.785092 (+ 19us) write_op.cc:276] Timestamp: P: 1761599279976776 usec, L: 0
1027 21:08:06.785220 (+ 128us) write_op.cc:270] Start()
1027 21:08:06.785245 (+ 25us) write_op.cc:276] Timestamp: P: 1761599279978112 usec, L: 0
1027 21:08:06.785404 (+ 159us) write_op.cc:270] Start()
1027 21:08:06.785442 (+ 38us) write_op.cc:276] Timestamp: P: 1761599279979964 usec, L: 0
1027 21:08:06.785608 (+ 166us) write_op.cc:270] Start()
1027 21:08:06.785628 (+ 20us) write_op.cc:276] Timestamp: P: 1761599279981713 usec, L: 0
1027 21:08:06.785735 (+ 107us) write_op.cc:270] Start()
1027 21:08:06.785751 (+ 16us) write_op.cc:276] Timestamp: P: 1761599280034018 usec, L: 0
1027 21:08:06.785852 (+ 101us) write_op.cc:270] Start()
1027 21:08:06.785869 (+ 17us) write_op.cc:276] Timestamp: P: 1761599280054787 usec, L: 0
1027 21:08:06.785977 (+ 108us) write_op.cc:270] Start()
1027 21:08:06.785992 (+ 15us) write_op.cc:276] Timestamp: P: 1761599280055476 usec, L: 0
1027 21:08:06.786115 (+ 123us) write_op.cc:270] Start()
1027 21:08:06.786130 (+ 15us) write_op.cc:276] Timestamp: P: 1761599280061563 usec, L: 0
1027 21:08:06.786298 (+ 168us) write_op.cc:270] Start()
1027 21:08:06.786377 (+ 79us) write_op.cc:276] Timestamp: P: 1761599280119454 usec, L: 0
1027 21:08:06.786528 (+ 151us) write_op.cc:270] Start()
1027 21:08:06.786545 (+ 17us) write_op.cc:276] Timestamp: P: 1761599280122114 usec, L: 0
1027 21:08:06.786693 (+ 148us) write_op.cc:270] Start()
1027 21:08:06.786713 (+ 20us) write_op.cc:276] Timestamp: P: 1761599280141687 usec, L: 0
1027 21:08:06.786837 (+ 124us) write_op.cc:270] Start()
1027 21:08:06.786861 (+ 24us) write_op.cc:276] Timestamp: P: 1761599280142375 usec, L: 0
1027 21:08:06.787008 (+ 147us) write_op.cc:270] Start()
1027 21:08:06.787025 (+ 17us) write_op.cc:276] Timestamp: P: 1761599280142913 usec, L: 0
1027 21:08:06.787138 (+ 113us) write_op.cc:270] Start()
1027 21:08:06.787158 (+ 20us) write_op.cc:276] Timestamp: P: 1761599280143516 usec, L: 0
1027 21:08:06.787294 (+ 136us) write_op.cc:270] Start()
1027 21:08:06.787312 (+ 18us) write_op.cc:276] Timestamp: P: 1761599280163642 usec, L: 0
1027 21:08:06.787428 (+ 116us) write_op.cc:270] Start()
1027 21:08:06.787445 (+ 17us) write_op.cc:276] Timestamp: P: 1761599280164532 usec, L: 0
1027 21:08:06.787554 (+ 109us) write_op.cc:270] Start()
1027 21:08:06.787568 (+ 14us) write_op.cc:276] Timestamp: P: 1761599280165812 usec, L: 0
1027 21:08:06.787673 (+ 105us) write_op.cc:270] Start()
1027 21:08:06.787688 (+ 15us) write_op.cc:276] Timestamp: P: 1761599280201143 usec, L: 0
1027 21:08:06.787829 (+ 141us) write_op.cc:270] Start()
1027 21:08:06.787846 (+ 17us) write_op.cc:276] Timestamp: P: 1761599280262325 usec, L: 0
1027 21:08:06.787964 (+ 118us) write_op.cc:270] Start()
1027 21:08:06.787986 (+ 22us) write_op.cc:276] Timestamp: P: 1761599280263044 usec, L: 0
1027 21:08:06.788139 (+ 153us) write_op.cc:270] Start()
1027 21:08:06.788157 (+ 18us) write_op.cc:276] Timestamp: P: 1761599280281719 usec, L: 0
1027 21:08:06.788288 (+ 131us) write_op.cc:270] Start()
1027 21:08:06.788307 (+ 19us) write_op.cc:276] Timestamp: P: 1761599280317988 usec, L: 0
1027 21:08:06.788444 (+ 137us) write_op.cc:270] Start()
1027 21:08:06.788462 (+ 18us) write_op.cc:276] Timestamp: P: 1761599280327397 usec, L: 0
1027 21:08:06.788600 (+ 138us) write_op.cc:270] Start()
1027 21:08:06.788620 (+ 20us) write_op.cc:276] Timestamp: P: 1761599280328059 usec, L: 0
1027 21:08:06.788767 (+ 147us) write_op.cc:270] Start()
1027 21:08:06.788793 (+ 26us) write_op.cc:276] Timestamp: P: 1761599280328585 usec, L: 0
1027 21:08:06.788903 (+ 110us) write_op.cc:270] Start()
1027 21:08:06.788920 (+ 17us) write_op.cc:276] Timestamp: P: 1761599280329097 usec, L: 0
1027 21:08:06.806178 (+ 17258us) write_op.cc:270] Start()
1027 21:08:06.806228 (+ 50us) write_op.cc:276] Timestamp: P: 1761599280346517 usec, L: 0
1027 21:08:06.806458 (+ 230us) write_op.cc:270] Start()
1027 21:08:06.806481 (+ 23us) write_op.cc:276] Timestamp: P: 1761599280347280 usec, L: 0
1027 21:08:06.806634 (+ 153us) write_op.cc:270] Start()
1027 21:08:06.806652 (+ 18us) write_op.cc:276] Timestamp: P: 1761599280348071 usec, L: 0
1027 21:08:06.806792 (+ 140us) write_op.cc:270] Start()
1027 21:08:06.806810 (+ 18us) write_op.cc:276] Timestamp: P: 1761599280348931 usec, L: 0
1027 21:08:06.806943 (+ 133us) write_op.cc:270] Start()
1027 21:08:06.806960 (+ 17us) write_op.cc:276] Timestamp: P: 1761599280349566 usec, L: 0
1027 21:08:06.810856 (+ 3896us) write_op.cc:270] Start()
1027 21:08:06.810889 (+ 33us) write_op.cc:276] Timestamp: P: 1761599280350447 usec, L: 0
1027 21:08:06.811070 (+ 181us) write_op.cc:270] Start()
1027 21:08:06.811091 (+ 21us) write_op.cc:276] Timestamp: P: 1761599280367966 usec, L: 0
1027 21:08:06.811216 (+ 125us) write_op.cc:270] Start()
1027 21:08:06.811235 (+ 19us) write_op.cc:276] Timestamp: P: 1761599280369007 usec, L: 0
1027 21:08:06.811376 (+ 141us) write_op.cc:270] Start()
1027 21:08:06.811397 (+ 21us) write_op.cc:276] Timestamp: P: 1761599280369775 usec, L: 0
1027 21:08:06.811536 (+ 139us) write_op.cc:270] Start()
1027 21:08:06.811554 (+ 18us) write_op.cc:276] Timestamp: P: 1761599280401306 usec, L: 0
1027 21:08:06.811689 (+ 135us) write_op.cc:270] Start()
1027 21:08:06.811710 (+ 21us) write_op.cc:276] Timestamp: P: 1761599280402058 usec, L: 0
1027 21:08:06.811848 (+ 138us) write_op.cc:270] Start()
1027 21:08:06.811885 (+ 37us) write_op.cc:276] Timestamp: P: 1761599280402762 usec, L: 0
1027 21:08:06.812016 (+ 131us) write_op.cc:270] Start()
1027 21:08:06.812040 (+ 24us) write_op.cc:276] Timestamp: P: 1761599
I20251027 21:08:07.454252 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "842c1a1298234d9ea2719bab29c54ae3" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:07.454774 10031 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.032s sys 0.083s
W20251027 21:08:07.454849 10031 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.574s user 0.032s sys 0.083s
I20251027 21:08:07.455796 10029 tablet_bootstrap.cc:492] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:07.455920 10029 ts_tablet_manager.cc:1403] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.121s user 0.003s sys 0.000s
I20251027 21:08:07.404947 10032 ts_tablet_manager.cc:1403] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.077s user 0.003s sys 0.000s
I20251027 21:08:07.405335 11177 raft_consensus.cc:493] T 986bf148b3e9482fa2fe53bbac54750d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.464449 11177 raft_consensus.cc:515] T 986bf148b3e9482fa2fe53bbac54750d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.406021 11162 raft_consensus.cc:493] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.465262 11162 raft_consensus.cc:515] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.465859 11162 leader_election.cc:290] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:07.408516 9860 leader_election.cc:343] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:07.466148 9860 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.65:44531, user_credentials={real_user=slave}} blocked reactor thread for 57607us
I20251027 21:08:07.466614 9860 leader_election.cc:304] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:07.467239 9859 leader_election.cc:343] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.467453 9859 leader_election.cc:304] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.468227 10031 tablet_bootstrap.cc:492] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:07.468366 10031 ts_tablet_manager.cc:1403] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.590s user 0.032s sys 0.084s
I20251027 21:08:07.468703 11234 raft_consensus.cc:2804] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:07.468839 11234 raft_consensus.cc:697] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Becoming Leader. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Running, Role: LEADER
I20251027 21:08:07.469126 11234 consensus_queue.cc:237] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:07.469259 10031 raft_consensus.cc:359] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.469444 10031 raft_consensus.cc:385] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.469494 10031 raft_consensus.cc:740] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.469911 10031 consensus_queue.cc:260] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.470407 11065 raft_consensus.cc:2749] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:07.470455 10031 ts_tablet_manager.cc:1434] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:07.470710 10031 tablet_bootstrap.cc:492] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:07.471709 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67cb71ce43244d8ab9e71ead6bb315c9" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:07.471851 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet c3b9a461567a48fcb4af514ecdf618b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.471554 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "986bf148b3e9482fa2fe53bbac54750d" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:07.465061 11177 leader_election.cc:290] T 986bf148b3e9482fa2fe53bbac54750d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:07.472508 9859 leader_election.cc:343] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:07.472733 9859 leader_election.cc:343] T 986bf148b3e9482fa2fe53bbac54750d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.475363 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a4aa1fe05114c1287ca580c1f411d3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.434151 10041 ts_tablet_manager.cc:1929] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:07.477819 10041 log.cc:1199] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/aabaa34787cf4d179822269b7f323314
I20251027 21:08:07.478325 10041 ts_tablet_manager.cc:1950] T aabaa34787cf4d179822269b7f323314 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:07.478336 9393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" candidate_term: 4 candidate_status { last_received { term: 3 index: 1191 } } ignore_live_leader: false dest_uuid: "18b15927ce5744ad8ba81dda52725c11" is_pre_election: true
I20251027 21:08:07.479123 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 919) took 3752 ms. Trace:
I20251027 21:08:07.480466 10887 leader_election.cc:304] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e86ca5e83dd4df5b72f045e7b787e81; no voters: 18b15927ce5744ad8ba81dda52725c11, a4b8b31d47da48aaa386a8e64a5749c2
I20251027 21:08:07.481527 11160 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251027 21:08:07.483368 11244 tablet.cc:1722] T 6600774e4c7b48479a2fdb02614ba505 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:08:07.468274 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "986bf148b3e9482fa2fe53bbac54750d" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:07.470734 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67cb71ce43244d8ab9e71ead6bb315c9" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:07.482386 11097 raft_consensus.cc:515] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.486259 11097 leader_election.cc:290] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.456722 10029 raft_consensus.cc:359] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.487454 10029 raft_consensus.cc:385] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.487572 10029 raft_consensus.cc:740] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.487896 10029 consensus_queue.cc:260] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.488548 10029 ts_tablet_manager.cc:1434] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20251027 21:08:07.494789 10041 rpcz_store.cc:276] 1027 21:08:03.726557 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:03.726612 (+ 55us) service_pool.cc:225] Handling call
1027 21:08:07.031964 (+3305352us) ts_tablet_manager.cc:1151] Deleting tablet aabaa34787cf4d179822269b7f323314
1027 21:08:07.031973 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:07.259172 (+227199us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.423040 (+163868us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.479093 (+ 56053us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3305177}
I20251027 21:08:07.496299 10041 tablet_replica.cc:333] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:07.497026 10041 raft_consensus.cc:2243] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:07.498327 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6b58b1f4f63461aa96e416cae879be8" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:07.498854 9566 leader_election.cc:343] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.483147 11243 tablet.cc:1722] T 6600774e4c7b48479a2fdb02614ba505 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251027 21:08:07.485356 9860 leader_election.cc:343] T 986bf148b3e9482fa2fe53bbac54750d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.500160 9860 leader_election.cc:304] T 986bf148b3e9482fa2fe53bbac54750d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:07.500630 9860 leader_election.cc:343] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.500764 9860 leader_election.cc:304] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.501184 11177 raft_consensus.cc:2749] T 986bf148b3e9482fa2fe53bbac54750d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:07.501374 11177 raft_consensus.cc:2749] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:07.501737 10041 raft_consensus.cc:2272] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:07.465088 10032 raft_consensus.cc:359] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.502803 10032 raft_consensus.cc:385] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.502971 10032 raft_consensus.cc:740] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.503234 10032 consensus_queue.cc:260] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.503820 10032 ts_tablet_manager.cc:1434] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
I20251027 21:08:07.504093 10032 tablet_bootstrap.cc:492] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:07.497673 9907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6b58b1f4f63461aa96e416cae879be8" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:07.504770 9907 raft_consensus.cc:3060] T e6b58b1f4f63461aa96e416cae879be8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:07.505180 10041 ts_tablet_manager.cc:1916] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:07.511868 11250 tablet.cc:1722] T 6600774e4c7b48479a2fdb02614ba505 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251027 21:08:07.461180 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 06d81182b73e46019fc18c828045001f consensus state has no leader
I20251027 21:08:07.515457 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_74f4c8e80cf44de296017c6e93b8899e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:07.516095 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74f4c8e80cf44de296017c6e93b8899e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:07.461325 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6a5071cb8c5a4ef59286b5970c68845b consensus state has no leader
I20251027 21:08:07.517503 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_090fa7c40bdb40d09777fd8458acd6b2" } new_table_name: "default.Table_f46fff92cdc244f6a020795467099e74" modify_external_catalogs: true
I20251027 21:08:07.462417 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 050e9442041d495496bfa7ce25ac6a88
I20251027 21:08:07.520569 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_57de17bcd9ef4ca3b5153678366c719d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:07.521124 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57de17bcd9ef4ca3b5153678366c719d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:07.463107 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 473cc9f814c943a8967d92e58ed7dfd2
I20251027 21:08:07.523159 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_81e4dd46fa6945bc956cc69e26dabcf1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:07.523726 10029 tablet_bootstrap.cc:492] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:07.524012 9605 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1045) took 1003 ms. Trace:
I20251027 21:08:07.524086 9605 rpcz_store.cc:276] 1027 21:08:06.520399 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.757269 (+236870us) service_pool.cc:225] Handling call
1027 21:08:06.757824 (+ 555us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.757854 (+ 30us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.184297 (+426443us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.490438 (+306141us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.523996 (+ 33558us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.525929 10029 tablet_bootstrap.cc:654] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.464114 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet a24dbceaa95e4f43a5ea416a20e7041e
I20251027 21:08:07.529109 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_01129369d93842049a67974b77112073" } new_table_name: "default.Table_59f5f9751881492e9c237e5926a96350" modify_external_catalogs: true
I20251027 21:08:07.532601 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 050e9442041d495496bfa7ce25ac6a88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.467211 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_503f3a993de6475ea92ce352816fc954"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:07.484238 10611 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:08:07.534379 10611 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: This master's current role is: LEADER
I20251027 21:08:07.495909 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet aabaa34787cf4d179822269b7f323314 (table default.Table_0eecf2c874144e7594c825f01fcc5770 [id=a0a10aefbf4c4ab083982e9e46bba513]) successfully deleted
I20251027 21:08:07.535710 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet a24dbceaa95e4f43a5ea416a20e7041e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:08:07.537524 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_503f3a993de6475ea92ce352816fc954 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:07.541972 9745 tablet_service.cc:1552] Processing DeleteTablet for tablet a24dbceaa95e4f43a5ea416a20e7041e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.542972 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet 050e9442041d495496bfa7ce25ac6a88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:07.543588 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet 473cc9f814c943a8967d92e58ed7dfd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:07.543880 9888 tablet_service.cc:1552] Processing DeleteTablet for tablet 473cc9f814c943a8967d92e58ed7dfd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:07.544401 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet a24dbceaa95e4f43a5ea416a20e7041e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:07.545717 9711 consensus_peers.cc:597] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67bb12de9f044c73ae9ab26d10498c90. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:07.547008 10032 tablet_bootstrap.cc:654] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.548722 10029 tablet_bootstrap.cc:492] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:07.548933 10029 ts_tablet_manager.cc:1403] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.003s
I20251027 21:08:07.549863 10029 raft_consensus.cc:359] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:07.550129 10029 raft_consensus.cc:385] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.550256 10029 raft_consensus.cc:740] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.550340 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 473cc9f814c943a8967d92e58ed7dfd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.550611 10029 consensus_queue.cc:260] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:07.551234 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 050e9442041d495496bfa7ce25ac6a88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:07.554034 9308 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet dd9e6656d8df43afbb7f70706bc14585 (table default.Table_f78d839900634631b6963b2d63f0d084 [id=3cdf432ec43d42e8bc2d253a92eac8d3]). Expected version 3 got 0
I20251027 21:08:07.554733 9308 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet aaaf70c950044387a63820079db3c829 (table default.Table_e08f0cbb71674a44a697686c1b6effc9 [id=471f86ddb9de40a7ae58ec202926a1fd]). Expected version 1 got 0
I20251027 21:08:07.555213 10029 ts_tablet_manager.cc:1434] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251027 21:08:07.557667 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ed4f9c61ad54e1292281b30bcd19533 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:07.559605 10029 tablet_bootstrap.cc:492] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:07.561461 9742 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1022) took 1682 ms. Trace:
I20251027 21:08:07.561611 9742 rpcz_store.cc:276] 1027 21:08:05.878949 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.844589 (+965640us) service_pool.cc:225] Handling call
1027 21:08:06.844915 (+ 326us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.844938 (+ 23us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.333715 (+488777us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.561304 (+227589us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.561444 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251027 21:08:07.562884 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81e4dd46fa6945bc956cc69e26dabcf1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:07.564774 10029 tablet_bootstrap.cc:654] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.568264 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet cb95618d9fd54a4f8121b43e717d5616 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:07.568826 10029 tablet_bootstrap.cc:492] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:07.569366 10029 ts_tablet_manager.cc:1403] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20251027 21:08:07.570809 10029 raft_consensus.cc:359] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.572255 10032 tablet_bootstrap.cc:492] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:07.572449 10032 ts_tablet_manager.cc:1403] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.003s
I20251027 21:08:07.573124 10032 raft_consensus.cc:359] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.573357 10032 raft_consensus.cc:385] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.573457 10032 raft_consensus.cc:740] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.573731 10032 consensus_queue.cc:260] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.574312 10032 ts_tablet_manager.cc:1434] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:07.574703 10032 tablet_bootstrap.cc:492] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:07.581318 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c2992034f072482382a4a593fb7f7aaa
I20251027 21:08:07.585268 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet c2992034f072482382a4a593fb7f7aaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:07.585584 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet c2992034f072482382a4a593fb7f7aaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.586886 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c0e2772d66404d74891e0893c49a66e0
I20251027 21:08:07.584740 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet c2992034f072482382a4a593fb7f7aaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:07.590298 10029 raft_consensus.cc:385] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.590389 10029 raft_consensus.cc:740] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.590682 10041 ts_tablet_manager.cc:1929] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:07.590859 10041 log.cc:1199] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/631ff72613734913818a3cad24f20dbc
I20251027 21:08:07.591212 10041 ts_tablet_manager.cc:1950] T 631ff72613734913818a3cad24f20dbc P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:07.591792 10032 tablet_bootstrap.cc:654] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.592793 10029 consensus_queue.cc:260] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.592760 10997 raft_consensus.cc:515] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.593573 10997 leader_election.cc:290] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:07.595474 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4aa1fe05114c1287ca580c1f411d3f" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:07.595665 9635 raft_consensus.cc:3060] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:07.596084 11234 raft_consensus.cc:493] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.596262 11234 raft_consensus.cc:515] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.596930 11234 leader_election.cc:290] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.597306 10029 ts_tablet_manager.cc:1434] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.027s user 0.000s sys 0.001s
I20251027 21:08:07.597681 10029 tablet_bootstrap.cc:492] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:07.598008 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4187b2f66f24055b1df170f1062cc88" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:07.599614 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet c0e2772d66404d74891e0893c49a66e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:07.600335 10029 tablet_bootstrap.cc:654] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:07.600461 9860 leader_election.cc:343] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.601423 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4187b2f66f24055b1df170f1062cc88" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:07.602924 9711 leader_election.cc:343] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 2a4aa1fe05114c1287ca580c1f411d3f
W20251027 21:08:07.603215 9859 leader_election.cc:343] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.603325 9859 leader_election.cc:304] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.604071 11234 raft_consensus.cc:2749] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:07.604450 9907 raft_consensus.cc:2468] T e6b58b1f4f63461aa96e416cae879be8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:07.609587 11254 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20251027 21:08:07.610397 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 935) took 2826 ms. Trace:
I20251027 21:08:07.610486 10040 rpcz_store.cc:276] 1027 21:08:04.783786 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.860050 (+ 76264us) service_pool.cc:225] Handling call
1027 21:08:06.786198 (+1926148us) ts_tablet_manager.cc:1151] Deleting tablet 4debc3f13f3a42939b6b83f6523aab7c
1027 21:08:06.786209 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:07.071515 (+285306us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.322220 (+250705us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.610380 (+288160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1925838}
I20251027 21:08:07.610956 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 939) took 2735 ms. Trace:
I20251027 21:08:07.611032 10040 rpcz_store.cc:276] 1027 21:08:04.875183 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.876724 (+ 1541us) service_pool.cc:225] Handling call
1027 21:08:07.610760 (+2734036us) ts_tablet_manager.cc:1151] Deleting tablet 532ea47b15b544dfb05ddb2bdacf3953
1027 21:08:07.610768 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:07.610881 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2733786}
I20251027 21:08:07.611271 10040 tablet_replica.cc:333] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:07.611449 10040 raft_consensus.cc:2243] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:07.611619 10040 raft_consensus.cc:2272] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:07.602572 9632 raft_consensus.cc:2468] T 29693cf06bc24930870dc60457126519 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 1.
I20251027 21:08:07.604892 9568 leader_election.cc:304] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.613075 9860 leader_election.cc:304] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.613538 11234 raft_consensus.cc:2804] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:07.613699 11234 raft_consensus.cc:697] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Becoming Leader. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Running, Role: LEADER
I20251027 21:08:07.614006 11234 consensus_queue.cc:237] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.615475 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 4debc3f13f3a42939b6b83f6523aab7c (table default.Table_4bd57498972a413ca2bb7de2cd588e8b [id=5ebbc228b61046f8aef2d6fe13f12e57]) successfully deleted
W20251027 21:08:07.615809 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 532ea47b15b544dfb05ddb2bdacf3953 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 532ea47b15b544dfb05ddb2bdacf3953 already in progress: creating tablet
I20251027 21:08:07.616488 11097 raft_consensus.cc:2804] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:07.616600 10040 ts_tablet_manager.cc:1916] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:07.616658 11097 raft_consensus.cc:697] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:07.616910 11097 consensus_queue.cc:237] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.605580 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 947) took 2778 ms. Trace:
I20251027 21:08:07.619656 10043 rpcz_store.cc:276] 1027 21:08:04.827587 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.827656 (+ 69us) service_pool.cc:225] Handling call
1027 21:08:07.194170 (+2366514us) ts_tablet_manager.cc:1151] Deleting tablet ccf10ef559f74e2db28023fd520c8de9
1027 21:08:07.194182 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:07.328180 (+133998us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.429170 (+100990us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.605556 (+176386us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2366319}
I20251027 21:08:07.619964 10043 tablet_replica.cc:333] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:07.620127 10043 raft_consensus.cc:2243] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:07.620234 10043 raft_consensus.cc:2272] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:07.620752 10043 ts_tablet_manager.cc:1916] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:07.625949 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet ccf10ef559f74e2db28023fd520c8de9 (table default.Table_17e67d6acbdd470d9b351522b7908579 [id=bbdedeb45c464c47b96d078308141f9c]) successfully deleted
I20251027 21:08:07.635459 9601 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1041) took 1811 ms. Trace:
I20251027 21:08:07.636690 9601 rpcz_store.cc:276] 1027 21:08:05.824071 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.729806 (+905735us) service_pool.cc:225] Handling call
1027 21:08:06.750103 (+ 20297us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.750126 (+ 23us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.433417 (+683291us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.635321 (+201904us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.635444 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.639902 9600 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59712 (request call id 1043) took 1119 ms. Trace:
I20251027 21:08:07.648290 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet c0e2772d66404d74891e0893c49a66e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:07.648519 9600 rpcz_store.cc:276] 1027 21:08:06.520198 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.761385 (+241187us) service_pool.cc:225] Handling call
1027 21:08:06.766545 (+ 5160us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.766575 (+ 30us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.446587 (+680012us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.639766 (+193179us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.639886 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.648953 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet c0e2772d66404d74891e0893c49a66e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:06 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.655396 10029 tablet_bootstrap.cc:492] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:07.655562 10029 ts_tablet_manager.cc:1403] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.000s
I20251027 21:08:07.656443 10029 raft_consensus.cc:359] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.656677 10029 raft_consensus.cc:385] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.656742 10029 raft_consensus.cc:740] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.659298 10029 consensus_queue.cc:260] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.660552 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet da26c83976df40328cf9b11734fcfa44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:07.663628 10029 ts_tablet_manager.cc:1434] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251027 21:08:07.666999 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_8cdf4f6df0ec4bc4bb7758526f980e58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:07.667681 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8cdf4f6df0ec4bc4bb7758526f980e58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:07.668524 10029 tablet_bootstrap.cc:492] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
W20251027 21:08:07.670488 9710 consensus_peers.cc:597] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 -> Peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Couldn't send request to peer 51902b6b61ff4d5584ff21896e847e14. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67bb12de9f044c73ae9ab26d10498c90. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:07.670635 10029 tablet_bootstrap.cc:654] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.672446 10032 tablet_bootstrap.cc:492] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:07.672654 10032 ts_tablet_manager.cc:1403] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.098s user 0.005s sys 0.001s
I20251027 21:08:07.673602 10032 raft_consensus.cc:359] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.673799 10032 raft_consensus.cc:385] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.673897 10032 raft_consensus.cc:740] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.674124 10032 consensus_queue.cc:260] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.674742 10032 ts_tablet_manager.cc:1434] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251027 21:08:07.675089 10032 tablet_bootstrap.cc:492] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:07.676909 10032 tablet_bootstrap.cc:654] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.682672 11234 raft_consensus.cc:493] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.682838 11234 raft_consensus.cc:515] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.683794 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30ce703650c94c06b93bce976cb5d1fd" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:07.684158 10031 tablet_bootstrap.cc:700] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218: Time spent opening tablet: real 0.212s user 0.001s sys 0.000s
I20251027 21:08:07.684474 10031 tablet_bootstrap.cc:654] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:07.684649 9860 leader_election.cc:343] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.685441 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30ce703650c94c06b93bce976cb5d1fd" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:07.686038 9859 leader_election.cc:343] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.686182 9859 leader_election.cc:304] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.686439 11162 raft_consensus.cc:2749] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:07.686743 11234 leader_election.cc:290] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.690892 10043 ts_tablet_manager.cc:1929] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:07.691087 10043 log.cc:1199] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/c3b9a461567a48fcb4af514ecdf618b4
I20251027 21:08:07.691481 10043 ts_tablet_manager.cc:1950] T c3b9a461567a48fcb4af514ecdf618b4 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:07.692799 10032 tablet_bootstrap.cc:492] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:07.693027 10032 ts_tablet_manager.cc:1403] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.001s
I20251027 21:08:07.693951 10032 raft_consensus.cc:359] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.695667 10032 raft_consensus.cc:385] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.695780 10032 raft_consensus.cc:740] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.696050 10032 consensus_queue.cc:260] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.696579 10032 ts_tablet_manager.cc:1434] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:08:07.696691 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_8fac6e33215e40128ceb1dfe8a622da6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:07.697237 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fac6e33215e40128ceb1dfe8a622da6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:07.698554 10032 tablet_bootstrap.cc:492] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:07.698557 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_1cda8ade41884ccc959f290f1b26ffdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:07.698983 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1cda8ade41884ccc959f290f1b26ffdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:07.700326 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 920) took 3973 ms. Trace:
I20251027 21:08:07.700425 10041 rpcz_store.cc:276] 1027 21:08:03.727020 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:03.727078 (+ 58us) service_pool.cc:225] Handling call
1027 21:08:07.496221 (+3769143us) ts_tablet_manager.cc:1151] Deleting tablet 631ff72613734913818a3cad24f20dbc
1027 21:08:07.496232 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:07.587231 (+ 90999us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.590537 (+ 3306us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.700276 (+109739us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3768974}
I20251027 21:08:07.700769 10041 tablet_replica.cc:333] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:07.700961 10041 raft_consensus.cc:2243] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:07.701076 11097 raft_consensus.cc:493] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.701129 10041 raft_consensus.cc:2272] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:07.701195 11097 raft_consensus.cc:515] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.701678 10041 ts_tablet_manager.cc:1916] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:07.701735 11097 leader_election.cc:290] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.702370 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d672118ecd6f4094855adf106e5f7115" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:07.702931 9566 leader_election.cc:343] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.703907 10029 tablet_bootstrap.cc:492] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:07.704033 10029 ts_tablet_manager.cc:1403] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.001s
I20251027 21:08:07.704592 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 631ff72613734913818a3cad24f20dbc (table default.Table_ed23972576b74b5fbc7167cf7b541bfc [id=e7b7bdb2ea7f468c98bfe88f900a5be8]) successfully deleted
I20251027 21:08:07.704896 9907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d672118ecd6f4094855adf106e5f7115" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:07.705050 9907 raft_consensus.cc:2468] T d672118ecd6f4094855adf106e5f7115 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:07.704991 10029 raft_consensus.cc:359] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.705157 10029 raft_consensus.cc:385] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.705212 10029 raft_consensus.cc:740] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.705408 9568 leader_election.cc:304] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.705472 10029 consensus_queue.cc:260] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.705749 10859 raft_consensus.cc:2804] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:07.705812 10859 raft_consensus.cc:493] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:07.705886 10859 raft_consensus.cc:3060] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:07.706475 10029 ts_tablet_manager.cc:1434] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:07.713084 9635 raft_consensus.cc:2468] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:07.714234 10029 tablet_bootstrap.cc:492] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:07.715435 9710 leader_election.cc:304] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:07.716728 10997 raft_consensus.cc:2804] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:07.716881 10997 raft_consensus.cc:697] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:07.717123 10997 consensus_queue.cc:237] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.717234 10029 tablet_bootstrap.cc:654] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.717677 10032 tablet_bootstrap.cc:654] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.720360 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet 76e4fb2bea7f4a4696fe9f6bb892bccc with delete_type TABLET_DATA_DELETED (replaced by tablet 5cb8ca3a50c148e6b602e949650353f8) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:07.720400 11234 raft_consensus.cc:493] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.720558 11234 raft_consensus.cc:515] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.721364 11234 leader_election.cc:290] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.722157 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "823620e5486e4cf982fdf9105e8cc4f1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:07.722133 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "823620e5486e4cf982fdf9105e8cc4f1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:07.722632 9860 leader_election.cc:343] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:07.722780 9859 leader_election.cc:343] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.722905 9859 leader_election.cc:304] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.723377 11234 raft_consensus.cc:2749] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:07.723997 9566 consensus_peers.cc:597] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:07.724208 11234 raft_consensus.cc:493] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51902b6b61ff4d5584ff21896e847e14)
I20251027 21:08:07.724361 11234 raft_consensus.cc:515] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.724884 11234 leader_election.cc:290] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.725781 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "345d3795a37b4a40b40967cfacf0f893" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:07.726006 9774 raft_consensus.cc:2393] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fbd567f929a4f6d8e4d78feef7ed678 in current term 2: Already voted for candidate 77b0513f17c0472088901a1ce8b08218 in this term.
I20251027 21:08:07.726610 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "345d3795a37b4a40b40967cfacf0f893" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:07.726964 9635 raft_consensus.cc:2468] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 2.
I20251027 21:08:07.728013 9860 leader_election.cc:304] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:07.730312 9566 consensus_peers.cc:597] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:07.730475 10029 tablet_bootstrap.cc:492] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:07.730648 10029 ts_tablet_manager.cc:1403] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.001s
W20251027 21:08:07.730891 9566 consensus_peers.cc:597] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:07.731114 11234 raft_consensus.cc:2804] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251027 21:08:07.731212 11234 raft_consensus.cc:493] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 51902b6b61ff4d5584ff21896e847e14)
I20251027 21:08:07.731356 11234 raft_consensus.cc:3060] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Advancing to term 2
I20251027 21:08:07.731691 10029 raft_consensus.cc:359] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.738191 10029 raft_consensus.cc:385] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.738368 10029 raft_consensus.cc:740] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.738713 10029 consensus_queue.cc:260] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.741927 10029 ts_tablet_manager.cc:1434] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20251027 21:08:07.742017 10926 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "77b0513f17c0472088901a1ce8b08218" instance_seqno: 1761599266252874) as {username='slave'} at 127.9.14.66:36571; Asking this server to re-register.
I20251027 21:08:07.742843 10032 tablet_bootstrap.cc:492] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:07.743512 9820 heartbeater.cc:461] Registering TS with master...
I20251027 21:08:07.743649 10032 ts_tablet_manager.cc:1403] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.045s user 0.000s sys 0.004s
I20251027 21:08:07.744578 10032 raft_consensus.cc:359] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.744787 10032 raft_consensus.cc:385] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.744874 10032 raft_consensus.cc:740] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.745102 10032 consensus_queue.cc:260] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.745744 10032 ts_tablet_manager.cc:1434] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:07.748262 10032 tablet_bootstrap.cc:492] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:07.748499 10926 ts_manager.cc:194] Registered new tserver with Master: 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.748952 10029 tablet_bootstrap.cc:492] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
W20251027 21:08:07.750223 9307 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.9.14.65:32849 (request call id 181) took 758 ms (client timeout 1000 ms). Trace:
I20251027 21:08:07.751631 10032 tablet_bootstrap.cc:654] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:07.750387 9307 rpcz_store.cc:269] 1027 21:08:06.991550 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.162963 (+171413us) service_pool.cc:225] Handling call
1027 21:08:07.750196 (+587233us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:08:07.508906 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:08:07.509000 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:08:07.509005 (+ 5us) write_op.cc:435] Acquired schema lock
1027 21:08:07.509007 (+ 2us) tablet.cc:660] Decoding operations
1027 21:08:07.509127 (+ 120us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:08:07.509138 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1027 21:08:07.509142 (+ 4us) tablet.cc:683] Acquiring locks for 3 operations
1027 21:08:07.509223 (+ 81us) tablet.cc:699] Row locks acquired
1027 21:08:07.509225 (+ 2us) write_op.cc:260] PREPARE: finished
1027 21:08:07.509257 (+ 32us) write_op.cc:270] Start()
1027 21:08:07.509296 (+ 39us) write_op.cc:276] Timestamp: P: 1761599287509222 usec, L: 0
1027 21:08:07.509300 (+ 4us) op_driver.cc:348] REPLICATION: starting
1027 21:08:07.509457 (+ 157us) log.cc:844] Serialized 1275 byte log entry
1027 21:08:07.661212 (+151755us) op_driver.cc:464] REPLICATION: finished
1027 21:08:07.661307 (+ 95us) write_op.cc:301] APPLY: starting
1027 21:08:07.661321 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1027 21:08:07.661367 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
1027 21:08:07.661392 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:08:07.661467 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:08:07.661511 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
1027 21:08:07.661518 (+ 7us) write_op.cc:312] APPLY: finished
1027 21:08:07.661593 (+ 75us) log.cc:844] Serialized 35 byte log entry
1027 21:08:07.661652 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:07.661687 (+ 35us) write_op.cc:454] Released schema lock
1027 21:08:07.661693 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":43,"apply.run_cpu_time_us":448,"apply.run_wall_time_us":447,"num_ops":3,"prepare.queue_time_us":104751,"prepare.run_cpu_time_us":630,"prepare.run_wall_time_us":886,"replication_time_us":151882}]]}
I20251027 21:08:07.754488 10029 tablet_bootstrap.cc:654] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.754560 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b6554b3e96a841fb975ff74ff767069a" } new_table_name: "default.Table_a70f309c799d4525bc2caae9e0ec9356" modify_external_catalogs: true
I20251027 21:08:07.760198 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 949) took 2920 ms. Trace:
I20251027 21:08:07.760428 10043 rpcz_store.cc:276] 1027 21:08:04.839565 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.839612 (+ 47us) service_pool.cc:225] Handling call
1027 21:08:07.619924 (+2780312us) ts_tablet_manager.cc:1151] Deleting tablet c3b9a461567a48fcb4af514ecdf618b4
1027 21:08:07.619933 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:07.687347 (+ 67414us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.690799 (+ 3452us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.760181 (+ 69382us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2780167}
I20251027 21:08:07.760654 9907 raft_consensus.cc:1275] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Refusing update from remote peer 51902b6b61ff4d5584ff21896e847e14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 21:08:07.760787 10043 tablet_replica.cc:333] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:07.761021 10043 raft_consensus.cc:2243] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:07.761175 10043 raft_consensus.cc:2272] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:07.761687 10043 ts_tablet_manager.cc:1916] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:07.762220 10032 tablet_bootstrap.cc:492] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:07.762382 10032 ts_tablet_manager.cc:1403] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20251027 21:08:07.765532 10032 raft_consensus.cc:359] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:07.765756 10032 raft_consensus.cc:385] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.765834 10032 raft_consensus.cc:740] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.766088 10032 consensus_queue.cc:260] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:07.766726 10032 ts_tablet_manager.cc:1434] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251027 21:08:07.766889 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet c3b9a461567a48fcb4af514ecdf618b4 (table default.Table_fc2da109d1934a3fbe14ae51b20e771d [id=56f8db49ad8c4670a73a29bf4ed613e9]) successfully deleted
I20251027 21:08:07.767117 11097 consensus_queue.cc:1048] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251027 21:08:07.767413 9751 consensus_meta.cc:287] T 0d75e6a4286e428eb85b24638382d23b P 77b0513f17c0472088901a1ce8b08218: Time spent flushing consensus metadata: real 0.582s user 0.020s sys 0.001s
I20251027 21:08:07.767745 9751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40732 (request call id 1021) took 1889 ms. Trace:
I20251027 21:08:07.767822 9751 rpcz_store.cc:276] 1027 21:08:05.878799 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.837591 (+958792us) service_pool.cc:225] Handling call
1027 21:08:06.875912 (+ 38321us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1027 21:08:06.875931 (+ 19us) ts_tablet_manager.cc:682] Creating new metadata...
1027 21:08:07.185671 (+309740us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.767580 (+581909us) tablet_replica.cc:206] Creating consensus instance
1027 21:08:07.767728 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:07.770963 10029 tablet_bootstrap.cc:492] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:07.771207 10029 ts_tablet_manager.cc:1403] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.002s
I20251027 21:08:07.772589 10029 raft_consensus.cc:359] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.772884 10029 raft_consensus.cc:385] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.773027 10029 raft_consensus.cc:740] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.773368 10029 consensus_queue.cc:260] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.777328 10029 ts_tablet_manager.cc:1434] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251027 21:08:07.777760 10032 tablet_bootstrap.cc:492] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:07.778090 10029 tablet_bootstrap.cc:492] T 9fa39697360c46138ef4cd21e8c7a950 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:07.781152 10032 tablet_bootstrap.cc:654] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.781333 10029 tablet_bootstrap.cc:654] T 9fa39697360c46138ef4cd21e8c7a950 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.785090 11057 raft_consensus.cc:493] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.785319 11097 raft_consensus.cc:493] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.785447 11097 raft_consensus.cc:515] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:07.786180 11097 leader_election.cc:290] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:07.786423 11057 raft_consensus.cc:515] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.786931 10031 tablet_bootstrap.cc:492] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:07.787024 11057 leader_election.cc:290] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.787045 10031 ts_tablet_manager.cc:1403] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.316s user 0.002s sys 0.001s
I20251027 21:08:07.788131 9920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c34c8f1433b40d491ae05e7c6e60ba1" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:07.789026 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c34c8f1433b40d491ae05e7c6e60ba1" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:07.789685 9568 leader_election.cc:343] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 631ff72613734913818a3cad24f20dbc
W20251027 21:08:07.789917 9568 leader_election.cc:343] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Illegal state: Tablet not RUNNING: STOPPED
I20251027 21:08:07.790516 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "631ff72613734913818a3cad24f20dbc" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:07.790715 10859 raft_consensus.cc:515] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.791260 10859 leader_election.cc:290] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:07.794890 9566 leader_election.cc:343] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.795003 9566 leader_election.cc:304] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.795078 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d672118ecd6f4094855adf106e5f7115" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:07.795393 9566 leader_election.cc:343] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.795532 9566 leader_election.cc:304] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:07.795940 9566 leader_election.cc:343] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.794704 11234 raft_consensus.cc:515] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.796087 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d672118ecd6f4094855adf106e5f7115" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:07.796757 9909 raft_consensus.cc:3060] T d672118ecd6f4094855adf106e5f7115 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:07.797231 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "345d3795a37b4a40b40967cfacf0f893" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:07.797415 9774 raft_consensus.cc:2393] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fbd567f929a4f6d8e4d78feef7ed678 in current term 2: Already voted for candidate 77b0513f17c0472088901a1ce8b08218 in this term.
I20251027 21:08:07.797847 11234 leader_election.cc:290] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 2 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.811058 11057 raft_consensus.cc:2749] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:07.811687 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "345d3795a37b4a40b40967cfacf0f893" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:07.813668 11097 consensus_queue.cc:1048] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20251027 21:08:07.815371 10859 raft_consensus.cc:2749] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:07.816777 10926 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" instance_seqno: 1761599265362526) as {username='slave'} at 127.9.14.65:54825; Asking this server to re-register.
I20251027 21:08:07.819022 9682 heartbeater.cc:461] Registering TS with master...
I20251027 21:08:07.820727 10040 ts_tablet_manager.cc:1929] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:07.826045 10926 ts_manager.cc:194] Registered new tserver with Master: 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:07.827366 10040 log.cc:1199] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/d94c695320f24a7f9776c493e462ed24
I20251027 21:08:07.836217 10040 ts_tablet_manager.cc:1950] T d94c695320f24a7f9776c493e462ed24 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:07.838865 10029 tablet_bootstrap.cc:492] T 9fa39697360c46138ef4cd21e8c7a950 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:07.839306 11234 raft_consensus.cc:493] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.839524 11234 raft_consensus.cc:515] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.841460 11234 leader_election.cc:290] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:07.845708 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe94124b239e442dbf644e4a483615fc consensus state has no leader
I20251027 21:08:07.846168 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c30777b91b944d75a8b8758a620faf1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:07.846755 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c30777b91b944d75a8b8758a620faf1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:07.860357 11250 tablet.cc:1722] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251027 21:08:07.840622 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28a2e85021394f9d8a9ea524959b52a1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:07.860818 10029 ts_tablet_manager.cc:1403] T 9fa39697360c46138ef4cd21e8c7a950 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.083s user 0.006s sys 0.003s
I20251027 21:08:07.855294 11243 tablet.cc:1722] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251027 21:08:07.861501 10029 raft_consensus.cc:359] T 9fa39697360c46138ef4cd21e8c7a950 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
W20251027 21:08:07.861588 9860 leader_election.cc:343] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.861961 10029 raft_consensus.cc:385] T 9fa39697360c46138ef4cd21e8c7a950 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.862119 10029 raft_consensus.cc:740] T 9fa39697360c46138ef4cd21e8c7a950 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.862458 10859 raft_consensus.cc:493] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.862453 10029 consensus_queue.cc:260] T 9fa39697360c46138ef4cd21e8c7a950 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.862684 10859 raft_consensus.cc:515] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.863216 10029 ts_tablet_manager.cc:1434] T 9fa39697360c46138ef4cd21e8c7a950 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:07.863545 10029 tablet_bootstrap.cc:492] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:07.863864 10859 leader_election.cc:290] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:07.864493 9568 leader_election.cc:343] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: a4d26f73b8534f08bcaa386d4de75c01
I20251027 21:08:07.868708 10029 tablet_bootstrap.cc:654] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.869185 10926 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" instance_seqno: 1761599266734167) as {username='slave'} at 127.9.14.67:44783; Asking this server to re-register.
I20251027 21:08:07.841332 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28a2e85021394f9d8a9ea524959b52a1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:07.866549 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4d26f73b8534f08bcaa386d4de75c01" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:07.870280 9859 leader_election.cc:343] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.870404 9859 leader_election.cc:304] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.870860 11162 raft_consensus.cc:2749] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:07.827841 10031 raft_consensus.cc:359] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.871253 10031 raft_consensus.cc:385] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.871331 10031 raft_consensus.cc:740] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.871588 10031 consensus_queue.cc:260] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:07.870687 9566 leader_election.cc:343] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.880168 9566 leader_election.cc:304] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.880640 11057 raft_consensus.cc:2749] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:07.876350 10859 consensus_queue.cc:1048] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:07.883965 11162 raft_consensus.cc:493] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.884085 11162 raft_consensus.cc:515] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.884658 11162 leader_election.cc:290] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.886564 10859 raft_consensus.cc:493] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.891345 9909 raft_consensus.cc:2468] T d672118ecd6f4094855adf106e5f7115 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:07.891839 11162 raft_consensus.cc:493] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.891950 11162 raft_consensus.cc:515] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.892318 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27bd3b2a011648b2b8f2f92944513751" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:07.892448 11162 leader_election.cc:290] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.893169 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cdbde6272af147829ecadd67d634f049" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:07.893658 9860 leader_election.cc:343] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:07.896436 9859 leader_election.cc:343] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.897012 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cdbde6272af147829ecadd67d634f049" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:07.898336 9859 leader_election.cc:343] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.898437 9859 leader_election.cc:304] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.898876 11162 raft_consensus.cc:2749] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:07.899689 10859 raft_consensus.cc:515] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:07.900290 10859 leader_election.cc:290] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:07.900905 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fde0980aa64c4496861a662f7ed3ef75" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:07.904847 10041 ts_tablet_manager.cc:1929] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:07.904989 10041 log.cc:1199] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/5c34c8f1433b40d491ae05e7c6e60ba1
I20251027 21:08:07.905318 10041 ts_tablet_manager.cc:1950] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:07.905599 9635 raft_consensus.cc:2468] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 2.
I20251027 21:08:07.906047 9860 leader_election.cc:304] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.906394 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27bd3b2a011648b2b8f2f92944513751" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:07.906888 9860 leader_election.cc:343] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.906993 9860 leader_election.cc:304] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.907734 9568 leader_election.cc:304] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.908532 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fde0980aa64c4496861a662f7ed3ef75" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:07.908720 9909 raft_consensus.cc:2468] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:07.909211 11162 raft_consensus.cc:2804] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 2 FOLLOWER]: Leader election won for term 2
I20251027 21:08:07.909346 11162 raft_consensus.cc:697] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 2 LEADER]: Becoming Leader. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Running, Role: LEADER
I20251027 21:08:07.909452 11177 raft_consensus.cc:2749] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:07.909606 11162 consensus_queue.cc:237] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:07.910917 9566 leader_election.cc:343] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.911371 9568 leader_election.cc:304] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.911795 11057 raft_consensus.cc:2804] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:07.911877 11057 raft_consensus.cc:493] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:07.911962 11057 raft_consensus.cc:3060] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:07.914541 10043 ts_tablet_manager.cc:1929] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:07.915148 10043 log.cc:1199] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/2a4aa1fe05114c1287ca580c1f411d3f
I20251027 21:08:07.915699 10043 ts_tablet_manager.cc:1950] T 2a4aa1fe05114c1287ca580c1f411d3f P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:07.914994 10859 raft_consensus.cc:2804] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:07.917726 10859 raft_consensus.cc:697] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:07.919034 10859 consensus_queue.cc:237] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.921275 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 946) took 2719 ms. Trace:
I20251027 21:08:07.921399 10040 rpcz_store.cc:276] 1027 21:08:05.202256 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.224744 (+ 22488us) service_pool.cc:225] Handling call
1027 21:08:07.611231 (+2386487us) ts_tablet_manager.cc:1151] Deleting tablet d94c695320f24a7f9776c493e462ed24
1027 21:08:07.611238 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:07.758272 (+147034us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.819850 (+ 61578us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.921258 (+101408us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2144997}
I20251027 21:08:07.921640 10029 tablet_bootstrap.cc:492] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:07.922293 10029 ts_tablet_manager.cc:1403] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.059s user 0.008s sys 0.004s
I20251027 21:08:07.922282 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet d94c695320f24a7f9776c493e462ed24 (table default.Table_3da5a86d9f104d1ba39726b5a6dcec8a [id=da40d7d63ce348dd8a275cdd54f8f018]) successfully deleted
I20251027 21:08:07.923409 10031 ts_tablet_manager.cc:1434] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.136s user 0.001s sys 0.001s
I20251027 21:08:07.931653 10029 raft_consensus.cc:359] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.932964 10029 raft_consensus.cc:385] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.933221 10029 raft_consensus.cc:740] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.933552 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_33a9f7934bc7496eaf4da402364c5179"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:07.933717 10029 consensus_queue.cc:260] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
W20251027 21:08:07.934188 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33a9f7934bc7496eaf4da402364c5179 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:07.934839 10029 ts_tablet_manager.cc:1434] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251027 21:08:07.935694 10029 tablet_bootstrap.cc:492] T 83f5fbf5757a49df93ab949b7657c61f P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:07.940052 10032 tablet_bootstrap.cc:492] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:07.942698 10032 ts_tablet_manager.cc:1403] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.165s user 0.027s sys 0.000s
I20251027 21:08:07.922031 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 988) took 2445 ms. Trace:
I20251027 21:08:07.944550 10040 rpcz_store.cc:276] 1027 21:08:05.476816 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.084189 (+607373us) service_pool.cc:225] Handling call
1027 21:08:07.921922 (+1837733us) ts_tablet_manager.cc:1151] Deleting tablet 83f5fbf5757a49df93ab949b7657c61f
1027 21:08:07.921931 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:07.922015 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1800808}
I20251027 21:08:07.944967 10031 tablet_bootstrap.cc:492] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:08:07.945374 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 83f5fbf5757a49df93ab949b7657c61f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83f5fbf5757a49df93ab949b7657c61f already in progress: creating tablet
I20251027 21:08:07.946956 10031 tablet_bootstrap.cc:654] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.953763 9970 heartbeater.cc:461] Registering TS with master...
I20251027 21:08:07.954457 10040 tablet_replica.cc:333] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:07.943714 10032 raft_consensus.cc:359] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.954856 10032 raft_consensus.cc:385] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.954942 10032 raft_consensus.cc:740] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.955253 10040 raft_consensus.cc:2243] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:07.956075 10029 tablet_bootstrap.cc:654] T 83f5fbf5757a49df93ab949b7657c61f P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.955191 10032 consensus_queue.cc:260] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.957733 10032 ts_tablet_manager.cc:1434] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20251027 21:08:07.958055 10032 tablet_bootstrap.cc:492] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:07.960875 10040 raft_consensus.cc:2272] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:07.960927 10032 tablet_bootstrap.cc:654] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.961644 10040 ts_tablet_manager.cc:1916] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:07.962409 10029 tablet_bootstrap.cc:492] T 83f5fbf5757a49df93ab949b7657c61f P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:07.962440 9310 master_service.cc:954] ReplaceTablet: received request to replace tablet 750b27c74ba3473dad743d60c21912c5 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:07.962581 10029 ts_tablet_manager.cc:1403] T 83f5fbf5757a49df93ab949b7657c61f P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.003s
I20251027 21:08:07.963301 10029 raft_consensus.cc:359] T 83f5fbf5757a49df93ab949b7657c61f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.963481 10029 raft_consensus.cc:385] T 83f5fbf5757a49df93ab949b7657c61f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.963583 10029 raft_consensus.cc:740] T 83f5fbf5757a49df93ab949b7657c61f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.965953 10029 consensus_queue.cc:260] T 83f5fbf5757a49df93ab949b7657c61f P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.966559 11162 raft_consensus.cc:493] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.966715 11162 raft_consensus.cc:515] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.967130 10032 tablet_bootstrap.cc:492] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:07.967249 10032 ts_tablet_manager.cc:1403] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20251027 21:08:07.967370 11162 leader_election.cc:290] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.967916 10032 raft_consensus.cc:359] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.968067 10032 raft_consensus.cc:385] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:07.968118 10032 raft_consensus.cc:740] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.968096 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd9e6656d8df43afbb7f70706bc14585" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:07.968276 10032 consensus_queue.cc:260] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:07.968696 9860 leader_election.cc:343] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.968756 10032 ts_tablet_manager.cc:1434] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:07.969017 10032 tablet_bootstrap.cc:492] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:07.969370 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd9e6656d8df43afbb7f70706bc14585" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:07.969956 9859 leader_election.cc:343] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:07.970053 9859 leader_election.cc:304] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:07.970680 11162 raft_consensus.cc:2749] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:07.974788 9305 master_service.cc:954] ReplaceTablet: received request to replace tablet 42371496a7494e77a7bf1169de5b083a from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:07.974788 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet 9b9ffa18afa54f2c9ffd7b5a03a3a324 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:07.975493 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_56d94ef7571b42b4ba4e66f6ce476ead" } new_table_name: "default.Table_02661276431440f2a5613113911debd1" modify_external_catalogs: true
I20251027 21:08:07.975421 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3476d7c0b579463789510f1632676738"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:07.975945 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3476d7c0b579463789510f1632676738 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:07.980865 9278 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 70854us
I20251027 21:08:07.981434 10032 tablet_bootstrap.cc:654] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:07.982331 10029 ts_tablet_manager.cc:1434] T 83f5fbf5757a49df93ab949b7657c61f P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251027 21:08:07.984472 10031 tablet_bootstrap.cc:492] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:07.984750 11234 raft_consensus.cc:493] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.984928 11234 raft_consensus.cc:515] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:07.985677 10029 tablet_bootstrap.cc:492] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:07.985864 11142 raft_consensus.cc:493] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.986318 11234 leader_election.cc:290] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.989193 11142 raft_consensus.cc:515] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:07.991823 9920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2710f265ae9485dafec389168328067" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:07.992017 9920 raft_consensus.cc:2468] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:07.993357 11142 leader_election.cc:290] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:07.998435 10859 raft_consensus.cc:493] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:07.998620 10927 ts_manager.cc:194] Registered new tserver with Master: 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:07.998587 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cccd253ae2a46328dc9f767b8e9ee97" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:07.999099 9860 leader_election.cc:343] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.006439 10859 raft_consensus.cc:515] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:08.007123 10859 leader_election.cc:290] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.011091 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 950) took 3171 ms. Trace:
I20251027 21:08:08.011229 10043 rpcz_store.cc:276] 1027 21:08:04.839865 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.839939 (+ 74us) service_pool.cc:225] Handling call
1027 21:08:07.760730 (+2920791us) ts_tablet_manager.cc:1151] Deleting tablet 2a4aa1fe05114c1287ca580c1f411d3f
1027 21:08:07.760738 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:07.791605 (+ 30867us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.914413 (+122808us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:08.011071 (+ 96658us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2920661}
I20251027 21:08:08.011624 10043 tablet_replica.cc:333] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:08.011868 10043 raft_consensus.cc:2243] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:08.012050 10043 raft_consensus.cc:2272] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:08.012606 10043 ts_tablet_manager.cc:1916] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:08.013805 11057 raft_consensus.cc:515] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:08.014391 11057 leader_election.cc:290] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:08.016338 10029 tablet_bootstrap.cc:654] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.021163 10032 tablet_bootstrap.cc:492] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.021310 10032 ts_tablet_manager.cc:1403] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.000s
I20251027 21:08:08.022054 10032 raft_consensus.cc:359] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.022264 10032 raft_consensus.cc:385] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.022415 10032 raft_consensus.cc:740] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.022678 10032 consensus_queue.cc:260] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.023241 10032 ts_tablet_manager.cc:1434] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:08.023532 10032 tablet_bootstrap.cc:492] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:08.025146 10032 tablet_bootstrap.cc:654] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.025760 11162 raft_consensus.cc:493] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.025974 11162 raft_consensus.cc:515] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:08.026806 11162 leader_election.cc:290] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:08.028056 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c2d713d72fd4b5ca34cb3e828c75761" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:08.028242 9632 raft_consensus.cc:2468] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 0.
I20251027 21:08:08.028744 9860 leader_election.cc:304] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:08.029107 11162 raft_consensus.cc:2804] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:08.029237 11162 raft_consensus.cc:493] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:08.029363 11162 raft_consensus.cc:3060] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:08.033236 9920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b8156d9ab2e4dfba383ddc72137aab4" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:08.033272 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 2a4aa1fe05114c1287ca580c1f411d3f (table default.Table_f08509866aae44609f26ce5e30963f6c [id=63bc2d46b97e4054b557025436c371d9]) successfully deleted
I20251027 21:08:08.033392 9920 raft_consensus.cc:2468] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:08.033530 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fde0980aa64c4496861a662f7ed3ef75" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:08.033690 9909 raft_consensus.cc:3060] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:08.034261 9568 leader_election.cc:304] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:08.034466 11142 raft_consensus.cc:2804] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:08.034518 11142 raft_consensus.cc:493] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:08.034567 11142 raft_consensus.cc:3060] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:08.036968 9568 leader_election.cc:304] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:08.037180 11283 raft_consensus.cc:2804] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:08.037261 11283 raft_consensus.cc:493] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:08.037326 11283 raft_consensus.cc:3060] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:08.038254 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 921) took 4310 ms. Trace:
I20251027 21:08:08.038362 10041 rpcz_store.cc:276] 1027 21:08:03.727647 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:03.727708 (+ 61us) service_pool.cc:225] Handling call
1027 21:08:07.700689 (+3972981us) ts_tablet_manager.cc:1151] Deleting tablet 5c34c8f1433b40d491ae05e7c6e60ba1
1027 21:08:07.700697 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:07.798786 (+ 98089us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:07.904737 (+105951us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:08.038234 (+133497us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3972834}
I20251027 21:08:08.038944 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 922) took 4247 ms. Trace:
I20251027 21:08:08.038993 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 5c34c8f1433b40d491ae05e7c6e60ba1 (table default.Table_5877315061df40eb80ba9d966632471a [id=c70fb6448778479fa17e811b21830ea5]) successfully deleted
I20251027 21:08:08.039033 10041 rpcz_store.cc:276] 1027 21:08:03.791004 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:03.791070 (+ 66us) service_pool.cc:225] Handling call
1027 21:08:08.038827 (+4247757us) ts_tablet_manager.cc:1151] Deleting tablet 19f3e1a9db434c519e26e864eb790706
1027 21:08:08.038839 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.038929 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4247576}
I20251027 21:08:08.039354 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 923) took 4117 ms. Trace:
I20251027 21:08:08.039438 10041 rpcz_store.cc:276] 1027 21:08:03.921558 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:03.921627 (+ 69us) service_pool.cc:225] Handling call
1027 21:08:08.039246 (+4117619us) ts_tablet_manager.cc:1151] Deleting tablet 6bbee5f1fb704709b6f166a153e69808
1027 21:08:08.039254 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.039340 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4117423}
W20251027 21:08:08.039528 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 19f3e1a9db434c519e26e864eb790706 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19f3e1a9db434c519e26e864eb790706 already in progress: creating tablet
I20251027 21:08:08.039707 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 924) took 4037 ms. Trace:
I20251027 21:08:08.039785 10041 rpcz_store.cc:276] 1027 21:08:04.002310 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.002838 (+ 528us) service_pool.cc:225] Handling call
1027 21:08:08.039623 (+4036785us) ts_tablet_manager.cc:1151] Deleting tablet d03918b19787449a88e0f17a925ce599
1027 21:08:08.039631 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.039694 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4036590}
I20251027 21:08:08.040051 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 925) took 4037 ms. Trace:
W20251027 21:08:08.040124 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 6bbee5f1fb704709b6f166a153e69808 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bbee5f1fb704709b6f166a153e69808 already in progress: creating tablet
I20251027 21:08:08.040127 10041 rpcz_store.cc:276] 1027 21:08:04.002468 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.003065 (+ 597us) service_pool.cc:225] Handling call
1027 21:08:08.039966 (+4036901us) ts_tablet_manager.cc:1151] Deleting tablet 9a2432f5c24b4bbca744d5f4806f2b58
1027 21:08:08.039974 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.040038 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4036799}
W20251027 21:08:08.040321 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet d03918b19787449a88e0f17a925ce599 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d03918b19787449a88e0f17a925ce599 already in progress: creating tablet
I20251027 21:08:08.040426 10041 tablet_replica.cc:333] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:08.040633 10041 raft_consensus.cc:2243] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:08.040796 10041 raft_consensus.cc:2272] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:08.040954 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 9a2432f5c24b4bbca744d5f4806f2b58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a2432f5c24b4bbca744d5f4806f2b58 already in progress: creating tablet
I20251027 21:08:08.041324 10041 ts_tablet_manager.cc:1916] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:08.051424 11243 tablet.cc:1722] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:08.053376 11250 tablet.cc:1722] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:08.060746 10032 tablet_bootstrap.cc:492] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.060876 10032 ts_tablet_manager.cc:1403] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20251027 21:08:08.061586 10032 raft_consensus.cc:359] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.061765 10032 raft_consensus.cc:385] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.061821 10032 raft_consensus.cc:740] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:07.986187 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cccd253ae2a46328dc9f767b8e9ee97" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:07.997220 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2710f265ae9485dafec389168328067" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:08.062037 10032 consensus_queue.cc:260] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.007799 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b8156d9ab2e4dfba383ddc72137aab4" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:08.062511 10032 ts_tablet_manager.cc:1434] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:08.062768 10032 tablet_bootstrap.cc:492] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:08:08.062810 9859 leader_election.cc:343] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.062937 9859 leader_election.cc:304] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.015062 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fde0980aa64c4496861a662f7ed3ef75" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:08.027379 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c2d713d72fd4b5ca34cb3e828c75761" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:08.063359 11177 raft_consensus.cc:2749] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:08.063412 9566 leader_election.cc:343] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:08.063747 9566 leader_election.cc:343] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:08.063977 9859 leader_election.cc:343] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:08.064144 9566 leader_election.cc:343] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.064345 10031 ts_tablet_manager.cc:1403] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.119s user 0.003s sys 0.000s
I20251027 21:08:08.064599 10032 tablet_bootstrap.cc:654] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.065550 10031 raft_consensus.cc:359] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.065717 10031 raft_consensus.cc:385] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.065846 10031 raft_consensus.cc:740] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.066077 10031 consensus_queue.cc:260] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.073724 11234 raft_consensus.cc:493] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.073850 11234 raft_consensus.cc:515] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.074507 11234 leader_election.cc:290] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.075606 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaaf70c950044387a63820079db3c829" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:08.076023 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaaf70c950044387a63820079db3c829" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:08.076575 9860 leader_election.cc:343] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:08.076944 9859 leader_election.cc:343] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.077035 9859 leader_election.cc:304] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.077497 11234 raft_consensus.cc:2749] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:08.079023 10029 tablet_bootstrap.cc:492] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:08.079174 10029 ts_tablet_manager.cc:1403] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.094s user 0.008s sys 0.000s
I20251027 21:08:08.079876 10029 raft_consensus.cc:359] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:08.080022 10029 raft_consensus.cc:385] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.081171 10029 raft_consensus.cc:740] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.081432 10029 consensus_queue.cc:260] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:08.089243 11244 tablet.cc:1722] T ca477937278f45b4b115fec715f32a92 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251027 21:08:08.094365 11142 raft_consensus.cc:515] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:08.095784 11142 leader_election.cc:290] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.095834 11296 tablet.cc:1722] T ca477937278f45b4b115fec715f32a92 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251027 21:08:08.096439 11297 tablet.cc:1722] T ca477937278f45b4b115fec715f32a92 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251027 21:08:08.097635 9309 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 9fde4fd91943498d8888d1783005e492 (table default.Table_87ac731a20bf4f6bbce8d82c1ad3bd97 [id=3a0c8b19df934d1580349b061fe0e8f5]). Expected version 1 got 0
I20251027 21:08:08.099188 10029 ts_tablet_manager.cc:1434] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.020s user 0.001s sys 0.000s
I20251027 21:08:08.099452 10029 tablet_bootstrap.cc:492] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:08.099522 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_93662efa6bc64f759a615db6db157bab" } new_table_name: "default.Table_ad2a3d4173ca45948828915983300f00" modify_external_catalogs: true
I20251027 21:08:08.101011 10029 tablet_bootstrap.cc:654] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.101051 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 27bd3b2a011648b2b8f2f92944513751 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:08.101244 9888 tablet_service.cc:1552] Processing DeleteTablet for tablet 28a2e85021394f9d8a9ea524959b52a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:08.101418 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet 842c1a1298234d9ea2719bab29c54ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:08.103474 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 9fde4fd91943498d8888d1783005e492 (table default.Table_87ac731a20bf4f6bbce8d82c1ad3bd97 [id=3a0c8b19df934d1580349b061fe0e8f5]). Expected version 1 got 0
I20251027 21:08:08.103708 9312 catalog_manager.cc:5649] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } }
I20251027 21:08:08.104066 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet e6b58b1f4f63461aa96e416cae879be8 (table default.Table_f3c2de263e8849caaef87b570fc69459 [id=ef50ef2238804c98ad19c5b1c0b881c4]). Expected version 1 got 0
I20251027 21:08:08.104278 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet d672118ecd6f4094855adf106e5f7115 (table default.Table_1b066b58b2c649a2887ed91461f15d7c [id=887ea33c2fbe4573a1d7169dfd226567]). Expected version 2 got 0
I20251027 21:08:08.104379 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 29693cf06bc24930870dc60457126519 (table default.Table_c09cc86fc2a64450af56fc584302da69 [id=d46947a29ddf4d94b5a9da22d9e5383e]). Expected version 3 got 0
I20251027 21:08:08.104476 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet f2710f265ae9485dafec389168328067 (table default.Table_eb44e5cb7bc24597a807d81c1207a162 [id=27d1cff8fc7940c9abb741c8c7d87e24]). Expected version 2 got 0
I20251027 21:08:08.104744 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet d56d323ef0e74344a479940b93fea11a (table default.Table_cf11a8c344c544c5ba54672fee38e4aa [id=71e02b21c9654e45a41d3e450ac3fbe1]). Expected version 2 got 0
I20251027 21:08:08.104890 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet ca477937278f45b4b115fec715f32a92 (table default.Table_69627cab65c2407b9226a05106b7bdf4 [id=fc5a0b5a96e6400da438307d26f73614]). Expected version 3 got 0
I20251027 21:08:08.106029 9304 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41710 (request call id 1274) took 1054 ms. Trace:
I20251027 21:08:08.106101 9304 rpcz_store.cc:276] 1027 21:08:07.051225 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.687008 (+635783us) service_pool.cc:225] Handling call
1027 21:08:07.697348 (+ 10340us) catalog_manager.cc:2101] Acquired catalog manager lock
1027 21:08:07.697811 (+ 463us) catalog_manager.cc:2147] Created new table and tablet info
1027 21:08:08.105832 (+408021us) catalog_manager.cc:2207] Wrote table and tablets to system table
1027 21:08:08.105925 (+ 93us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1027 21:08:08.106011 (+ 86us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:08:08.059622 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:08:08.059730 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:08:08.059736 (+ 6us) write_op.cc:435] Acquired schema lock
1027 21:08:08.059738 (+ 2us) tablet.cc:660] Decoding operations
1027 21:08:08.059830 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:08:08.059839 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1027 21:08:08.059843 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:08:08.059912 (+ 69us) tablet.cc:699] Row locks acquired
1027 21:08:08.059914 (+ 2us) write_op.cc:260] PREPARE: finished
1027 21:08:08.059949 (+ 35us) write_op.cc:270] Start()
1027 21:08:08.059980 (+ 31us) write_op.cc:276] Timestamp: P: 1761599288059902 usec, L: 0
1027 21:08:08.059984 (+ 4us) op_driver.cc:348] REPLICATION: starting
1027 21:08:08.060150 (+ 166us) log.cc:844] Serialized 433 byte log entry
1027 21:08:08.086320 (+ 26170us) op_driver.cc:464] REPLICATION: finished
1027 21:08:08.087321 (+ 1001us) write_op.cc:301] APPLY: starting
1027 21:08:08.087331 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1027 21:08:08.087363 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
1027 21:08:08.087390 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:08:08.105510 (+ 18120us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:08:08.105546 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1027 21:08:08.105553 (+ 7us) write_op.cc:312] APPLY: finished
1027 21:08:08.105618 (+ 65us) log.cc:844] Serialized 29 byte log entry
1027 21:08:08.105667 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:08.105686 (+ 19us) write_op.cc:454] Released schema lock
1027 21:08:08.105692 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":962,"apply.run_cpu_time_us":492,"apply.run_wall_time_us":18425,"num_ops":2,"prepare.queue_time_us":340190,"replication_time_us":26311}]]}
I20251027 21:08:08.107249 10032 tablet_bootstrap.cc:492] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.107344 10032 ts_tablet_manager.cc:1403] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.000s
I20251027 21:08:08.107874 10032 raft_consensus.cc:359] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.108021 10032 raft_consensus.cc:385] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.108062 10032 raft_consensus.cc:740] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.108225 10032 consensus_queue.cc:260] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.108594 10032 ts_tablet_manager.cc:1434] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:08.108788 10032 tablet_bootstrap.cc:492] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:08:08.110654 9859 consensus_peers.cc:597] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d94c695320f24a7f9776c493e462ed24. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:08.115092 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 98ea3c7f97134166b327f31699ca1251 (DEFAULT_TABLE table=default.Table_4d37173734c0410292144b74c11b4d9f [id=af1f3579b76f4a2d8ec1ce9cbce2c755]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.115126 9751 tablet_service.cc:1505] Processing CreateTablet for tablet 3d3c5ec60b0a4a9ba8d89a63e8a518d3 (DEFAULT_TABLE table=default.Table_2430eb08210e4f16ad19473f4682c43c [id=678ed03dfeb54dd781bd21d154819ca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.115317 9744 tablet_service.cc:1505] Processing CreateTablet for tablet 98ea3c7f97134166b327f31699ca1251 (DEFAULT_TABLE table=default.Table_4d37173734c0410292144b74c11b4d9f [id=af1f3579b76f4a2d8ec1ce9cbce2c755]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.115525 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 3d3c5ec60b0a4a9ba8d89a63e8a518d3 (DEFAULT_TABLE table=default.Table_2430eb08210e4f16ad19473f4682c43c [id=678ed03dfeb54dd781bd21d154819ca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.115788 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 9acdcc8243554868915c2180e2ae16b0 (DEFAULT_TABLE table=default.Table_5b230fb9992d42798e2da304f545376c [id=1bbf758feaef4404afe41d34805f51d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.116202 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 712b111361414043b5a9e9d0e51968ac (DEFAULT_TABLE table=default.Table_9dfdbd16e4f34bc4ba44d300083c774d [id=9603dfb99fd34f959569cf6a9a99e599]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.116504 10997 raft_consensus.cc:493] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.116613 10997 raft_consensus.cc:515] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.117223 10997 leader_election.cc:290] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:08.117863 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 9acdcc8243554868915c2180e2ae16b0 (DEFAULT_TABLE table=default.Table_5b230fb9992d42798e2da304f545376c [id=1bbf758feaef4404afe41d34805f51d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.118212 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "980eb6755aee4bcfb2ea44d84009021f" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:08.118945 11177 raft_consensus.cc:493] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.119066 11177 raft_consensus.cc:515] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:08.119567 11177 leader_election.cc:290] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:08.120056 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83f5fbf5757a49df93ab949b7657c61f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:08.120206 9632 raft_consensus.cc:2468] T 83f5fbf5757a49df93ab949b7657c61f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 0.
I20251027 21:08:08.120554 9860 leader_election.cc:304] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:08.121397 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83f5fbf5757a49df93ab949b7657c61f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:08.121716 11177 raft_consensus.cc:2804] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:08.121827 11177 raft_consensus.cc:493] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:08.121902 11177 raft_consensus.cc:3060] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:08.122371 9859 leader_election.cc:343] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.122800 9920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "980eb6755aee4bcfb2ea44d84009021f" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:08.123260 9711 leader_election.cc:304] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:08.123620 10997 raft_consensus.cc:2749] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:08.126113 11162 raft_consensus.cc:515] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:08.126899 10032 tablet_bootstrap.cc:654] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.127238 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c2d713d72fd4b5ca34cb3e828c75761" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:08.127635 10859 raft_consensus.cc:493] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.127739 10859 raft_consensus.cc:515] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:08.128185 10859 leader_election.cc:290] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.128806 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aabaa34787cf4d179822269b7f323314" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:08.129180 9859 leader_election.cc:343] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:08.129215 9566 leader_election.cc:343] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.129933 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c2d713d72fd4b5ca34cb3e828c75761" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:08.130120 9632 raft_consensus.cc:3060] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:08.130640 11162 leader_election.cc:290] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:08.131685 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2710f265ae9485dafec389168328067" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:08.131925 11300 tablet.cc:1722] T 87d5af4607f04f92abcc7292b4176cbd P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251027 21:08:08.132118 9566 leader_election.cc:343] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.132929 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 712b111361414043b5a9e9d0e51968ac (DEFAULT_TABLE table=default.Table_9dfdbd16e4f34bc4ba44d300083c774d [id=9603dfb99fd34f959569cf6a9a99e599]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.133430 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 712b111361414043b5a9e9d0e51968ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.134672 9903 tablet_service.cc:1505] Processing CreateTablet for tablet 01cfe828d88646438ef96f8a7d08f6db (DEFAULT_TABLE table=default.Table_6bc36020327e4d289973d93a0c1def37 [id=c605be7b6a4e46ab82c0acca63fef426]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.134696 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 01cfe828d88646438ef96f8a7d08f6db (DEFAULT_TABLE table=default.Table_6bc36020327e4d289973d93a0c1def37 [id=c605be7b6a4e46ab82c0acca63fef426]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.134934 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01cfe828d88646438ef96f8a7d08f6db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.135104 9745 tablet_service.cc:1505] Processing CreateTablet for tablet bc387163e2d64532b87c32c5842a285a (DEFAULT_TABLE table=default.Table_f4d6cc291d944761aae3234c8ebdd6f1 [id=08c009a675754bcb9b40230cf2066327]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.137274 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 74289987750b489e9a37563879a25dbc (DEFAULT_TABLE table=default.Table_a611f21a506d4b8badac8f08f6a4ad55 [id=9b7ea3dd3bb04ff492a0c61c455dc61c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.137414 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 74289987750b489e9a37563879a25dbc (DEFAULT_TABLE table=default.Table_a611f21a506d4b8badac8f08f6a4ad55 [id=9b7ea3dd3bb04ff492a0c61c455dc61c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.137785 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74289987750b489e9a37563879a25dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.137928 9894 tablet_service.cc:1505] Processing CreateTablet for tablet bc387163e2d64532b87c32c5842a285a (DEFAULT_TABLE table=default.Table_f4d6cc291d944761aae3234c8ebdd6f1 [id=08c009a675754bcb9b40230cf2066327]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.138223 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc387163e2d64532b87c32c5842a285a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.141194 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 98ea3c7f97134166b327f31699ca1251 (DEFAULT_TABLE table=default.Table_4d37173734c0410292144b74c11b4d9f [id=af1f3579b76f4a2d8ec1ce9cbce2c755]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.141307 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 3d3c5ec60b0a4a9ba8d89a63e8a518d3 (DEFAULT_TABLE table=default.Table_2430eb08210e4f16ad19473f4682c43c [id=678ed03dfeb54dd781bd21d154819ca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.141788 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 9acdcc8243554868915c2180e2ae16b0 (DEFAULT_TABLE table=default.Table_5b230fb9992d42798e2da304f545376c [id=1bbf758feaef4404afe41d34805f51d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.142079 9920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2710f265ae9485dafec389168328067" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:08.142230 9920 raft_consensus.cc:3060] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:08.142346 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 712b111361414043b5a9e9d0e51968ac (DEFAULT_TABLE table=default.Table_9dfdbd16e4f34bc4ba44d300083c774d [id=9603dfb99fd34f959569cf6a9a99e599]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.142927 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 01cfe828d88646438ef96f8a7d08f6db (DEFAULT_TABLE table=default.Table_6bc36020327e4d289973d93a0c1def37 [id=c605be7b6a4e46ab82c0acca63fef426]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.143471 9615 tablet_service.cc:1505] Processing CreateTablet for tablet bc387163e2d64532b87c32c5842a285a (DEFAULT_TABLE table=default.Table_f4d6cc291d944761aae3234c8ebdd6f1 [id=08c009a675754bcb9b40230cf2066327]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.143664 10029 tablet_bootstrap.cc:492] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:08.143784 10029 ts_tablet_manager.cc:1403] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.000s
I20251027 21:08:08.144003 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 74289987750b489e9a37563879a25dbc (DEFAULT_TABLE table=default.Table_a611f21a506d4b8badac8f08f6a4ad55 [id=9b7ea3dd3bb04ff492a0c61c455dc61c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:08.144394 9568 leader_election.cc:343] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: aabaa34787cf4d179822269b7f323314
I20251027 21:08:08.146335 9568 leader_election.cc:304] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.144496 10029 raft_consensus.cc:359] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.145969 10859 raft_consensus.cc:493] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.146682 11142 raft_consensus.cc:2749] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:08.146766 10859 raft_consensus.cc:515] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:08.147257 10859 leader_election.cc:290] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.147656 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a033551a591146039b017b4193b221ca" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:08.147773 9743 tablet_service.cc:1505] Processing CreateTablet for tablet 1dc45f9d5219461b9fca93a7719d3f6c (DEFAULT_TABLE table=default.Table_a6cf2819ed9241cea09738477423c84a [id=f5ef094809e24355a7e021408e2ddd20]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:08.148069 9566 leader_election.cc:343] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.148200 10029 raft_consensus.cc:385] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.148269 10029 raft_consensus.cc:740] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.148473 10029 consensus_queue.cc:260] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.148701 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 1dc45f9d5219461b9fca93a7719d3f6c (DEFAULT_TABLE table=default.Table_a6cf2819ed9241cea09738477423c84a [id=f5ef094809e24355a7e021408e2ddd20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.148974 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dc45f9d5219461b9fca93a7719d3f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.152452 9738 tablet_service.cc:1505] Processing CreateTablet for tablet dbf29208d34b4e6f92f5394b14dcc1cc (DEFAULT_TABLE table=default.Table_1b463a3e59984a129fc2824928138b26 [id=7422045718af4223aadf1927576e0b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.152758 9895 tablet_service.cc:1505] Processing CreateTablet for tablet dbf29208d34b4e6f92f5394b14dcc1cc (DEFAULT_TABLE table=default.Table_1b463a3e59984a129fc2824928138b26 [id=7422045718af4223aadf1927576e0b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.152953 9748 tablet_service.cc:1505] Processing CreateTablet for tablet fcd43479e12e4ababfb4f781aa1827e3 (DEFAULT_TABLE table=default.Table_b81999a26dfe4176a646bd057c455f92 [id=5a7302b9fde24d2c9d289e34e0ef4ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.153034 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbf29208d34b4e6f92f5394b14dcc1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.153271 10029 ts_tablet_manager.cc:1434] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251027 21:08:08.153424 9741 tablet_service.cc:1505] Processing CreateTablet for tablet 079da25c806741ea879625a95cd8781c (DEFAULT_TABLE table=default.Table_56eaa1a50d974ff2bbef789f8054226a [id=45b16830b429484eaa129e08ff31778a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.153508 10029 tablet_bootstrap.cc:492] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:08.153934 9747 tablet_service.cc:1505] Processing CreateTablet for tablet 9cac773211f64b5abdb28ecf19d26d81 (DEFAULT_TABLE table=default.Table_58a7455c41ba48e9bd85661c6f1f91bd [id=e122bed29d30494ea0727000d4d64e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.156117 9891 tablet_service.cc:1505] Processing CreateTablet for tablet fcd43479e12e4ababfb4f781aa1827e3 (DEFAULT_TABLE table=default.Table_b81999a26dfe4176a646bd057c455f92 [id=5a7302b9fde24d2c9d289e34e0ef4ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.156383 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcd43479e12e4ababfb4f781aa1827e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.158900 10029 tablet_bootstrap.cc:654] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.159461 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 079da25c806741ea879625a95cd8781c (DEFAULT_TABLE table=default.Table_56eaa1a50d974ff2bbef789f8054226a [id=45b16830b429484eaa129e08ff31778a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.159718 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 079da25c806741ea879625a95cd8781c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:08.160681 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 58c9cd6828a1497babb9c91bee938041 consensus state has no leader
I20251027 21:08:08.160852 9307 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:41710 (request call id 1276) took 1109 ms. Trace:
I20251027 21:08:08.160918 9307 rpcz_store.cc:276] 1027 21:08:07.051382 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.754410 (+703028us) service_pool.cc:225] Handling call
1027 21:08:07.754877 (+ 467us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1027 21:08:07.755080 (+ 203us) catalog_manager.cc:3708] Apply alter schema
1027 21:08:07.755251 (+ 171us) catalog_manager.cc:3740] Acquired catalog manager lock
1027 21:08:07.755324 (+ 73us) catalog_manager.cc:3898] Updating metadata on disk
1027 21:08:08.160525 (+405201us) catalog_manager.cc:3929] Committing alterations to in-memory state
1027 21:08:08.160836 (+ 311us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:08:08.155041 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:08:08.155127 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:08:08.155132 (+ 5us) write_op.cc:435] Acquired schema lock
1027 21:08:08.155134 (+ 2us) tablet.cc:660] Decoding operations
1027 21:08:08.155244 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:08:08.155255 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1027 21:08:08.155259 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1027 21:08:08.155311 (+ 52us) tablet.cc:699] Row locks acquired
1027 21:08:08.155313 (+ 2us) write_op.cc:260] PREPARE: finished
1027 21:08:08.155340 (+ 27us) write_op.cc:270] Start()
1027 21:08:08.155370 (+ 30us) write_op.cc:276] Timestamp: P: 1761599288155323 usec, L: 0
1027 21:08:08.155373 (+ 3us) op_driver.cc:348] REPLICATION: starting
1027 21:08:08.155627 (+ 254us) log.cc:844] Serialized 405 byte log entry
1027 21:08:08.159028 (+ 3401us) op_driver.cc:464] REPLICATION: finished
1027 21:08:08.160098 (+ 1070us) write_op.cc:301] APPLY: starting
1027 21:08:08.160110 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
1027 21:08:08.160143 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
1027 21:08:08.160166 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:08:08.160207 (+ 41us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:08:08.160233 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:08:08.160239 (+ 6us) write_op.cc:312] APPLY: finished
1027 21:08:08.160315 (+ 76us) log.cc:844] Serialized 23 byte log entry
1027 21:08:08.160373 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:08.160391 (+ 18us) write_op.cc:454] Released schema lock
1027 21:08:08.160397 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1020,"apply.run_cpu_time_us":368,"apply.run_wall_time_us":368,"num_ops":1,"prepare.queue_time_us":399119,"raft.queue_time_us":40,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":271,"replication_time_us":3628}]]}
I20251027 21:08:08.164460 11162 raft_consensus.cc:493] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.164629 11162 raft_consensus.cc:515] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.165486 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9c86253ce914916bee6584675a82067" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:08.165910 9860 leader_election.cc:343] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.166661 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9c86253ce914916bee6584675a82067" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:08.167127 9859 leader_election.cc:343] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.167235 9859 leader_election.cc:304] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.167466 11234 raft_consensus.cc:2749] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:08.167667 11162 leader_election.cc:290] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.167814 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7d0f803e9bb44ef6b7f03e11e0dae838" } modify_external_catalogs: true
I20251027 21:08:08.167958 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7d0f803e9bb44ef6b7f03e11e0dae838" } modify_external_catalogs: true
I20251027 21:08:08.170586 10032 tablet_bootstrap.cc:492] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.170693 10032 ts_tablet_manager.cc:1403] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.005s
I20251027 21:08:08.171262 10032 raft_consensus.cc:359] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.171392 10032 raft_consensus.cc:385] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.171429 10032 raft_consensus.cc:740] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.171612 10032 consensus_queue.cc:260] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.177346 10032 ts_tablet_manager.cc:1434] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20251027 21:08:08.182096 9311 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41710 (request call id 1277) took 1130 ms. Trace:
I20251027 21:08:08.182417 10032 tablet_bootstrap.cc:492] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:08.182204 9311 rpcz_store.cc:276] 1027 21:08:07.051466 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.845992 (+794526us) service_pool.cc:225] Handling call
1027 21:08:07.846851 (+ 859us) catalog_manager.cc:2101] Acquired catalog manager lock
1027 21:08:07.847275 (+ 424us) catalog_manager.cc:2147] Created new table and tablet info
1027 21:08:08.181909 (+334634us) catalog_manager.cc:2207] Wrote table and tablets to system table
1027 21:08:08.182001 (+ 92us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1027 21:08:08.182078 (+ 77us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:08:08.177174 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:08:08.177260 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:08:08.177264 (+ 4us) write_op.cc:435] Acquired schema lock
1027 21:08:08.177266 (+ 2us) tablet.cc:660] Decoding operations
1027 21:08:08.177344 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:08:08.177353 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1027 21:08:08.177358 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:08:08.177478 (+ 120us) tablet.cc:699] Row locks acquired
1027 21:08:08.177480 (+ 2us) write_op.cc:260] PREPARE: finished
1027 21:08:08.177533 (+ 53us) write_op.cc:270] Start()
1027 21:08:08.178123 (+ 590us) write_op.cc:276] Timestamp: P: 1761599288177489 usec, L: 0
1027 21:08:08.178127 (+ 4us) op_driver.cc:348] REPLICATION: starting
1027 21:08:08.178265 (+ 138us) log.cc:844] Serialized 433 byte log entry
1027 21:08:08.180992 (+ 2727us) op_driver.cc:464] REPLICATION: finished
1027 21:08:08.181509 (+ 517us) write_op.cc:301] APPLY: starting
1027 21:08:08.181521 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
1027 21:08:08.181554 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
1027 21:08:08.181575 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:08:08.181657 (+ 82us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:08:08.181682 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1027 21:08:08.181689 (+ 7us) write_op.cc:312] APPLY: finished
1027 21:08:08.181740 (+ 51us) log.cc:844] Serialized 29 byte log entry
1027 21:08:08.181793 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:08.181808 (+ 15us) write_op.cc:454] Released schema lock
1027 21:08:08.181813 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":462,"apply.run_cpu_time_us":362,"apply.run_wall_time_us":362,"num_ops":2,"prepare.queue_time_us":329461,"raft.queue_time_us":30,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":201,"replication_time_us":2842}]]}
I20251027 21:08:08.183816 10032 tablet_bootstrap.cc:654] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.186280 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a47950555ce34e4aabcf33ae10344421" } modify_external_catalogs: true
I20251027 21:08:08.186426 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a47950555ce34e4aabcf33ae10344421" } modify_external_catalogs: true
I20251027 21:08:08.187871 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d3c5ec60b0a4a9ba8d89a63e8a518d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.188436 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98ea3c7f97134166b327f31699ca1251. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:08.190029 9711 consensus_peers.cc:597] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b30052ff19744a9bf4bfaaa1bc77b36. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:08.190485 10029 tablet_bootstrap.cc:492] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:08.190600 10029 ts_tablet_manager.cc:1403] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.003s
I20251027 21:08:08.191215 10029 raft_consensus.cc:359] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.191350 10029 raft_consensus.cc:385] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.191390 10029 raft_consensus.cc:740] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.191586 10032 tablet_bootstrap.cc:492] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.191538 10029 consensus_queue.cc:260] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.191681 10032 ts_tablet_manager.cc:1403] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20251027 21:08:08.192286 10032 raft_consensus.cc:359] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.192436 10032 raft_consensus.cc:385] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.192483 10032 raft_consensus.cc:740] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.192649 10032 consensus_queue.cc:260] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.192829 11283 raft_consensus.cc:515] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:08.193094 10032 ts_tablet_manager.cc:1434] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:08.193328 11283 leader_election.cc:290] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.193468 10032 tablet_bootstrap.cc:492] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:08.193794 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b8156d9ab2e4dfba383ddc72137aab4" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:08.194157 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcd43479e12e4ababfb4f781aa1827e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.194885 10032 tablet_bootstrap.cc:654] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.195719 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 079da25c806741ea879625a95cd8781c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.196159 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74289987750b489e9a37563879a25dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.196517 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cac773211f64b5abdb28ecf19d26d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.196897 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbf29208d34b4e6f92f5394b14dcc1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.197269 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc387163e2d64532b87c32c5842a285a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.197631 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 712b111361414043b5a9e9d0e51968ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.197999 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01cfe828d88646438ef96f8a7d08f6db. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:08.198491 9566 leader_election.cc:343] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.198702 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dc45f9d5219461b9fca93a7719d3f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.199070 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9acdcc8243554868915c2180e2ae16b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.204356 10029 ts_tablet_manager.cc:1434] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251027 21:08:08.208997 10029 tablet_bootstrap.cc:492] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:08.210263 10029 tablet_bootstrap.cc:654] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.216194 9909 raft_consensus.cc:2468] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:08.216593 11283 raft_consensus.cc:493] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.216701 11283 raft_consensus.cc:515] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:08.217206 11283 leader_election.cc:290] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.217707 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9861a0efcba64141aae450663c5d83de" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:08.218174 9566 leader_election.cc:343] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.228003 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98ea3c7f97134166b327f31699ca1251. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.228233 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d3c5ec60b0a4a9ba8d89a63e8a518d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.228402 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74289987750b489e9a37563879a25dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.228786 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9acdcc8243554868915c2180e2ae16b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.229816 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc387163e2d64532b87c32c5842a285a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.230252 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01cfe828d88646438ef96f8a7d08f6db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.230613 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 712b111361414043b5a9e9d0e51968ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.252950 9568 leader_election.cc:304] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.253224 9907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b8156d9ab2e4dfba383ddc72137aab4" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:08.253371 11283 raft_consensus.cc:2804] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:08.253451 9907 raft_consensus.cc:3060] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:08.253484 11283 raft_consensus.cc:697] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:08.253690 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 1dc45f9d5219461b9fca93a7719d3f6c (DEFAULT_TABLE table=default.Table_a6cf2819ed9241cea09738477423c84a [id=f5ef094809e24355a7e021408e2ddd20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.253737 11283 consensus_queue.cc:237] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:08.254272 9911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9861a0efcba64141aae450663c5d83de" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:08.254371 9603 tablet_service.cc:1505] Processing CreateTablet for tablet fcd43479e12e4ababfb4f781aa1827e3 (DEFAULT_TABLE table=default.Table_b81999a26dfe4176a646bd057c455f92 [id=5a7302b9fde24d2c9d289e34e0ef4ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.254969 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 079da25c806741ea879625a95cd8781c (DEFAULT_TABLE table=default.Table_56eaa1a50d974ff2bbef789f8054226a [id=45b16830b429484eaa129e08ff31778a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:08.255412 9568 leader_election.cc:343] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Illegal state: Tablet not RUNNING: STOPPED
I20251027 21:08:08.255488 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 9cac773211f64b5abdb28ecf19d26d81 (DEFAULT_TABLE table=default.Table_58a7455c41ba48e9bd85661c6f1f91bd [id=e122bed29d30494ea0727000d4d64e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.255576 9605 tablet_service.cc:1505] Processing CreateTablet for tablet dbf29208d34b4e6f92f5394b14dcc1cc (DEFAULT_TABLE table=default.Table_1b463a3e59984a129fc2824928138b26 [id=7422045718af4223aadf1927576e0b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.255524 9568 leader_election.cc:304] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.256067 11097 raft_consensus.cc:2749] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:08.256873 9568 leader_election.cc:343] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: a033551a591146039b017b4193b221ca
I20251027 21:08:08.256978 9568 leader_election.cc:304] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.257341 11283 raft_consensus.cc:2749] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:08.269510 10031 ts_tablet_manager.cc:1434] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.205s user 0.002s sys 0.000s
I20251027 21:08:08.270915 11162 raft_consensus.cc:493] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.271046 11162 raft_consensus.cc:515] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.271504 10031 tablet_bootstrap.cc:492] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:08.271518 11162 leader_election.cc:290] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.272522 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "520f7142f8af4599ac4880b0692e3151" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:08.273000 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "520f7142f8af4599ac4880b0692e3151" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:08.273759 10031 tablet_bootstrap.cc:654] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:08.274983 9860 leader_election.cc:343] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:08.275374 9859 leader_election.cc:343] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.275471 9859 leader_election.cc:304] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.275847 11162 raft_consensus.cc:2749] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:08.282331 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9f371b16693a415896291c6a3edf3d25" } modify_external_catalogs: true
I20251027 21:08:08.282493 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9f371b16693a415896291c6a3edf3d25" } modify_external_catalogs: true
I20251027 21:08:08.312952 11162 raft_consensus.cc:493] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.313133 11162 raft_consensus.cc:515] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:08.313880 11162 leader_election.cc:290] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:08.314549 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fa39697360c46138ef4cd21e8c7a950" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:08.314760 9635 raft_consensus.cc:2468] T 9fa39697360c46138ef4cd21e8c7a950 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 0.
I20251027 21:08:08.315462 9860 leader_election.cc:304] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:08.315814 11162 raft_consensus.cc:2804] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:08.315900 11162 raft_consensus.cc:493] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:08.315977 11162 raft_consensus.cc:3060] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:08.317066 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fa39697360c46138ef4cd21e8c7a950" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:08.317765 9711 consensus_peers.cc:597] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3b9a461567a48fcb4af514ecdf618b4. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:08.318051 9859 leader_election.cc:343] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.323318 9632 raft_consensus.cc:2468] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 1.
I20251027 21:08:08.323802 11177 raft_consensus.cc:515] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:08.324421 11177 leader_election.cc:290] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:08.328652 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83f5fbf5757a49df93ab949b7657c61f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:08.329249 9859 leader_election.cc:343] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:08.340728 9298 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet bd6152f7ac8a4123872098955b8837ac (table default.Table_352763845d1f4fb2a97474f8c82883b7 [id=1e0d20c5adfa4c52845b9ea7cf5d8037], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bd6152f7ac8a4123872098955b8837ac consensus state has no leader
I20251027 21:08:08.341648 10029 tablet_bootstrap.cc:492] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:08.341797 10029 ts_tablet_manager.cc:1403] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.133s user 0.004s sys 0.020s
I20251027 21:08:08.342563 10029 raft_consensus.cc:359] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.342713 10029 raft_consensus.cc:385] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.342770 10029 raft_consensus.cc:740] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.342948 10029 consensus_queue.cc:260] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.343456 10029 ts_tablet_manager.cc:1434] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:08.343703 10029 tablet_bootstrap.cc:492] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:08.347347 10029 tablet_bootstrap.cc:654] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.352360 9860 leader_election.cc:304] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.353091 11234 raft_consensus.cc:2804] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:08.353195 11234 raft_consensus.cc:697] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Becoming Leader. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Running, Role: LEADER
I20251027 21:08:08.353408 11234 consensus_queue.cc:237] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:08.354641 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83f5fbf5757a49df93ab949b7657c61f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:08.354799 9635 raft_consensus.cc:3060] T 83f5fbf5757a49df93ab949b7657c61f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:08.356271 10997 consensus_queue.cc:1048] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:08.362790 9894 tablet_service.cc:1505] Processing CreateTablet for tablet 9cac773211f64b5abdb28ecf19d26d81 (DEFAULT_TABLE table=default.Table_58a7455c41ba48e9bd85661c6f1f91bd [id=e122bed29d30494ea0727000d4d64e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:08.377985 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9acdcc8243554868915c2180e2ae16b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.378453 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d3c5ec60b0a4a9ba8d89a63e8a518d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.378808 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98ea3c7f97134166b327f31699ca1251. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.378837 10032 tablet_bootstrap.cc:492] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.378983 10032 ts_tablet_manager.cc:1403] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.186s user 0.062s sys 0.083s
I20251027 21:08:08.379483 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cac773211f64b5abdb28ecf19d26d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.379863 10032 raft_consensus.cc:359] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.380043 10032 raft_consensus.cc:385] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.380100 10032 raft_consensus.cc:740] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.380328 10032 consensus_queue.cc:260] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.386790 11314 raft_consensus.cc:493] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.387446 11314 raft_consensus.cc:515] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.388056 10032 ts_tablet_manager.cc:1434] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.009s user 0.002s sys 0.002s
I20251027 21:08:08.387262 11313 raft_consensus.cc:493] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.388248 11313 raft_consensus.cc:515] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.388622 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d18ff6d8ddb465fbf479fdc7ebe7a05" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:08.388854 11313 leader_election.cc:290] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.389032 11314 leader_election.cc:290] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.388332 10032 tablet_bootstrap.cc:492] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:08.389384 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22dbfa50996b47269983c107cc399000" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:08.389782 9860 leader_election.cc:343] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:08.390189 9860 leader_election.cc:343] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.390679 10032 tablet_bootstrap.cc:654] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.393535 10041 ts_tablet_manager.cc:1929] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:08.394057 9635 raft_consensus.cc:2468] T 83f5fbf5757a49df93ab949b7657c61f P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 1.
W20251027 21:08:08.403842 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dbf46e4ce7714398b2c2019314e44578 consensus state has no leader
I20251027 21:08:08.403911 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dc45f9d5219461b9fca93a7719d3f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.404197 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c31d7158e88c44b18707ae0ad36fe482" } modify_external_catalogs: true
I20251027 21:08:08.404350 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c31d7158e88c44b18707ae0ad36fe482" } modify_external_catalogs: true
I20251027 21:08:08.404414 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcd43479e12e4ababfb4f781aa1827e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.405556 10043 ts_tablet_manager.cc:1929] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:08.405642 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbf29208d34b4e6f92f5394b14dcc1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.407054 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 079da25c806741ea879625a95cd8781c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.407546 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cac773211f64b5abdb28ecf19d26d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:08.405653 10043 log.cc:1199] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/da26c83976df40328cf9b11734fcfa44
I20251027 21:08:08.437417 10043 ts_tablet_manager.cc:1950] T da26c83976df40328cf9b11734fcfa44 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:08.441119 10031 tablet_bootstrap.cc:492] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:08.441344 10031 ts_tablet_manager.cc:1403] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.170s user 0.001s sys 0.002s
I20251027 21:08:08.442716 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f46fff92cdc244f6a020795467099e74" } modify_external_catalogs: true
I20251027 21:08:08.442888 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f46fff92cdc244f6a020795467099e74" } modify_external_catalogs: true
I20251027 21:08:08.394255 9920 raft_consensus.cc:2468] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:08.400321 11314 raft_consensus.cc:493] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.445011 11314 raft_consensus.cc:515] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.445786 11314 leader_election.cc:290] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:08.423671 9859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 68221.3us
I20251027 21:08:08.447499 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d18ff6d8ddb465fbf479fdc7ebe7a05" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:08.447573 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22dbfa50996b47269983c107cc399000" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:08.431330 9860 leader_election.cc:304] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.448664 11177 raft_consensus.cc:2804] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:08.448800 11177 raft_consensus.cc:697] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Becoming Leader. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Running, Role: LEADER
I20251027 21:08:08.449154 11177 consensus_queue.cc:237] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:08.449265 11313 consensus_queue.cc:1048] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:08.452814 11314 consensus_queue.cc:1048] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:08.453588 10029 tablet_bootstrap.cc:492] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:08.453734 10029 ts_tablet_manager.cc:1403] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.110s user 0.003s sys 0.001s
I20251027 21:08:08.454541 10029 raft_consensus.cc:359] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.454684 10029 raft_consensus.cc:385] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.454730 10029 raft_consensus.cc:740] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.454777 10032 tablet_bootstrap.cc:492] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.454886 10032 ts_tablet_manager.cc:1403] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.067s user 0.022s sys 0.026s
I20251027 21:08:08.454926 10029 consensus_queue.cc:260] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.455348 10029 ts_tablet_manager.cc:1434] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:08.455616 10029 tablet_bootstrap.cc:492] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:08.455680 10032 raft_consensus.cc:359] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.455839 10032 raft_consensus.cc:385] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.455938 10032 raft_consensus.cc:740] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.456144 10032 consensus_queue.cc:260] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.456676 10032 ts_tablet_manager.cc:1434] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:08.456969 10032 tablet_bootstrap.cc:492] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:08.459228 10032 tablet_bootstrap.cc:654] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.459821 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 750b27c74ba3473dad743d60c21912c5
I20251027 21:08:08.460345 9310 catalog_manager.cc:6719] ReplaceTablet: tablet 750b27c74ba3473dad743d60c21912c5 deleted and replaced by tablet 8d61afc9ce6b4b80b5e64259b9c04f85
I20251027 21:08:08.460443 9310 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:41710 (request call id 1279) took 1113 ms. Trace:
I20251027 21:08:08.460515 9310 rpcz_store.cc:276] 1027 21:08:07.347052 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.962395 (+615343us) service_pool.cc:225] Handling call
1027 21:08:08.053222 (+ 90827us) spinlock_profiling.cc:243] Waited 84.5 ms on lock 0x616000051154. stack: 00007f956e0a89f6 00007f956e0a87b0 00007f956d0843ad 0000000000983f04 0000000000983ecd 00007f957d3af6bd 00007f957d3af7b3 00007f9579f76441 00007f9581c309dd 00007f9581dc9fab 00007f95788da107 00007f95788d9e35 00007f957504727e 00007f957504615e 00007f957504c395 00007f957504ccbf
1027 21:08:08.460424 (+407202us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:08:08.444264 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:08:08.444387 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:08:08.444392 (+ 5us) write_op.cc:435] Acquired schema lock
1027 21:08:08.444405 (+ 13us) tablet.cc:660] Decoding operations
1027 21:08:08.444505 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:08:08.444516 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1027 21:08:08.444520 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:08:08.444590 (+ 70us) tablet.cc:699] Row locks acquired
1027 21:08:08.444592 (+ 2us) write_op.cc:260] PREPARE: finished
1027 21:08:08.444625 (+ 33us) write_op.cc:270] Start()
1027 21:08:08.444667 (+ 42us) write_op.cc:276] Timestamp: P: 1761599288444595 usec, L: 0
1027 21:08:08.444670 (+ 3us) op_driver.cc:348] REPLICATION: starting
1027 21:08:08.444861 (+ 191us) log.cc:844] Serialized 622 byte log entry
1027 21:08:08.458858 (+ 13997us) op_driver.cc:464] REPLICATION: finished
1027 21:08:08.459051 (+ 193us) write_op.cc:301] APPLY: starting
1027 21:08:08.459067 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1027 21:08:08.459113 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
1027 21:08:08.459140 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:08:08.459244 (+ 104us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:08:08.459309 (+ 65us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:08:08.459316 (+ 7us) write_op.cc:312] APPLY: finished
1027 21:08:08.459384 (+ 68us) log.cc:844] Serialized 29 byte log entry
1027 21:08:08.459453 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:08.459477 (+ 24us) write_op.cc:454] Released schema lock
1027 21:08:08.459483 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":194344320,"child_traces":[["op",{"apply.queue_time_us":136,"apply.run_cpu_time_us":495,"apply.run_wall_time_us":494,"num_ops":2,"prepare.queue_time_us":347087,"raft.queue_time_us":11302,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":304,"replication_time_us":14150}]]}
I20251027 21:08:08.461694 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 750b27c74ba3473dad743d60c21912c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d61afc9ce6b4b80b5e64259b9c04f85) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:08.463374 9741 tablet_service.cc:1552] Processing DeleteTablet for tablet 750b27c74ba3473dad743d60c21912c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d61afc9ce6b4b80b5e64259b9c04f85) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:08.464484 10029 tablet_bootstrap.cc:654] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.466666 10041 log.cc:1199] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/9861a0efcba64141aae450663c5d83de
I20251027 21:08:08.467068 10041 ts_tablet_manager.cc:1950] T 9861a0efcba64141aae450663c5d83de P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
W20251027 21:08:08.467482 9859 leader_election.cc:343] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.470224 9859 leader_election.cc:304] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.470570 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c8cb97c3834e406ca7d8a2d047d7d249" } new_table_name: "default.Table_64cbe911866949e285d945765603b0c1" modify_external_catalogs: true
I20251027 21:08:08.470870 11234 raft_consensus.cc:2749] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:08.471751 9859 leader_election.cc:343] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.472532 9859 leader_election.cc:304] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:08.473186 9859 consensus_peers.cc:597] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:08.473388 11234 raft_consensus.cc:2749] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:08.476190 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 423047dfbf084bfcb881efeb0352f88d consensus state has no leader
I20251027 21:08:08.481559 10031 raft_consensus.cc:359] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.481747 10031 raft_consensus.cc:385] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.481801 10031 raft_consensus.cc:740] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.488471 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 948) took 3649 ms. Trace:
I20251027 21:08:08.488629 10043 rpcz_store.cc:276] 1027 21:08:04.839454 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.840146 (+ 692us) service_pool.cc:225] Handling call
1027 21:08:08.011579 (+3171433us) ts_tablet_manager.cc:1151] Deleting tablet da26c83976df40328cf9b11734fcfa44
1027 21:08:08.011588 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.140234 (+128646us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:08.405434 (+265200us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:08.488451 (+ 83017us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32,"tablet-delete.queue_time_us":3171301}
I20251027 21:08:08.490224 10043 tablet_replica.cc:333] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:08.490556 10043 raft_consensus.cc:2243] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:08.490785 10043 pending_rounds.cc:70] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14: Trying to abort 1 pending ops.
I20251027 21:08:08.490886 10043 pending_rounds.cc:77] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510680332038144 op_type: NO_OP noop_request { }
I20251027 21:08:08.491055 10043 raft_consensus.cc:2889] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:08.491211 10043 raft_consensus.cc:2272] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:08.492053 10043 ts_tablet_manager.cc:1916] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:08.493391 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_10ff6b8aba2c40b2b4edd6466409d1f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:08.493932 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10ff6b8aba2c40b2b4edd6466409d1f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:08.493918 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet da26c83976df40328cf9b11734fcfa44 (table default.Table_ac66ea1435fe49518c073dfc9b2368c8 [id=985d97a0b33f4ef79a9b0df362a75c32]) successfully deleted
I20251027 21:08:08.498107 10040 ts_tablet_manager.cc:1929] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:08.498237 10040 log.cc:1199] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/6f1d1bd52556463fa306e51eb253af40
I20251027 21:08:08.500131 9907 raft_consensus.cc:2468] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:08.508155 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 926) took 4352 ms. Trace:
I20251027 21:08:08.508355 10041 rpcz_store.cc:276] 1027 21:08:04.155668 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.155896 (+ 228us) service_pool.cc:225] Handling call
1027 21:08:08.040378 (+3884482us) ts_tablet_manager.cc:1151] Deleting tablet 9861a0efcba64141aae450663c5d83de
1027 21:08:08.040388 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.377361 (+336973us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:08.393455 (+ 16094us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:08.508083 (+114628us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3884307}
I20251027 21:08:08.508863 10041 tablet_replica.cc:333] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:08.509123 10041 raft_consensus.cc:2243] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:08.509563 10041 pending_rounds.cc:70] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678: Trying to abort 1 pending ops.
I20251027 21:08:08.510058 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 9861a0efcba64141aae450663c5d83de (table default.Table_a8906ade8f93405bb76d1ca9b7a0afa6 [id=a0c9cfaafd6c46dfb9a999f25f8977fa]) successfully deleted
I20251027 21:08:08.509792 10041 pending_rounds.cc:77] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510685098246144 op_type: NO_OP noop_request { }
I20251027 21:08:08.512548 10041 raft_consensus.cc:2889] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:08.512665 10041 raft_consensus.cc:2272] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:08.513502 10041 ts_tablet_manager.cc:1916] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:08.514228 9568 leader_election.cc:304] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.514592 11283 raft_consensus.cc:2804] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:08.514726 11283 raft_consensus.cc:697] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:08.515022 11283 consensus_queue.cc:237] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:08.522635 9568 leader_election.cc:304] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.523041 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72c1dc3b9eff47ee81df821deb789307" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:08.525204 9860 leader_election.cc:343] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.526211 11097 raft_consensus.cc:2804] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:08.526397 11097 raft_consensus.cc:697] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:08.526708 11097 consensus_queue.cc:237] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:08.531939 11142 raft_consensus.cc:493] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.532090 11142 raft_consensus.cc:515] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.532781 11142 leader_election.cc:290] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.536049 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet a21a6a3725934266bbfdfc70b40de687
I20251027 21:08:08.536413 10032 tablet_bootstrap.cc:492] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.536626 10032 ts_tablet_manager.cc:1403] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.080s user 0.008s sys 0.000s
I20251027 21:08:08.536891 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet 520f7142f8af4599ac4880b0692e3151 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:08.537345 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e53b2c28c75b43bdafa0d20f4ad61fcc
I20251027 21:08:08.537550 10032 raft_consensus.cc:359] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.537796 10032 raft_consensus.cc:385] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.537875 10032 raft_consensus.cc:740] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.538149 10032 consensus_queue.cc:260] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.538372 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 8827e4746ecc408a8fea98b6b5ddd57b
I20251027 21:08:08.539672 10032 ts_tablet_manager.cc:1434] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:08:08.540025 10032 tablet_bootstrap.cc:492] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:08.540225 9903 tablet_service.cc:1552] Processing DeleteTablet for tablet a21a6a3725934266bbfdfc70b40de687 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:08.540484 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet e53b2c28c75b43bdafa0d20f4ad61fcc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:08.540668 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet 8827e4746ecc408a8fea98b6b5ddd57b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:08.556550 10032 tablet_bootstrap.cc:654] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.562259 11234 raft_consensus.cc:493] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.562429 11234 raft_consensus.cc:515] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.563544 11234 leader_election.cc:290] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.568311 10032 tablet_bootstrap.cc:492] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.568485 10032 ts_tablet_manager.cc:1403] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.004s
I20251027 21:08:08.569437 10032 raft_consensus.cc:359] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.569675 10032 raft_consensus.cc:385] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.569792 10032 raft_consensus.cc:740] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.570044 10032 consensus_queue.cc:260] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.570713 10032 ts_tablet_manager.cc:1434] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:08.571036 10032 tablet_bootstrap.cc:492] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:08.573518 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "334c580862bd4bf09be3f1710aa6a092" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:08.574035 10032 tablet_bootstrap.cc:654] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:08.574105 9860 leader_election.cc:343] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:08.574702 9568 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 52102.3us
I20251027 21:08:08.575985 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet 750b27c74ba3473dad743d60c21912c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d61afc9ce6b4b80b5e64259b9c04f85) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:08.576562 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet a21a6a3725934266bbfdfc70b40de687 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:08.576936 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet e53b2c28c75b43bdafa0d20f4ad61fcc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:08.577248 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet 8827e4746ecc408a8fea98b6b5ddd57b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:08.577924 9907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2d26784bbb24ae3893103802c41affb" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:08.578177 9907 raft_consensus.cc:2468] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:08.579094 9568 leader_election.cc:304] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:08.579377 11142 raft_consensus.cc:2804] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:08.579481 11142 raft_consensus.cc:493] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:08.579553 11142 raft_consensus.cc:3060] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:08.587613 10029 tablet_bootstrap.cc:492] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:08.587787 10029 ts_tablet_manager.cc:1403] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.132s user 0.004s sys 0.027s
I20251027 21:08:08.588675 10029 raft_consensus.cc:359] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.588892 10029 raft_consensus.cc:385] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.588963 10029 raft_consensus.cc:740] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.589160 10029 consensus_queue.cc:260] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.591166 10029 ts_tablet_manager.cc:1434] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:08:08.591436 10029 tablet_bootstrap.cc:492] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:08.594170 10029 tablet_bootstrap.cc:654] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.594084 11162 raft_consensus.cc:515] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:08.594800 11162 leader_election.cc:290] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Requested vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:08.595415 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_27e2b6a6c7684a7baefcca99d2f93e87" } new_table_name: "default.Table_6aa99aba6b874d73a7fd8f904960790d" modify_external_catalogs: true
I20251027 21:08:08.596983 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c073b4449e4f49a1acb074b3130e3e3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:08.597489 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c073b4449e4f49a1acb074b3130e3e3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:08.599804 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet cc97e55cfbcb41ec8c6c7410bbf14fb3
I20251027 21:08:08.600026 10029 tablet_bootstrap.cc:492] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:08.600129 10029 ts_tablet_manager.cc:1403] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20251027 21:08:08.601058 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet cc97e55cfbcb41ec8c6c7410bbf14fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:08.601728 9304 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 22dbfa50996b47269983c107cc399000 (table default.Table_cb55ac1fcbad4075b84ec473bf89aa2b [id=be549850a5104d288311e062e1362583]). Expected version 1 got 0
I20251027 21:08:08.601840 10029 raft_consensus.cc:359] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.601950 9304 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 8cccd253ae2a46328dc9f767b8e9ee97 (table default.Table_bb9c3894cb5d40738ca8be284b1d1e25 [id=501b230a824d4a17bb1903c8409407b7]). Expected version 1 got 0
I20251027 21:08:08.602015 10029 raft_consensus.cc:385] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.602061 10029 raft_consensus.cc:740] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.602260 10029 consensus_queue.cc:260] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.602176 9304 catalog_manager.cc:5649] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 reported cstate change: term changed from 1 to 2, leader changed from 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65) to 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67). New cstate: current_term: 2 leader_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:08.602763 9304 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet aaa8910d9fdb4146945621ec4ebf969e (table default.Table_d8da07f0725347338012d992737c5d04 [id=bb676562b97f48088767941239cfbf33]). Expected version 2 got 0
I20251027 21:08:08.602955 9304 catalog_manager.cc:5649] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67). New cstate: current_term: 1 leader_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:08.603248 9304 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 29693cf06bc24930870dc60457126519 (table default.Table_c09cc86fc2a64450af56fc584302da69 [id=d46947a29ddf4d94b5a9da22d9e5383e]). Expected version 3 got 0
I20251027 21:08:08.603397 9304 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 980eb6755aee4bcfb2ea44d84009021f (table default.Table_bb56319b6fe0405991a4d171477621e7 [id=42c8fedcabe0435dbaa11427141228ed]). Expected version 1 got 0
I20251027 21:08:08.603523 9304 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 520f7142f8af4599ac4880b0692e3151 (table default.Table_e6955caf15904f3bae744add8221897f [id=bd9615f2023e45688ae88cd7d323e61b]). Expected version 1 got 0
I20251027 21:08:08.604007 11142 raft_consensus.cc:515] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.604183 9308 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_d7530773668f42c5a07a921df02a95ab [id=914b728e358e4fdba6d38d6c7beadaca] alter complete (version 1)
W20251027 21:08:08.604379 9308 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.9.14.66:44229 (request call id 184) took 944 ms (client timeout 1000 ms). Trace:
I20251027 21:08:08.604657 11142 leader_election.cc:290] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:08.604447 9308 rpcz_store.cc:269] 1027 21:08:07.659922 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.660107 (+ 185us) service_pool.cc:225] Handling call
1027 21:08:07.806754 (+146647us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x616000051154. stack: 00007f956e0a89f6 00007f956e0a87b0 00007f956d0843ad 0000000000983f04 0000000000983ecd 00007f957d3af6bd 00007f957d3af7b3 00007f9579f76441 00007f9581c309dd 00007f9581db7aa8 00007f95788caca6 00007f95788ca9d5 00007f957504727e 00007f957504615e 00007f957504c395 00007f957504ccbf
1027 21:08:08.604362 (+797608us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:08:08.170213 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:08:08.170293 (+ 80us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:08:08.170298 (+ 5us) write_op.cc:435] Acquired schema lock
1027 21:08:08.170300 (+ 2us) tablet.cc:660] Decoding operations
1027 21:08:08.170376 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:08:08.170388 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1027 21:08:08.170392 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1027 21:08:08.170438 (+ 46us) tablet.cc:699] Row locks acquired
1027 21:08:08.170440 (+ 2us) write_op.cc:260] PREPARE: finished
1027 21:08:08.170465 (+ 25us) write_op.cc:270] Start()
1027 21:08:08.170488 (+ 23us) write_op.cc:276] Timestamp: P: 1761599288170448 usec, L: 0
1027 21:08:08.170492 (+ 4us) op_driver.cc:348] REPLICATION: starting
1027 21:08:08.170697 (+ 205us) log.cc:844] Serialized 356 byte log entry
1027 21:08:08.173350 (+ 2653us) op_driver.cc:464] REPLICATION: finished
1027 21:08:08.173870 (+ 520us) write_op.cc:301] APPLY: starting
1027 21:08:08.173883 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
1027 21:08:08.173917 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
1027 21:08:08.173945 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:08:08.173989 (+ 44us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:08:08.174016 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:08:08.174024 (+ 8us) write_op.cc:312] APPLY: finished
1027 21:08:08.174080 (+ 56us) log.cc:844] Serialized 23 byte log entry
1027 21:08:08.174130 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:08.174165 (+ 35us) write_op.cc:454] Released schema lock
1027 21:08:08.174170 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1027 21:08:08.177102 (+ 2932us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:08.177107 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":336210304,"child_traces":[["op",{"apply.queue_time_us":466,"apply.run_cpu_time_us":363,"apply.run_wall_time_us":363,"num_ops":1,"prepare.queue_time_us":334966,"prepare.run_cpu_time_us":563,"prepare.run_wall_time_us":6885,"raft.queue_time_us":60,"raft.run_cpu_time_us":384,"raft.run_wall_time_us":385,"replication_time_us":2833}]]}
I20251027 21:08:08.605544 10029 ts_tablet_manager.cc:1434] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251027 21:08:08.605796 10029 tablet_bootstrap.cc:492] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:08.606693 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fa39697360c46138ef4cd21e8c7a950" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:08.606843 9632 raft_consensus.cc:3060] T 9fa39697360c46138ef4cd21e8c7a950 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:08.606993 9312 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.9.14.65:32849 (request call id 182) took 812 ms (client timeout 1000 ms). Trace:
W20251027 21:08:08.607091 9312 rpcz_store.cc:269] 1027 21:08:07.794635 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.822945 (+ 28310us) service_pool.cc:225] Handling call
1027 21:08:08.606975 (+784030us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:08:08.494120 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:08:08.494248 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:08:08.494254 (+ 6us) write_op.cc:435] Acquired schema lock
1027 21:08:08.494259 (+ 5us) tablet.cc:660] Decoding operations
1027 21:08:08.494348 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:08:08.494360 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1027 21:08:08.494367 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
1027 21:08:08.494422 (+ 55us) tablet.cc:699] Row locks acquired
1027 21:08:08.494426 (+ 4us) write_op.cc:260] PREPARE: finished
1027 21:08:08.494456 (+ 30us) write_op.cc:270] Start()
1027 21:08:08.494486 (+ 30us) write_op.cc:276] Timestamp: P: 1761599288494425 usec, L: 0
1027 21:08:08.494491 (+ 5us) op_driver.cc:348] REPLICATION: starting
1027 21:08:08.494673 (+ 182us) log.cc:844] Serialized 527 byte log entry
1027 21:08:08.519328 (+ 24655us) op_driver.cc:464] REPLICATION: finished
1027 21:08:08.519446 (+ 118us) write_op.cc:301] APPLY: starting
1027 21:08:08.519461 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1027 21:08:08.519496 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
1027 21:08:08.519541 (+ 45us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:08:08.519589 (+ 48us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:08:08.519619 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:08:08.519625 (+ 6us) write_op.cc:312] APPLY: finished
1027 21:08:08.519697 (+ 72us) log.cc:844] Serialized 23 byte log entry
1027 21:08:08.519770 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:08.519811 (+ 41us) write_op.cc:454] Released schema lock
1027 21:08:08.519817 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1027 21:08:08.519887 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:08.519893 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":49,"apply.run_cpu_time_us":437,"apply.run_wall_time_us":436,"num_ops":1,"prepare.queue_time_us":388710,"prepare.run_cpu_time_us":678,"prepare.run_wall_time_us":1122,"raft.queue_time_us":66,"raft.run_cpu_time_us":346,"raft.run_wall_time_us":345,"replication_time_us":24793}]]}
I20251027 21:08:08.608932 10029 tablet_bootstrap.cc:654] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.610508 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 94f2819dd0394530ae93754bdd1e482f
I20251027 21:08:08.613005 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 94f2819dd0394530ae93754bdd1e482f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:08.615095 9907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2d26784bbb24ae3893103802c41affb" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:08.615248 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet cc97e55cfbcb41ec8c6c7410bbf14fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:08.615305 9907 raft_consensus.cc:3060] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:08.615502 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet a4d26f73b8534f08bcaa386d4de75c01 with delete_type TABLET_DATA_DELETED (replaced by tablet db2735604dc14039810df4ecd330c671) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:08.615599 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 9861a0efcba64141aae450663c5d83de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:08.615846 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet a033551a591146039b017b4193b221ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:08.616111 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet aabaa34787cf4d179822269b7f323314 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:08.616350 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet fde0980aa64c4496861a662f7ed3ef75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:08.616600 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 94f2819dd0394530ae93754bdd1e482f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:08.618278 9611 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c34c8f1433b40d491ae05e7c6e60ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:08.618494 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 96a680e8f16343b1908829a67e960b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:08.618683 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet 631ff72613734913818a3cad24f20dbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:08.620589 10032 tablet_bootstrap.cc:492] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.620882 10032 ts_tablet_manager.cc:1403] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.050s user 0.006s sys 0.000s
I20251027 21:08:08.621578 11142 consensus_queue.cc:1048] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:08.621985 10032 raft_consensus.cc:359] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.622253 10032 raft_consensus.cc:385] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.622352 10032 raft_consensus.cc:740] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.625936 10032 consensus_queue.cc:260] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.626468 10032 ts_tablet_manager.cc:1434] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251027 21:08:08.451195 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72c1dc3b9eff47ee81df821deb789307" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:08.630373 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_fd9d066e8f9c42ee9c82bb11275d1b5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:08.481987 10031 consensus_queue.cc:260] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:08.630971 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd9d066e8f9c42ee9c82bb11275d1b5c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:08.631268 10032 tablet_bootstrap.cc:492] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:08.632117 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5a743fc873a6414cab6c548e0041134c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:08.632537 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a743fc873a6414cab6c548e0041134c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:08.633625 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_794b0b7aec3346f2bbcb4ad84958e797"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:08.634039 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_794b0b7aec3346f2bbcb4ad84958e797 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:08.634186 10032 tablet_bootstrap.cc:654] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.635965 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_97d7c5edb9a54c508aadda3dcd8c1a99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:08.636514 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97d7c5edb9a54c508aadda3dcd8c1a99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:08.644644 10032 tablet_bootstrap.cc:492] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.644765 10032 ts_tablet_manager.cc:1403] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
W20251027 21:08:08.648262 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3654aa289154b3ea613b6ff94cbd57e consensus state has no leader
I20251027 21:08:08.648597 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e74563f5a1ec4fbc9ddca17283f4bec2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:08.649058 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e74563f5a1ec4fbc9ddca17283f4bec2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:08.652098 10032 raft_consensus.cc:359] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.652303 10032 raft_consensus.cc:385] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.652405 10032 raft_consensus.cc:740] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.652642 10032 consensus_queue.cc:260] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.653177 10032 ts_tablet_manager.cc:1434] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251027 21:08:08.653429 10032 tablet_bootstrap.cc:492] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:08:08.655187 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cbd77880ad934843b659d7b669eaafc7 consensus state has no leader
I20251027 21:08:08.655405 9306 master_service.cc:954] ReplaceTablet: received request to replace tablet 5c34c8f1433b40d491ae05e7c6e60ba1 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:08.659094 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1595883e735448e2b56b64cccacc7219" } new_table_name: "default.Table_f3522d955cb8407ea625509254346d42" modify_external_catalogs: true
I20251027 21:08:08.660283 11177 raft_consensus.cc:493] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.660427 11177 raft_consensus.cc:515] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:08.660682 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_19886f4eae9949fdb78d539336995823"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:08.661015 11177 leader_election.cc:290] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:08.661134 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19886f4eae9949fdb78d539336995823 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:08.665195 10032 tablet_bootstrap.cc:654] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.667640 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2203ee62f7e404c95c5f86b5ae85feb" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:08.669818 9860 leader_election.cc:343] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.675019 11319 raft_consensus.cc:493] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.675158 11319 raft_consensus.cc:515] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.675804 11319 leader_election.cc:290] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:08.680271 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2d26784bbb24ae3893103802c41affb" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:08.680238 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_ab5de3d4fb7b41bf84ab54cbc5effb74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:08.680752 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab5de3d4fb7b41bf84ab54cbc5effb74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:08.681159 10040 ts_tablet_manager.cc:1950] T 6f1d1bd52556463fa306e51eb253af40 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:08.682381 9741 tablet_service.cc:1552] Processing DeleteTablet for tablet a21a6a3725934266bbfdfc70b40de687 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:08:08.683836 9566 leader_election.cc:343] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:08.684197 9566 consensus_peers.cc:597] T e6b58b1f4f63461aa96e416cae879be8 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:08.684345 9566 consensus_peers.cc:597] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:08.684538 9747 tablet_service.cc:1552] Processing DeleteTablet for tablet 8827e4746ecc408a8fea98b6b5ddd57b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:08:08.685307 9859 leader_election.cc:343] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.685451 9859 leader_election.cc:304] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:08.685892 9859 consensus_peers.cc:597] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:08.686058 9747 tablet_service.cc:1552] Processing DeleteTablet for tablet cc97e55cfbcb41ec8c6c7410bbf14fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:08.686237 9741 tablet_service.cc:1552] Processing DeleteTablet for tablet 94f2819dd0394530ae93754bdd1e482f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:08.686210 11297 tablet.cc:1722] T d56d323ef0e74344a479940b93fea11a P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:08.687160 11296 tablet.cc:1722] T d56d323ef0e74344a479940b93fea11a P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:08.688133 11177 raft_consensus.cc:2749] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:08.692953 10043 ts_tablet_manager.cc:1929] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:08.693079 10043 log.cc:1199] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/8b30052ff19744a9bf4bfaaa1bc77b36
I20251027 21:08:08.693178 9888 tablet_service.cc:1552] Processing DeleteTablet for tablet d2203ee62f7e404c95c5f86b5ae85feb with delete_type TABLET_DATA_DELETED (replaced by tablet 00bfb0bdf2ca4b288eac0c1dbee5765a) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:08.693228 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet e6ed9d7a700446edafcd0daa02aa7ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:08.693439 10043 ts_tablet_manager.cc:1950] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:08.693748 10041 ts_tablet_manager.cc:1929] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:08.693866 10041 log.cc:1199] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/9c2d713d72fd4b5ca34cb3e828c75761
I20251027 21:08:08.694285 9632 raft_consensus.cc:2468] T 9fa39697360c46138ef4cd21e8c7a950 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 1.
I20251027 21:08:08.694665 9907 raft_consensus.cc:2468] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:08.694808 9860 leader_election.cc:304] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:08.695087 11177 raft_consensus.cc:2804] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:08.695202 11177 raft_consensus.cc:697] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Becoming Leader. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Running, Role: LEADER
I20251027 21:08:08.695441 11177 consensus_queue.cc:237] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:08.695879 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet c9c86253ce914916bee6584675a82067 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:08.696306 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet cdbde6272af147829ecadd67d634f049 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:08.696347 9568 leader_election.cc:304] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:08.696622 9903 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e4d35f229bc46e1859ded7c8196ab9d with delete_type TABLET_DATA_DELETED (replaced by tablet 280ee23d80d64587b18811ff06340eee) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:08.696820 11097 raft_consensus.cc:2804] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:08.696928 11097 raft_consensus.cc:697] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:08.697057 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0feda3c71548490f8eae7b3f70073767"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:08.697136 11097 consensus_queue.cc:237] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:08.697611 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0feda3c71548490f8eae7b3f70073767 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:08.698118 10041 ts_tablet_manager.cc:1950] T 9c2d713d72fd4b5ca34cb3e828c75761 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:08.698426 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_303556148a4445a191ca828fb6582bd2" } new_table_name: "default.Table_cd02221d475648e69ac0fe6f18cdbda4" modify_external_catalogs: true
I20251027 21:08:08.702302 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet e53b2c28c75b43bdafa0d20f4ad61fcc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:08.703831 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "334c580862bd4bf09be3f1710aa6a092" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:08.704905 9710 consensus_peers.cc:597] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 -> Peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Couldn't send request to peer 51902b6b61ff4d5584ff21896e847e14. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:08.705050 9710 consensus_peers.cc:597] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 -> Peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Couldn't send request to peer 51902b6b61ff4d5584ff21896e847e14. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3b9a461567a48fcb4af514ecdf618b4. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:08.705147 9710 leader_election.cc:343] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: da26c83976df40328cf9b11734fcfa44
I20251027 21:08:08.705613 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2d26784bbb24ae3893103802c41affb" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:08.707034 11162 raft_consensus.cc:493] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251027 21:08:08.707110 9566 leader_election.cc:343] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.707190 11162 raft_consensus.cc:515] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.707860 11162 leader_election.cc:290] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.708066 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fa39697360c46138ef4cd21e8c7a950" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:08.711220 9771 raft_consensus.cc:3060] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:08.712333 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 990) took 3163 ms. Trace:
I20251027 21:08:08.712400 10040 rpcz_store.cc:276] 1027 21:08:05.548366 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.121131 (+572765us) service_pool.cc:225] Handling call
1027 21:08:07.954405 (+1833274us) ts_tablet_manager.cc:1151] Deleting tablet 6f1d1bd52556463fa306e51eb253af40
1027 21:08:07.954415 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.400549 (+446134us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:08.497997 (+ 97448us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:08.712320 (+214323us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1833144}
I20251027 21:08:08.712725 11142 raft_consensus.cc:493] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.712854 11142 raft_consensus.cc:515] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.713362 11142 leader_election.cc:290] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.713961 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2203ee62f7e404c95c5f86b5ae85feb" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:08.714604 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 989) took 3166 ms. Trace:
I20251027 21:08:08.714679 10040 rpcz_store.cc:276] 1027 21:08:05.548267 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.085657 (+537390us) service_pool.cc:225] Handling call
1027 21:08:08.712692 (+2627035us) ts_tablet_manager.cc:1151] Deleting tablet 8649a3b611cc48ff8832c0fdeb128efe
1027 21:08:08.712704 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.714590 (+ 1886us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2588869}
I20251027 21:08:08.714619 9770 raft_consensus.cc:1275] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218 [term 2 FOLLOWER]: Refusing update from remote peer 0fbd567f929a4f6d8e4d78feef7ed678: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251027 21:08:08.715922 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 6f1d1bd52556463fa306e51eb253af40 (table default.Table_9b06fbc2554c4780a754ed9cfb964fa9 [id=47a1de450a474742ae41e277716f5381]) successfully deleted
W20251027 21:08:08.716269 9859 leader_election.cc:343] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.716372 9859 leader_election.cc:304] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:08.716778 9859 leader_election.cc:343] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:08.717064 9859 leader_election.cc:343] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.717095 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7292bc6c22ca4ab5b8f53a69520833ba" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:08.717197 9859 leader_election.cc:304] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.717317 9909 raft_consensus.cc:2468] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:08.717116 11234 raft_consensus.cc:2749] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:08.717609 11234 raft_consensus.cc:2749] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:08.717712 9568 leader_election.cc:304] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:08.717939 11142 raft_consensus.cc:2804] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:08.717942 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6ed9d7a700446edafcd0daa02aa7ef6" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:08.718014 11142 raft_consensus.cc:493] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:08.718077 11142 raft_consensus.cc:3060] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:08.714946 10040 tablet_replica.cc:333] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:08.718454 10040 raft_consensus.cc:2243] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:08.718583 10040 raft_consensus.cc:2272] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:08.719081 10040 ts_tablet_manager.cc:1916] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:08.721952 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d3be7b0de84549a38b43f01c7efd058a" } new_table_name: "default.Table_27255b9546f64e8e83f1de3c76447020" modify_external_catalogs: true
I20251027 21:08:08.725765 10032 tablet_bootstrap.cc:492] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.725925 10032 ts_tablet_manager.cc:1403] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.073s user 0.000s sys 0.008s
I20251027 21:08:08.726727 10032 raft_consensus.cc:359] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.726934 10032 raft_consensus.cc:385] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.727011 10032 raft_consensus.cc:740] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.727222 10032 consensus_queue.cc:260] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.727712 10032 ts_tablet_manager.cc:1434] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:08.728008 10032 tablet_bootstrap.cc:492] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:08.729975 10032 tablet_bootstrap.cc:654] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.732532 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6ed9d7a700446edafcd0daa02aa7ef6" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:08.733807 9308 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:50804 (request call id 1) took 1097 ms. Trace:
I20251027 21:08:08.733906 9308 rpcz_store.cc:276] 1027 21:08:07.636143 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.732302 (+1096159us) service_pool.cc:225] Handling call
1027 21:08:08.733602 (+ 1300us) spinlock_profiling.cc:243] Waited 1.03 ms on lock 0x616000051154. stack: 00007f956e0a89f6 00007f956e0a87b0 00007f956d0843ad 0000000000983f04 0000000000983ecd 00007f957d3af6bd 00007f957d3af7b3 00007f9579f76441 00007f9581c309dd 00007f9581dc5965 00007f95788df747 00007f95788df475 00007f957504727e 00007f957504615e 00007f957504c395 00007f957504ccbf
1027 21:08:08.733717 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2362752}
W20251027 21:08:08.734398 9711 leader_election.cc:343] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: da26c83976df40328cf9b11734fcfa44
I20251027 21:08:08.734510 9711 leader_election.cc:304] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:08.734721 10997 raft_consensus.cc:2749] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:08.734967 9711 consensus_peers.cc:597] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2a4aa1fe05114c1287ca580c1f411d3f. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:08.735451 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 8649a3b611cc48ff8832c0fdeb128efe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8649a3b611cc48ff8832c0fdeb128efe already in progress: creating tablet
I20251027 21:08:08.736567 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7292bc6c22ca4ab5b8f53a69520833ba" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:08.736981 9566 leader_election.cc:343] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.738675 9308 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 980eb6755aee4bcfb2ea44d84009021f (table default.Table_bb56319b6fe0405991a4d171477621e7 [id=42c8fedcabe0435dbaa11427141228ed]). Expected version 1 got 0
W20251027 21:08:08.739820 9859 leader_election.cc:343] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.740020 11162 consensus_queue.cc:1048] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 0, Time since last communication: 0.000s
W20251027 21:08:08.742835 9860 leader_election.cc:343] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.742959 9860 leader_election.cc:304] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.743453 11234 raft_consensus.cc:2749] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:08.746446 10032 tablet_bootstrap.cc:492] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.746642 10032 ts_tablet_manager.cc:1403] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20251027 21:08:08.747570 10032 raft_consensus.cc:359] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.747908 10032 raft_consensus.cc:385] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.748026 10032 raft_consensus.cc:740] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.748235 10032 consensus_queue.cc:260] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.748754 10032 ts_tablet_manager.cc:1434] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:08.749037 10032 tablet_bootstrap.cc:492] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:08.756749 10032 tablet_bootstrap.cc:654] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.756961 11097 raft_consensus.cc:493] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.757105 11097 raft_consensus.cc:515] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.757752 11097 leader_election.cc:290] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.758217 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f6789dc2f7cb4a3c826ac507d5c9a239" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:08.758694 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f6789dc2f7cb4a3c826ac507d5c9a239" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:08.758893 9909 raft_consensus.cc:2468] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
W20251027 21:08:08.758910 9566 leader_election.cc:343] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.759245 9568 leader_election.cc:304] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.759482 11097 raft_consensus.cc:2804] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:08.759557 11097 raft_consensus.cc:493] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:08.759615 11097 raft_consensus.cc:3060] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:08.766244 11300 tablet.cc:1722] T d56d323ef0e74344a479940b93fea11a P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:08.770349 11177 raft_consensus.cc:493] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.770493 11177 raft_consensus.cc:515] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.771395 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "371051fb3491426e8cf7797317baa3db" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:08.772004 9860 leader_election.cc:343] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.772274 11177 leader_election.cc:290] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.773937 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "371051fb3491426e8cf7797317baa3db" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:08.775520 9859 leader_election.cc:343] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.775642 9859 leader_election.cc:304] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.776100 11177 raft_consensus.cc:2749] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:08.779194 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4eb30f4e4b094693a2c1088cfaa68ba3 consensus state has no leader
I20251027 21:08:08.792150 11142 raft_consensus.cc:515] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.792729 11142 leader_election.cc:290] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.793540 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 951) took 3945 ms. Trace:
I20251027 21:08:08.793632 10043 rpcz_store.cc:276] 1027 21:08:04.848237 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.848286 (+ 49us) service_pool.cc:225] Handling call
1027 21:08:08.488993 (+3640707us) ts_tablet_manager.cc:1151] Deleting tablet 8b30052ff19744a9bf4bfaaa1bc77b36
1027 21:08:08.490173 (+ 1180us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.598308 (+108135us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:08.692812 (+ 94504us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:08.793504 (+100692us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3640545}
I20251027 21:08:08.794092 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 952) took 3917 ms. Trace:
I20251027 21:08:08.794188 10043 rpcz_store.cc:276] 1027 21:08:04.876197 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.876443 (+ 246us) service_pool.cc:225] Handling call
1027 21:08:08.793952 (+3917509us) ts_tablet_manager.cc:1151] Deleting tablet 532ea47b15b544dfb05ddb2bdacf3953
1027 21:08:08.793964 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.794077 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3917356}
I20251027 21:08:08.794452 10043 tablet_replica.cc:333] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:08.794677 10043 raft_consensus.cc:2243] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:08.794817 10043 raft_consensus.cc:2272] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:08.795383 10043 ts_tablet_manager.cc:1916] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:08.797379 11162 raft_consensus.cc:493] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.797542 11162 raft_consensus.cc:515] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.798153 11162 leader_election.cc:290] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.798779 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6210d3d10fc44b9e9e2b0780883186ce" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:08.799288 9860 leader_election.cc:343] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.801062 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 927) took 4605 ms. Trace:
I20251027 21:08:08.801151 10041 rpcz_store.cc:276] 1027 21:08:04.195196 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.195255 (+ 59us) service_pool.cc:225] Handling call
1027 21:08:08.508770 (+4313515us) ts_tablet_manager.cc:1151] Deleting tablet 9c2d713d72fd4b5ca34cb3e828c75761
1027 21:08:08.508779 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.595068 (+ 86289us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:08.693644 (+ 98576us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:08.801008 (+107364us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4313340}
I20251027 21:08:08.801558 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 928) took 4552 ms. Trace:
I20251027 21:08:08.807246 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 9c2d713d72fd4b5ca34cb3e828c75761 (table default.Table_294e3c4092274dc1a146d3f604bc2179 [id=bd8f38a55b9f42b99e372c133a148007]) successfully deleted
I20251027 21:08:08.807674 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6210d3d10fc44b9e9e2b0780883186ce" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:08.808177 9859 leader_election.cc:343] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.808265 9859 leader_election.cc:304] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.808586 11177 raft_consensus.cc:2749] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:08.811538 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7292bc6c22ca4ab5b8f53a69520833ba" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:08.811724 9909 raft_consensus.cc:3060] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:08.812177 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bc505d437e774afcbd030a4e88375649 consensus state has no leader
W20251027 21:08:08.815080 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b260fe61fd6d43d59ef7fd77a2532cc2 consensus state has no leader
I20251027 21:08:08.818238 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7292bc6c22ca4ab5b8f53a69520833ba" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:08.818810 9566 leader_election.cc:343] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.823642 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 8b30052ff19744a9bf4bfaaa1bc77b36 (table default.Table_5dc88891b75b4f1ca7b3ffba0d2b1fda [id=b46f1e4f1e6d4bd8b4a3edd1d6787612]) successfully deleted
W20251027 21:08:08.823936 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 532ea47b15b544dfb05ddb2bdacf3953 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 532ea47b15b544dfb05ddb2bdacf3953 already in progress: creating tablet
I20251027 21:08:08.837082 11177 raft_consensus.cc:493] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.837229 11177 raft_consensus.cc:515] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.837889 11177 leader_election.cc:290] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.838795 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "465be1b24d16457e834023a2b2194049" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:08.839257 9860 leader_election.cc:343] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.839648 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "465be1b24d16457e834023a2b2194049" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:08.840090 9859 leader_election.cc:343] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.840188 9859 leader_election.cc:304] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.840585 11177 raft_consensus.cc:2749] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:08.840597 11319 raft_consensus.cc:493] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.840749 11319 raft_consensus.cc:515] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:08.842053 9711 leader_election.cc:343] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: bbb4116d1edd492893adfbf5fb3ddeef
I20251027 21:08:08.842855 11319 leader_election.cc:290] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:08.843125 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bbb4116d1edd492893adfbf5fb3ddeef" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:08.843326 9632 raft_consensus.cc:2468] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:08:08.843743 9710 leader_election.cc:304] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:08.844187 11317 raft_consensus.cc:2804] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:08.844288 11317 raft_consensus.cc:493] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:08.844383 11317 raft_consensus.cc:3060] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:08.849220 10029 tablet_bootstrap.cc:492] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:08.851260 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7ed5ee38ed7a4cf1a5a706eb6e233d41" } new_table_name: "default.Table_04741665fd574abfada5d7d96a5fe910" modify_external_catalogs: true
I20251027 21:08:08.852051 10032 tablet_bootstrap.cc:492] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.852177 10032 ts_tablet_manager.cc:1403] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.103s user 0.000s sys 0.004s
I20251027 21:08:08.852828 10032 raft_consensus.cc:359] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.852979 10032 raft_consensus.cc:385] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.853030 10032 raft_consensus.cc:740] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.853195 10032 consensus_queue.cc:260] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.853647 10032 ts_tablet_manager.cc:1434] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:08.854045 10029 ts_tablet_manager.cc:1403] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.248s user 0.008s sys 0.000s
I20251027 21:08:08.854883 10029 raft_consensus.cc:359] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.855652 10041 rpcz_store.cc:276] 1027 21:08:04.249389 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.249459 (+ 70us) service_pool.cc:225] Handling call
1027 21:08:08.801379 (+4551920us) ts_tablet_manager.cc:1151] Deleting tablet 196a594025ad4564b028b39f1afae48f
1027 21:08:08.801398 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.801497 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4551748}
I20251027 21:08:08.855906 10029 raft_consensus.cc:385] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.856123 10029 raft_consensus.cc:740] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
W20251027 21:08:08.856310 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 196a594025ad4564b028b39f1afae48f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 196a594025ad4564b028b39f1afae48f already in progress: creating tablet
I20251027 21:08:08.856789 10029 consensus_queue.cc:260] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.857549 10029 ts_tablet_manager.cc:1434] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251027 21:08:08.862525 10032 tablet_bootstrap.cc:492] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:08.863901 10032 tablet_bootstrap.cc:654] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.868599 10031 ts_tablet_manager.cc:1434] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.427s user 0.001s sys 0.011s
I20251027 21:08:08.868959 10031 tablet_bootstrap.cc:492] T 9fde4fd91943498d8888d1783005e492 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:08.870676 10031 tablet_bootstrap.cc:654] T 9fde4fd91943498d8888d1783005e492 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.872020 9632 raft_consensus.cc:1275] T 345d3795a37b4a40b40967cfacf0f893 P 51902b6b61ff4d5584ff21896e847e14 [term 2 FOLLOWER]: Refusing update from remote peer 0fbd567f929a4f6d8e4d78feef7ed678: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251027 21:08:08.872972 10032 tablet_bootstrap.cc:492] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.873098 10032 ts_tablet_manager.cc:1403] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.005s
I20251027 21:08:08.873096 11177 consensus_queue.cc:1048] T 345d3795a37b4a40b40967cfacf0f893 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 21:08:08.873790 10032 raft_consensus.cc:359] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.873941 10032 raft_consensus.cc:385] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.873989 10032 raft_consensus.cc:740] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.874198 10032 consensus_queue.cc:260] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.874644 10032 ts_tablet_manager.cc:1434] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 21:08:08.874898 10032 tablet_bootstrap.cc:492] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:08.876077 11339 raft_consensus.cc:493] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.876209 10032 tablet_bootstrap.cc:654] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.876305 11142 raft_consensus.cc:493] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.876338 11339 raft_consensus.cc:515] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.876437 11142 raft_consensus.cc:515] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:08.877018 11142 leader_election.cc:290] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.877542 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaa8910d9fdb4146945621ec4ebf969e" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:08.877977 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d64a13667194c12864fa388eb88a380" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:08.878381 9920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaa8910d9fdb4146945621ec4ebf969e" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:08.878903 9711 leader_election.cc:304] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
W20251027 21:08:08.879614 9566 leader_election.cc:343] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.880151 9920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d64a13667194c12864fa388eb88a380" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:08.880311 9920 raft_consensus.cc:2468] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:08.881687 9568 leader_election.cc:304] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.882253 11283 raft_consensus.cc:2804] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:08.882334 11283 raft_consensus.cc:493] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:08.882400 11283 raft_consensus.cc:3060] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:08.882790 11317 raft_consensus.cc:515] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.883544 11317 leader_election.cc:290] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:08.883715 11317 raft_consensus.cc:2749] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:08.884734 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bbb4116d1edd492893adfbf5fb3ddeef" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:08.884956 9633 raft_consensus.cc:3060] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:08.885599 9711 leader_election.cc:343] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: bbb4116d1edd492893adfbf5fb3ddeef
I20251027 21:08:08.885914 11339 leader_election.cc:290] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:08.886653 11097 raft_consensus.cc:515] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.886895 11300 tablet.cc:1722] T 345d3795a37b4a40b40967cfacf0f893 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:08.887272 11097 leader_election.cc:290] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.887462 10040 ts_tablet_manager.cc:1929] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:08.887571 10040 log.cc:1199] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/e9edb62a8e4b4dfea6a31a2202545a6e
I20251027 21:08:08.887573 9909 raft_consensus.cc:2468] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:08.887939 10040 ts_tablet_manager.cc:1950] T e9edb62a8e4b4dfea6a31a2202545a6e P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:08.890941 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c40e78267cfd4b84b63987b1c910818f" } modify_external_catalogs: true
I20251027 21:08:08.891155 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c40e78267cfd4b84b63987b1c910818f" } modify_external_catalogs: true
I20251027 21:08:08.891264 10032 tablet_bootstrap.cc:492] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.891397 10032 ts_tablet_manager.cc:1403] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20251027 21:08:08.892125 10032 raft_consensus.cc:359] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.892274 10032 raft_consensus.cc:385] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.892319 10032 raft_consensus.cc:740] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.892529 10032 consensus_queue.cc:260] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.892829 9568 leader_election.cc:304] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.892988 10032 ts_tablet_manager.cc:1434] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:08.893230 10032 tablet_bootstrap.cc:492] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:08.893543 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f6789dc2f7cb4a3c826ac507d5c9a239" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:08.893723 9909 raft_consensus.cc:3060] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:08.893836 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6d50127567904f7cab81c7a90c655c3d" } modify_external_catalogs: true
I20251027 21:08:08.894043 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6d50127567904f7cab81c7a90c655c3d" } modify_external_catalogs: true
I20251027 21:08:08.894573 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2aad78a009654cb88858d59b51ee78cb" } modify_external_catalogs: true
I20251027 21:08:08.894714 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2aad78a009654cb88858d59b51ee78cb" } modify_external_catalogs: true
I20251027 21:08:08.894982 10032 tablet_bootstrap.cc:654] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.895182 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7057cb41215c437391bce90bc612d1df" } modify_external_catalogs: true
I20251027 21:08:08.895350 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7057cb41215c437391bce90bc612d1df" } modify_external_catalogs: true
I20251027 21:08:08.895828 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c305a5155cce4754bcbc792f3900b690" } modify_external_catalogs: true
I20251027 21:08:08.895985 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c305a5155cce4754bcbc792f3900b690" } modify_external_catalogs: true
I20251027 21:08:08.896723 11097 raft_consensus.cc:2804] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:08.896910 11097 raft_consensus.cc:697] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:08.897171 11097 consensus_queue.cc:237] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.897750 10032 tablet_bootstrap.cc:492] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.897857 10032 ts_tablet_manager.cc:1403] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251027 21:08:08.898595 10032 raft_consensus.cc:359] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.898725 10032 raft_consensus.cc:385] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.898764 10032 raft_consensus.cc:740] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.898960 10032 consensus_queue.cc:260] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.899371 10032 ts_tablet_manager.cc:1434] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251027 21:08:08.902684 10032 tablet_bootstrap.cc:492] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:08.903184 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f6789dc2f7cb4a3c826ac507d5c9a239" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:08.903251 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1023) took 2463 ms. Trace:
I20251027 21:08:08.903386 10040 rpcz_store.cc:276] 1027 21:08:06.439486 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.879877 (+440391us) service_pool.cc:225] Handling call
1027 21:08:08.714903 (+1835026us) ts_tablet_manager.cc:1151] Deleting tablet e9edb62a8e4b4dfea6a31a2202545a6e
1027 21:08:08.714912 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.790527 (+ 75615us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:08.887342 (+ 96815us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:08.903212 (+ 15870us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15616,"tablet-delete.queue_time_us":1834878}
I20251027 21:08:08.904224 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1027) took 2248 ms. Trace:
I20251027 21:08:08.904281 10032 tablet_bootstrap.cc:654] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.904383 10040 rpcz_store.cc:276] 1027 21:08:06.655949 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.113453 (+457504us) service_pool.cc:225] Handling call
1027 21:08:08.904090 (+1790637us) ts_tablet_manager.cc:1151] Deleting tablet c548d2413a5940c8b114ced667b4dc41
1027 21:08:08.904099 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.904195 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1790490}
I20251027 21:08:08.904439 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet e9edb62a8e4b4dfea6a31a2202545a6e (table default.Table_2d874641d88e434c8022d7026802972f [id=7b94fb706c8e429b911e1e19265a3883]) successfully deleted
W20251027 21:08:08.904731 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c548d2413a5940c8b114ced667b4dc41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c548d2413a5940c8b114ced667b4dc41 already in progress: creating tablet
I20251027 21:08:08.904891 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1028) took 2110 ms. Trace:
I20251027 21:08:08.905010 10040 rpcz_store.cc:276] 1027 21:08:06.794246 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.113612 (+319366us) service_pool.cc:225] Handling call
1027 21:08:08.904741 (+1791129us) ts_tablet_manager.cc:1151] Deleting tablet 4c67c5cb3b244feb8478f9594b834fbf
1027 21:08:08.904749 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.904834 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1791011}
W20251027 21:08:08.905081 9566 leader_election.cc:343] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.905470 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1029) took 2049 ms. Trace:
W20251027 21:08:08.905514 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 4c67c5cb3b244feb8478f9594b834fbf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c67c5cb3b244feb8478f9594b834fbf already in progress: creating tablet
I20251027 21:08:08.905671 10040 rpcz_store.cc:276] 1027 21:08:06.856318 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.113741 (+257423us) service_pool.cc:225] Handling call
1027 21:08:08.905359 (+1791618us) ts_tablet_manager.cc:1151] Deleting tablet 17a19ce92da8448bba471f94e3ee4647
1027 21:08:08.905367 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.905439 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1791517}
I20251027 21:08:08.906205 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1030) took 2049 ms. Trace:
W20251027 21:08:08.906312 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 17a19ce92da8448bba471f94e3ee4647 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17a19ce92da8448bba471f94e3ee4647 already in progress: creating tablet
I20251027 21:08:08.906365 10040 rpcz_store.cc:276] 1027 21:08:06.856399 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.113861 (+257462us) service_pool.cc:225] Handling call
1027 21:08:08.906063 (+1792202us) ts_tablet_manager.cc:1151] Deleting tablet 35b99f8c57e847b1848cbdea9e50da11
1027 21:08:08.906072 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.906171 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1792102}
W20251027 21:08:08.906950 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 35b99f8c57e847b1848cbdea9e50da11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35b99f8c57e847b1848cbdea9e50da11 already in progress: creating tablet
I20251027 21:08:08.907517 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1031) took 2051 ms. Trace:
I20251027 21:08:08.907662 10040 rpcz_store.cc:276] 1027 21:08:06.856503 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.113970 (+257467us) service_pool.cc:225] Handling call
1027 21:08:08.907405 (+1793435us) ts_tablet_manager.cc:1151] Deleting tablet 933fb952d66d4cb1ade70c334fbbb3e8
1027 21:08:08.907413 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.907481 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1793344}
I20251027 21:08:08.908227 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1032) took 2051 ms. Trace:
I20251027 21:08:08.908390 10040 rpcz_store.cc:276] 1027 21:08:06.856587 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.114067 (+257480us) service_pool.cc:225] Handling call
1027 21:08:08.908123 (+1794056us) ts_tablet_manager.cc:1151] Deleting tablet 4a621d5246f14997a2f2952ab640efc8
1027 21:08:08.908130 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.908195 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1725894}
W20251027 21:08:08.908627 9710 consensus_peers.cc:597] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 -> Peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Couldn't send request to peer 51902b6b61ff4d5584ff21896e847e14. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2a4aa1fe05114c1287ca580c1f411d3f. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:08.908944 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1033) took 2052 ms. Trace:
I20251027 21:08:08.909111 10040 rpcz_store.cc:276] 1027 21:08:06.856664 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.182261 (+325597us) service_pool.cc:225] Handling call
1027 21:08:08.908834 (+1726573us) ts_tablet_manager.cc:1151] Deleting tablet 8dc358445526415d8cd4f1ac6779126d
1027 21:08:08.908842 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.908912 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1726385}
I20251027 21:08:08.909595 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1034) took 2052 ms. Trace:
I20251027 21:08:08.909759 10040 rpcz_store.cc:276] 1027 21:08:06.856755 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.182467 (+325712us) service_pool.cc:225] Handling call
1027 21:08:08.909484 (+1727017us) ts_tablet_manager.cc:1151] Deleting tablet db2735604dc14039810df4ecd330c671
1027 21:08:08.909494 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.909562 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1726857}
W20251027 21:08:08.910274 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 933fb952d66d4cb1ade70c334fbbb3e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 933fb952d66d4cb1ade70c334fbbb3e8 already in progress: creating tablet
W20251027 21:08:08.910516 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 4a621d5246f14997a2f2952ab640efc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a621d5246f14997a2f2952ab640efc8 already in progress: creating tablet
W20251027 21:08:08.910704 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 8dc358445526415d8cd4f1ac6779126d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dc358445526415d8cd4f1ac6779126d already in progress: creating tablet
W20251027 21:08:08.910925 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet db2735604dc14039810df4ecd330c671 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db2735604dc14039810df4ecd330c671 already in progress: creating tablet
W20251027 21:08:08.911262 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93e8b973b39244be9de1107b14d6d93f consensus state has no leader
I20251027 21:08:08.911753 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6f18157ff71c4f37b28566ed2e696c74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:08.912339 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f18157ff71c4f37b28566ed2e696c74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:08.912895 10040 tablet_replica.cc:333] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:08.913158 10040 raft_consensus.cc:2243] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:08.915436 10040 raft_consensus.cc:2272] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:08.916133 10040 ts_tablet_manager.cc:1916] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:08.916474 10032 tablet_bootstrap.cc:492] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.916625 10032 ts_tablet_manager.cc:1403] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20251027 21:08:08.917747 10032 raft_consensus.cc:359] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.917922 10032 raft_consensus.cc:385] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.917974 10032 raft_consensus.cc:740] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.918211 10032 consensus_queue.cc:260] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.918684 10032 ts_tablet_manager.cc:1434] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:08.922181 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_80cde2bc1ed84de29cec9255f33b03d3" } new_table_name: "default.Table_fc85306eb7b54fd0ac3e6dd117bd3cc8" modify_external_catalogs: true
I20251027 21:08:08.923527 10031 tablet_bootstrap.cc:492] T 9fde4fd91943498d8888d1783005e492 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:08.924965 10031 ts_tablet_manager.cc:1403] T 9fde4fd91943498d8888d1783005e492 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.000s
I20251027 21:08:08.926710 10031 raft_consensus.cc:359] T 9fde4fd91943498d8888d1783005e492 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.927083 10032 tablet_bootstrap.cc:492] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:08.929240 10032 tablet_bootstrap.cc:654] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:08.936074 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 38e6a73ff9314e47a16a6c5164650828 consensus state has no leader
I20251027 21:08:08.936892 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_cd6e6c1476a64edbb448e788e60755fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:08.938448 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd6e6c1476a64edbb448e788e60755fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:08.942965 10029 tablet_bootstrap.cc:492] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:08.944428 10032 tablet_bootstrap.cc:492] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.944576 10032 ts_tablet_manager.cc:1403] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.002s
I20251027 21:08:08.945310 10032 raft_consensus.cc:359] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.945478 10032 raft_consensus.cc:385] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.945542 10032 raft_consensus.cc:740] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.945777 10032 consensus_queue.cc:260] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.946318 10032 ts_tablet_manager.cc:1434] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:08.946597 10032 tablet_bootstrap.cc:492] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:08.947721 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_14d171d5f33e4baf9d9945618ae3e2d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:08.948132 10032 tablet_bootstrap.cc:654] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:08.948447 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14d171d5f33e4baf9d9945618ae3e2d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:08.950562 10031 raft_consensus.cc:385] T 9fde4fd91943498d8888d1783005e492 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.950671 10031 raft_consensus.cc:740] T 9fde4fd91943498d8888d1783005e492 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.950956 10031 consensus_queue.cc:260] T 9fde4fd91943498d8888d1783005e492 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.951503 10031 ts_tablet_manager.cc:1434] T 9fde4fd91943498d8888d1783005e492 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.026s user 0.001s sys 0.000s
I20251027 21:08:08.951833 10031 tablet_bootstrap.cc:492] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:08.954644 10032 tablet_bootstrap.cc:492] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.954769 10032 ts_tablet_manager.cc:1403] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20251027 21:08:08.955477 10032 raft_consensus.cc:359] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.955648 10032 raft_consensus.cc:385] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.955698 10032 raft_consensus.cc:740] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.955919 10032 consensus_queue.cc:260] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.956367 10032 ts_tablet_manager.cc:1434] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:08.956651 10032 tablet_bootstrap.cc:492] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:08.957454 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 929) took 4707 ms. Trace:
I20251027 21:08:08.957620 10041 rpcz_store.cc:276] 1027 21:08:04.250297 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.250442 (+ 145us) service_pool.cc:225] Handling call
1027 21:08:08.856811 (+4606369us) ts_tablet_manager.cc:1151] Deleting tablet 070167be7d8d4ddf8da561df14448360
1027 21:08:08.946021 (+ 89210us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.957435 (+ 11414us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4606237}
I20251027 21:08:08.958638 10041 tablet_replica.cc:333] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
W20251027 21:08:08.958846 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 070167be7d8d4ddf8da561df14448360 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 070167be7d8d4ddf8da561df14448360 already in progress: creating tablet
I20251027 21:08:08.958906 10041 raft_consensus.cc:2243] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:08.959157 11162 raft_consensus.cc:493] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.959272 11162 raft_consensus.cc:515] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.959352 10041 raft_consensus.cc:2272] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:08.959846 11162 leader_election.cc:290] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:08.961179 10029 tablet_bootstrap.cc:654] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.961410 10032 tablet_bootstrap.cc:654] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.961565 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "371aa8a57aee466ab01652d6061db4c2" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:08.962085 9860 leader_election.cc:343] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.962477 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "371aa8a57aee466ab01652d6061db4c2" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:08.963115 9859 leader_election.cc:343] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:08.963207 9859 leader_election.cc:304] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:08.963316 9771 raft_consensus.cc:3060] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:08.963554 11162 raft_consensus.cc:2749] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:08.966440 10041 ts_tablet_manager.cc:1916] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:08.976608 10032 tablet_bootstrap.cc:492] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:08.976780 10032 ts_tablet_manager.cc:1403] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.002s
I20251027 21:08:08.976804 10031 tablet_bootstrap.cc:654] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.977701 10032 raft_consensus.cc:359] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.977871 10032 raft_consensus.cc:385] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.977924 10032 raft_consensus.cc:740] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.978152 10032 consensus_queue.cc:260] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.978586 10032 ts_tablet_manager.cc:1434] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251027 21:08:08.978976 9566 consensus_peers.cc:597] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:08.983520 10031 tablet_bootstrap.cc:492] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:08.983688 10031 ts_tablet_manager.cc:1403] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20251027 21:08:08.984042 10032 tablet_bootstrap.cc:492] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:08.985250 10031 raft_consensus.cc:359] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.985487 10031 raft_consensus.cc:385] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:08.985601 10032 tablet_bootstrap.cc:654] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:08.985740 10031 raft_consensus.cc:740] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:08.986001 10031 consensus_queue.cc:260] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.987007 10031 ts_tablet_manager.cc:1434] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:08:08.989163 9633 raft_consensus.cc:2468] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:08.998651 9710 leader_election.cc:304] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:08.999110 11339 raft_consensus.cc:2804] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:08.999182 11234 raft_consensus.cc:493] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:08.999306 11234 raft_consensus.cc:515] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:08.999575 11339 raft_consensus.cc:697] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:08.999961 11234 leader_election.cc:290] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.000502 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e4d35f229bc46e1859ded7c8196ab9d" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:09.001044 9860 leader_election.cc:343] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.001591 10032 tablet_bootstrap.cc:492] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:09.001705 10032 ts_tablet_manager.cc:1403] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.001s
I20251027 21:08:09.002490 10032 raft_consensus.cc:359] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.002748 10032 raft_consensus.cc:385] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:09.002854 10032 raft_consensus.cc:740] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:09.003058 10032 consensus_queue.cc:260] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.004726 10032 ts_tablet_manager.cc:1434] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:08:09.004987 10032 tablet_bootstrap.cc:492] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:09.005420 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 9fa39697360c46138ef4cd21e8c7a950 (table default.Table_b2a1b49649c54e5ab7a23fad7a157502 [id=434ed8c436c149ef9c7eb86ad252291d]). Expected version 4 got 0
I20251027 21:08:09.005853 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet aaa8910d9fdb4146945621ec4ebf969e (table default.Table_d8da07f0725347338012d992737c5d04 [id=bb676562b97f48088767941239cfbf33]). Expected version 2 got 0
I20251027 21:08:09.006634 10032 tablet_bootstrap.cc:654] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:09.007588 9903 tablet_service.cc:1505] Processing CreateTablet for tablet 2044c73d56bf4434879dd65f35c5612b (DEFAULT_TABLE table=default.Table_8fac6e33215e40128ceb1dfe8a622da6 [id=283672b4445b4be58a90a2f2573819ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.009871 9895 tablet_service.cc:1505] Processing CreateTablet for tablet a0d3203d12fc4823b3b3b1689f6bde62 (DEFAULT_TABLE table=default.Table_1cda8ade41884ccc959f290f1b26ffdb [id=af683a208c4a49d58e31366efee959f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.010910 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 1050ac06e1e74d54ba4014b49cc84467 (DEFAULT_TABLE table=default.Table_81e4dd46fa6945bc956cc69e26dabcf1 [id=692c338752a44253ba4abae40fd8ecab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.011488 10043 ts_tablet_manager.cc:1929] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:09.011665 10043 log.cc:1199] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/59b6818d29624da5a49dd1a7d20994c6
I20251027 21:08:09.012089 10043 ts_tablet_manager.cc:1950] T 59b6818d29624da5a49dd1a7d20994c6 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:09.012046 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 2044c73d56bf4434879dd65f35c5612b (DEFAULT_TABLE table=default.Table_8fac6e33215e40128ceb1dfe8a622da6 [id=283672b4445b4be58a90a2f2573819ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.012058 9312 catalog_manager.cc:5649] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } }
I20251027 21:08:09.012494 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2044c73d56bf4434879dd65f35c5612b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.012481 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet f2710f265ae9485dafec389168328067 (table default.Table_eb44e5cb7bc24597a807d81c1207a162 [id=27d1cff8fc7940c9abb741c8c7d87e24]). Expected version 2 got 0
I20251027 21:08:09.012774 9909 raft_consensus.cc:2468] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:09.012795 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 980eb6755aee4bcfb2ea44d84009021f (table default.Table_bb56319b6fe0405991a4d171477621e7 [id=42c8fedcabe0435dbaa11427141228ed]). Expected version 1 got 0
I20251027 21:08:09.013024 9312 catalog_manager.cc:5649] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } }
I20251027 21:08:09.014437 9603 tablet_service.cc:1505] Processing CreateTablet for tablet a0d3203d12fc4823b3b3b1689f6bde62 (DEFAULT_TABLE table=default.Table_1cda8ade41884ccc959f290f1b26ffdb [id=af683a208c4a49d58e31366efee959f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.014600 9905 tablet_service.cc:1505] Processing CreateTablet for tablet c442d337f5f945c097fa71b3833654f3 (DEFAULT_TABLE table=default.Table_57de17bcd9ef4ca3b5153678366c719d [id=bdeb9e4bcc9d40d19992b3cc93aa0126]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.014758 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0d3203d12fc4823b3b3b1689f6bde62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.015316 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 29693cf06bc24930870dc60457126519 (table default.Table_c09cc86fc2a64450af56fc584302da69 [id=d46947a29ddf4d94b5a9da22d9e5383e]). Expected version 3 got 0
I20251027 21:08:09.015693 9312 catalog_manager.cc:5649] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } }
I20251027 21:08:09.016067 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet d672118ecd6f4094855adf106e5f7115 (table default.Table_1b066b58b2c649a2887ed91461f15d7c [id=887ea33c2fbe4573a1d7169dfd226567]). Expected version 2 got 0
I20251027 21:08:09.016449 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 8766339bdedc42c2a41944b4928fa55e (DEFAULT_TABLE table=default.Table_74f4c8e80cf44de296017c6e93b8899e [id=a095e66c3c1e4944a314ff637b43e0f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.017556 9894 tablet_service.cc:1505] Processing CreateTablet for tablet 04c1f18b611d45c9ae2c7b895240f545 (DEFAULT_TABLE table=default.Table_1155ecb9419544b7b940e6699cb4464c [id=c1c73a49a8d149e9b129f11e8dfcd599]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.018543 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 1050ac06e1e74d54ba4014b49cc84467 (DEFAULT_TABLE table=default.Table_81e4dd46fa6945bc956cc69e26dabcf1 [id=692c338752a44253ba4abae40fd8ecab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.018657 9892 tablet_service.cc:1505] Processing CreateTablet for tablet fad3a2aa57f04f40ba65bf1b2832b9a6 (DEFAULT_TABLE table=default.Table_33a9f7934bc7496eaf4da402364c5179 [id=a7688ecec27747b388587b6c827ae09b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.018903 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1050ac06e1e74d54ba4014b49cc84467. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.020138 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 3f41ebd90a534b6a9ee8d0886641646a (DEFAULT_TABLE table=default.Table_8cdf4f6df0ec4bc4bb7758526f980e58 [id=f6edbfc5c0d043a382a8c89ed0d83279]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.021660 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 918a4a19c64f4261b58bea03d1b8251e (DEFAULT_TABLE table=default.Table_503f3a993de6475ea92ce352816fc954 [id=96587333b48b4774afca17f073c92c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.023348 9899 tablet_service.cc:1505] Processing CreateTablet for tablet aaccda3227f44a28a18d77f7bc7e6633 (DEFAULT_TABLE table=default.Table_c30777b91b944d75a8b8758a620faf1c [id=90dcef522b8045c1a5c9b8cd4a0a4228]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:09.026305 9278 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 145331us
I20251027 21:08:09.029706 11234 raft_consensus.cc:493] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.029834 11234 raft_consensus.cc:515] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.030452 11234 leader_election.cc:290] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.032506 10029 tablet_bootstrap.cc:492] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:09.037611 9310 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 29693cf06bc24930870dc60457126519 (table default.Table_c09cc86fc2a64450af56fc584302da69 [id=d46947a29ddf4d94b5a9da22d9e5383e]). Expected version 3 got 0
I20251027 21:08:09.037839 9310 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 76bee322d0b94abe9166170733608853 (table default.Table_4b0bc60a7c5e4888b79210296b7dfa20 [id=c9e7aa435d5b48e096c2b67b0e678eeb]). Expected version 1 got 0
I20251027 21:08:09.038120 9310 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 334c580862bd4bf09be3f1710aa6a092 (table default.Table_06c15ff076cd4c24bc49b16b01bff965 [id=cf9029e184114ac0a5b641a496701122]). Expected version 1 got 0
I20251027 21:08:09.038300 9310 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet e6b58b1f4f63461aa96e416cae879be8 (table default.Table_f3c2de263e8849caaef87b570fc69459 [id=ef50ef2238804c98ad19c5b1c0b881c4]). Expected version 1 got 0
I20251027 21:08:09.038448 9310 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 0e64d21abef342de86555e876f020787 (table default.Table_8ebde3403af34f4589ee6222700b2283 [id=cfdf2d0e9745403ea4bbeca6b4ed7439]). Expected version 2 got 0
I20251027 21:08:09.038430 10029 ts_tablet_manager.cc:1403] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.090s user 0.002s sys 0.001s
I20251027 21:08:09.038446 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1e58f6228254e3eb25dbf627a505ca1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:09.038662 9310 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 72c1dc3b9eff47ee81df821deb789307 (table default.Table_ff5458dda9f042e189bd8cd69c06d025 [id=2a96776404334cf79ddb611a234a5844]). Expected version 2 got 0
I20251027 21:08:09.039983 9308 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_c78ef9ebe86b4da08443e37c83d4e80d [id=cdb023218cda4e9ea04a226b37a220d9] alter complete (version 2)
W20251027 21:08:09.040256 9860 leader_election.cc:343] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.052724 10031 tablet_bootstrap.cc:492] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:09.053727 9744 tablet_service.cc:1505] Processing CreateTablet for tablet 2044c73d56bf4434879dd65f35c5612b (DEFAULT_TABLE table=default.Table_8fac6e33215e40128ceb1dfe8a622da6 [id=283672b4445b4be58a90a2f2573819ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.054298 9741 tablet_service.cc:1505] Processing CreateTablet for tablet a0d3203d12fc4823b3b3b1689f6bde62 (DEFAULT_TABLE table=default.Table_1cda8ade41884ccc959f290f1b26ffdb [id=af683a208c4a49d58e31366efee959f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.054931 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet e1e58f6228254e3eb25dbf627a505ca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.054925 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_27d4df68d5e7453295b814842d4089cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:09.055518 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27d4df68d5e7453295b814842d4089cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.055591 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet 42371496a7494e77a7bf1169de5b083a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.055799 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet d8691aa22fe84cf29893ed4e79d156cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.055980 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet 465be1b24d16457e834023a2b2194049 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.053798 9312 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_69627cab65c2407b9226a05106b7bdf4 [id=fc5a0b5a96e6400da438307d26f73614] alter complete (version 3)
I20251027 21:08:09.057165 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet 6210d3d10fc44b9e9e2b0780883186ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.057935 9747 tablet_service.cc:1505] Processing CreateTablet for tablet 1050ac06e1e74d54ba4014b49cc84467 (DEFAULT_TABLE table=default.Table_81e4dd46fa6945bc956cc69e26dabcf1 [id=692c338752a44253ba4abae40fd8ecab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.058328 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c2d713d72fd4b5ca34cb3e828c75761 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:08.999886 11339 consensus_queue.cc:237] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.058749 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e4d35f229bc46e1859ded7c8196ab9d" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:09.059818 9753 tablet_service.cc:1505] Processing CreateTablet for tablet c442d337f5f945c097fa71b3833654f3 (DEFAULT_TABLE table=default.Table_57de17bcd9ef4ca3b5153678366c719d [id=bdeb9e4bcc9d40d19992b3cc93aa0126]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.062826 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b42751988b8e445b9193301e2ef472e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:09.063112 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 04c1f18b611d45c9ae2c7b895240f545 (DEFAULT_TABLE table=default.Table_1155ecb9419544b7b940e6699cb4464c [id=c1c73a49a8d149e9b129f11e8dfcd599]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.063166 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 8766339bdedc42c2a41944b4928fa55e (DEFAULT_TABLE table=default.Table_74f4c8e80cf44de296017c6e93b8899e [id=a095e66c3c1e4944a314ff637b43e0f7]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:09.063762 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b42751988b8e445b9193301e2ef472e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.064739 9751 tablet_service.cc:1505] Processing CreateTablet for tablet aaccda3227f44a28a18d77f7bc7e6633 (DEFAULT_TABLE table=default.Table_c30777b91b944d75a8b8758a620faf1c [id=90dcef522b8045c1a5c9b8cd4a0a4228]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.065639 9738 tablet_service.cc:1505] Processing CreateTablet for tablet fad3a2aa57f04f40ba65bf1b2832b9a6 (DEFAULT_TABLE table=default.Table_33a9f7934bc7496eaf4da402364c5179 [id=a7688ecec27747b388587b6c827ae09b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.066432 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet bbb4116d1edd492893adfbf5fb3ddeef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:09.067571 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 3f41ebd90a534b6a9ee8d0886641646a (DEFAULT_TABLE table=default.Table_8cdf4f6df0ec4bc4bb7758526f980e58 [id=f6edbfc5c0d043a382a8c89ed0d83279]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:09.068368 9859 leader_election.cc:343] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.068490 9859 leader_election.cc:304] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:09.068943 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 229ea4caa6af493cb5b3d593f22aa113
I20251027 21:08:09.068970 11162 raft_consensus.cc:2749] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:09.068970 9743 tablet_service.cc:1505] Processing CreateTablet for tablet 918a4a19c64f4261b58bea03d1b8251e (DEFAULT_TABLE table=default.Table_503f3a993de6475ea92ce352816fc954 [id=96587333b48b4774afca17f073c92c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.069247 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1e58f6228254e3eb25dbf627a505ca1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:09.069661 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b1128d40cd914a9c9170d4d77fefa7cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:09.069851 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet b8c368698cfb40799571567e8bfdd8dc
W20251027 21:08:09.070158 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1128d40cd914a9c9170d4d77fefa7cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.070547 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet 229ea4caa6af493cb5b3d593f22aa113 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.071252 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 95446c4333424f10a71829ce32d857e4
I20251027 21:08:09.072129 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2da8f285aede4af7b444347f7183293f" } new_table_name: "default.Table_38143bba2f3d444aa0aa1eabb7bb1ff1" modify_external_catalogs: true
I20251027 21:08:09.072394 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b431103b133e4f548240ad2a4b4f9110"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:09.072891 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b431103b133e4f548240ad2a4b4f9110 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.073247 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet f1a931c36fec45029715b01754d0b161
I20251027 21:08:09.074064 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e82b4a8ed42e40f29b491317173bd664"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:09.074627 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e82b4a8ed42e40f29b491317173bd664 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.076260 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet cb5ac0fff70b491e8c8842f7a331005e
I20251027 21:08:09.077046 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_58ec53ca465d4c2b82b50f300edba453"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:09.083240 10032 tablet_bootstrap.cc:492] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:09.083353 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet cb5ac0fff70b491e8c8842f7a331005e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.083397 10032 ts_tablet_manager.cc:1403] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.078s user 0.007s sys 0.022s
I20251027 21:08:09.083539 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet f1a931c36fec45029715b01754d0b161 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.083714 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet 95446c4333424f10a71829ce32d857e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:08:09.084010 9859 leader_election.cc:343] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.084120 9859 leader_election.cc:304] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:09.084218 10032 raft_consensus.cc:359] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:09.084352 10032 raft_consensus.cc:385] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:09.084398 10032 raft_consensus.cc:740] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:09.084630 11162 raft_consensus.cc:2749] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:09.084601 10032 consensus_queue.cc:260] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:09.085103 10032 ts_tablet_manager.cc:1434] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:09.086652 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 520f7142f8af4599ac4880b0692e3151
I20251027 21:08:09.087271 9307 catalog_manager.cc:6719] ReplaceTablet: tablet 520f7142f8af4599ac4880b0692e3151 deleted and replaced by tablet 781a162e17dd495b888f7e389c062013
I20251027 21:08:09.087425 9307 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:41710 (request call id 1292) took 1110 ms. Trace:
I20251027 21:08:09.087903 10032 tablet_bootstrap.cc:492] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:09.087522 9307 rpcz_store.cc:276] 1027 21:08:07.976996 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.536842 (+559846us) service_pool.cc:225] Handling call
1027 21:08:08.596723 (+ 59881us) spinlock_profiling.cc:243] Waited 59.6 ms on lock 0x616000051154. stack: 00007f956e0a89f6 00007f956e0a87b0 00007f956d0843ad 0000000000983f04 0000000000983ecd 00007f957d3af6bd 00007f957d3af7b3 00007f9579f76441 00007f9581c309dd 00007f9581dc9fab 00007f95788da107 00007f95788d9e35 00007f957504727e 00007f957504615e 00007f957504c395 00007f957504ccbf
1027 21:08:09.087362 (+490639us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:08:09.006285 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:08:09.006392 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:08:09.006397 (+ 5us) write_op.cc:435] Acquired schema lock
1027 21:08:09.006401 (+ 4us) tablet.cc:660] Decoding operations
1027 21:08:09.006516 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:08:09.006525 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1027 21:08:09.006530 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:08:09.006617 (+ 87us) tablet.cc:699] Row locks acquired
1027 21:08:09.006621 (+ 4us) write_op.cc:260] PREPARE: finished
1027 21:08:09.006657 (+ 36us) write_op.cc:270] Start()
1027 21:08:09.006685 (+ 28us) write_op.cc:276] Timestamp: P: 1761599289006622 usec, L: 0
1027 21:08:09.006689 (+ 4us) op_driver.cc:348] REPLICATION: starting
1027 21:08:09.006875 (+ 186us) log.cc:844] Serialized 622 byte log entry
1027 21:08:09.027452 (+ 20577us) op_driver.cc:464] REPLICATION: finished
1027 21:08:09.031271 (+ 3819us) write_op.cc:301] APPLY: starting
1027 21:08:09.031285 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1027 21:08:09.031330 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
1027 21:08:09.031354 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:08:09.031466 (+ 112us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:08:09.031532 (+ 66us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:08:09.031539 (+ 7us) write_op.cc:312] APPLY: finished
1027 21:08:09.031604 (+ 65us) log.cc:844] Serialized 29 byte log entry
1027 21:08:09.031664 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:09.031682 (+ 18us) write_op.cc:454] Released schema lock
1027 21:08:09.031688 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1027 21:08:09.031748 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:09.031751 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":136972672,"child_traces":[["op",{"apply.queue_time_us":3776,"apply.run_cpu_time_us":471,"apply.run_wall_time_us":471,"num_ops":2,"prepare.queue_time_us":2063,"prepare.run_cpu_time_us":655,"prepare.run_wall_time_us":653,"replication_time_us":20724}]]}
I20251027 21:08:09.091048 10032 tablet_bootstrap.cc:654] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:09.091048 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7d6df7f8ab8945a894c5984d320667b0" } new_table_name: "default.Table_875795479fc54534888d2b89e3480f29" modify_external_catalogs: true
I20251027 21:08:09.093135 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet 520f7142f8af4599ac4880b0692e3151 with delete_type TABLET_DATA_DELETED (replaced by tablet 781a162e17dd495b888f7e389c062013) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.101420 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 95446c4333424f10a71829ce32d857e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:09.101799 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 229ea4caa6af493cb5b3d593f22aa113 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:09.109195 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c442d337f5f945c097fa71b3833654f3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:09.110024 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58ec53ca465d4c2b82b50f300edba453 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.113947 10031 tablet_bootstrap.cc:654] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:09.115388 11234 raft_consensus.cc:493] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.115558 11234 raft_consensus.cc:515] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.116217 11234 leader_election.cc:290] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.118978 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0a1b25591a094198b8123b3ac2af4715" } new_table_name: "default.Table_cd5a149804dd4eaa8b78dc67124958dd" modify_external_catalogs: true
I20251027 21:08:09.119702 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet b8c368698cfb40799571567e8bfdd8dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.121277 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8766339bdedc42c2a41944b4928fa55e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.121634 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fad3a2aa57f04f40ba65bf1b2832b9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.124886 11234 raft_consensus.cc:493] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.125065 11234 raft_consensus.cc:515] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.125898 11234 leader_election.cc:290] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.128686 11254 catalog_manager.cc:1269] Loaded cluster ID: 074cb2d5dc574bb39dc828c9d64a191a
I20251027 21:08:09.128859 11254 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: loading cluster ID for follower catalog manager: success
I20251027 21:08:09.130954 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 8766339bdedc42c2a41944b4928fa55e (DEFAULT_TABLE table=default.Table_74f4c8e80cf44de296017c6e93b8899e [id=a095e66c3c1e4944a314ff637b43e0f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.132697 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet cb5ac0fff70b491e8c8842f7a331005e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:09.133783 9615 tablet_service.cc:1505] Processing CreateTablet for tablet fad3a2aa57f04f40ba65bf1b2832b9a6 (DEFAULT_TABLE table=default.Table_33a9f7934bc7496eaf4da402364c5179 [id=a7688ecec27747b388587b6c827ae09b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.133906 11097 raft_consensus.cc:493] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.134174 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 918a4a19c64f4261b58bea03d1b8251e (DEFAULT_TABLE table=default.Table_503f3a993de6475ea92ce352816fc954 [id=96587333b48b4774afca17f073c92c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.134289 11097 raft_consensus.cc:515] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.133924 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 3f41ebd90a534b6a9ee8d0886641646a (DEFAULT_TABLE table=default.Table_8cdf4f6df0ec4bc4bb7758526f980e58 [id=f6edbfc5c0d043a382a8c89ed0d83279]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.134080 9602 tablet_service.cc:1505] Processing CreateTablet for tablet c442d337f5f945c097fa71b3833654f3 (DEFAULT_TABLE table=default.Table_57de17bcd9ef4ca3b5153678366c719d [id=bdeb9e4bcc9d40d19992b3cc93aa0126]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.135695 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1050ac06e1e74d54ba4014b49cc84467. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.135895 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8766339bdedc42c2a41944b4928fa55e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.136047 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 918a4a19c64f4261b58bea03d1b8251e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.136194 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaccda3227f44a28a18d77f7bc7e6633. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.136407 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f41ebd90a534b6a9ee8d0886641646a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.136520 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet a2d26784bbb24ae3893103802c41affb with delete_type TABLET_DATA_DELETED (replaced by tablet 9ace4188d3494a05a5678381f8d8fd17) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.136763 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.136970 9612 tablet_service.cc:1552] Processing DeleteTablet for tablet 07a9c3b7ef1548ec9d5ae9f20545f263 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.137132 9595 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b30052ff19744a9bf4bfaaa1bc77b36 with delete_type TABLET_DATA_DELETED (replaced by tablet 283b72f626644ac2b9b61186387b3839) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.137312 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0d3203d12fc4823b3b3b1689f6bde62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.139334 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c442d337f5f945c097fa71b3833654f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.140837 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fad3a2aa57f04f40ba65bf1b2832b9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.142334 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04c1f18b611d45c9ae2c7b895240f545. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.146616 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2044c73d56bf4434879dd65f35c5612b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.156168 9609 tablet_service.cc:1505] Processing CreateTablet for tablet aaccda3227f44a28a18d77f7bc7e6633 (DEFAULT_TABLE table=default.Table_c30777b91b944d75a8b8758a620faf1c [id=90dcef522b8045c1a5c9b8cd4a0a4228]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.157140 9568 leader_election.cc:304] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:09.158571 10029 raft_consensus.cc:359] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.158821 9612 tablet_service.cc:1552] Processing DeleteTablet for tablet f6789dc2f7cb4a3c826ac507d5c9a239 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.159266 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet 229ea4caa6af493cb5b3d593f22aa113 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.159461 9614 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d64a13667194c12864fa388eb88a380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.159591 9595 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c2d713d72fd4b5ca34cb3e828c75761 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.162734 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet b8c368698cfb40799571567e8bfdd8dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.163254 11283 raft_consensus.cc:515] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:09.163717 9920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07a9c3b7ef1548ec9d5ae9f20545f263" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:09.163875 11283 leader_election.cc:290] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.163913 9920 raft_consensus.cc:2468] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:09.164093 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 04c1f18b611d45c9ae2c7b895240f545 (DEFAULT_TABLE table=default.Table_1155ecb9419544b7b940e6699cb4464c [id=c1c73a49a8d149e9b129f11e8dfcd599]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.164474 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 956) took 4102 ms. Trace:
I20251027 21:08:09.164615 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet 7292bc6c22ca4ab5b8f53a69520833ba with delete_type TABLET_DATA_DELETED (replaced by tablet 37ae3b6bf590473e8e09092fc5f7df1b) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.164587 10043 rpcz_store.cc:276] 1027 21:08:05.028622 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.028681 (+ 59us) service_pool.cc:225] Handling call
1027 21:08:08.794404 (+3765723us) ts_tablet_manager.cc:1151] Deleting tablet 59b6818d29624da5a49dd1a7d20994c6
1027 21:08:08.794414 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.887885 (+ 93471us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:09.011364 (+123479us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:09.131475 (+120111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3765564}
I20251027 21:08:09.164934 10043 tablet_replica.cc:333] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:09.165141 10043 raft_consensus.cc:2243] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:09.165362 10043 raft_consensus.cc:2272] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:09.165946 10043 ts_tablet_manager.cc:1916] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:09.166877 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d64a13667194c12864fa388eb88a380" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:09.167030 9909 raft_consensus.cc:3060] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:09.167112 9568 leader_election.cc:304] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:09.167912 11142 raft_consensus.cc:2804] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:09.167997 11142 raft_consensus.cc:493] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:09.168084 11142 raft_consensus.cc:3060] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:09.168720 11097 leader_election.cc:290] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.169111 10029 raft_consensus.cc:385] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:09.169206 10029 raft_consensus.cc:740] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:09.169406 10029 consensus_queue.cc:260] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.169669 10859 raft_consensus.cc:2804] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:09.169826 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8766339bdedc42c2a41944b4928fa55e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.169874 9608 tablet_service.cc:1552] Processing DeleteTablet for tablet f1a931c36fec45029715b01754d0b161 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.170034 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet cb5ac0fff70b491e8c8842f7a331005e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.170202 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet 520f7142f8af4599ac4880b0692e3151 with delete_type TABLET_DATA_DELETED (replaced by tablet 781a162e17dd495b888f7e389c062013) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.170351 9598 tablet_service.cc:1552] Processing DeleteTablet for tablet 95446c4333424f10a71829ce32d857e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.171281 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f41ebd90a534b6a9ee8d0886641646a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.173274 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 918a4a19c64f4261b58bea03d1b8251e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.174844 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaccda3227f44a28a18d77f7bc7e6633. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.176476 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fad3a2aa57f04f40ba65bf1b2832b9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.179670 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c442d337f5f945c097fa71b3833654f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.182338 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04c1f18b611d45c9ae2c7b895240f545. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.185923 10029 ts_tablet_manager.cc:1434] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.147s user 0.000s sys 0.002s
I20251027 21:08:09.186326 10029 tablet_bootstrap.cc:492] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:09.188104 10029 tablet_bootstrap.cc:654] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:09.190634 10041 ts_tablet_manager.cc:1929] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:09.190809 10041 log.cc:1199] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/96a680e8f16343b1908829a67e960b60
I20251027 21:08:09.191018 10859 raft_consensus.cc:697] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:09.191215 10041 ts_tablet_manager.cc:1950] T 96a680e8f16343b1908829a67e960b60 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:09.191354 10859 consensus_queue.cc:237] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.197116 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 59b6818d29624da5a49dd1a7d20994c6 (table default.Table_9904948d307e479ea184d6dc441b5e61 [id=500c0fe58264475087ef89652bf83263]) successfully deleted
I20251027 21:08:09.210402 11254 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: acquiring CA information for follower catalog manager: success
I20251027 21:08:09.212864 11234 raft_consensus.cc:493] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.213019 11234 raft_consensus.cc:515] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.213630 11234 leader_election.cc:290] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.225142 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8dc358445526415d8cd4f1ac6779126d" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:09.225095 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76bee322d0b94abe9166170733608853" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:09.226094 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59300e16b3e148a08a217fc24d1e572f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:09.228040 9860 leader_election.cc:343] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:09.228488 9860 leader_election.cc:343] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:09.228921 9860 leader_election.cc:343] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.231778 11283 consensus_queue.cc:1048] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:09.236336 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3167089879c54e8ca0bd12fb186da304"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:09.236958 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3167089879c54e8ca0bd12fb186da304 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.240865 10032 tablet_bootstrap.cc:492] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:09.241009 10032 ts_tablet_manager.cc:1403] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.153s user 0.001s sys 0.012s
I20251027 21:08:09.242033 10032 raft_consensus.cc:359] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.242233 10032 raft_consensus.cc:385] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:09.242290 10032 raft_consensus.cc:740] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:09.242529 10032 consensus_queue.cc:260] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.250589 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 520f7142f8af4599ac4880b0692e3151 with delete_type TABLET_DATA_DELETED (replaced by tablet 781a162e17dd495b888f7e389c062013) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:09.256865 9909 raft_consensus.cc:2468] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:09.259433 9568 leader_election.cc:304] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:09.259786 11097 raft_consensus.cc:2804] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:09.260082 11097 raft_consensus.cc:697] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:09.260409 11097 consensus_queue.cc:237] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:09.268769 11142 raft_consensus.cc:515] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.271662 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07a9c3b7ef1548ec9d5ae9f20545f263" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:09.271834 9909 raft_consensus.cc:3060] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:09.273582 11142 leader_election.cc:290] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.274711 9909 raft_consensus.cc:2468] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:09.275231 9568 leader_election.cc:304] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:09.276202 11373 raft_consensus.cc:2804] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:09.277160 11283 raft_consensus.cc:493] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.277307 11283 raft_consensus.cc:515] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.277856 11283 leader_election.cc:290] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.278874 9907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d18ff6d8ddb465fbf479fdc7ebe7a05" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:09.279040 9907 raft_consensus.cc:2468] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:09.279564 9568 leader_election.cc:304] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:09.279820 11283 raft_consensus.cc:2804] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:09.279898 11283 raft_consensus.cc:493] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:09.279987 11283 raft_consensus.cc:3060] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:09.282641 11283 raft_consensus.cc:515] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.283571 11283 leader_election.cc:290] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.284142 9907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d18ff6d8ddb465fbf479fdc7ebe7a05" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:09.284294 9907 raft_consensus.cc:3060] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:09.287029 9907 raft_consensus.cc:2468] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:09.287598 9568 leader_election.cc:304] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:09.288462 11283 raft_consensus.cc:2804] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:09.288638 11283 raft_consensus.cc:697] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:09.288897 11283 consensus_queue.cc:237] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.299589 10859 consensus_queue.cc:1048] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:09.276376 11373 raft_consensus.cc:697] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:09.301918 11373 consensus_queue.cc:237] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.302311 10043 ts_tablet_manager.cc:1929] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:09.302443 10043 log.cc:1199] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/c9efffa9766e46aca8eb286fc2ad48d2
I20251027 21:08:09.302901 10043 ts_tablet_manager.cc:1950] T c9efffa9766e46aca8eb286fc2ad48d2 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:09.303777 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 957) took 4274 ms. Trace:
I20251027 21:08:09.303891 10043 rpcz_store.cc:276] 1027 21:08:05.029252 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.029300 (+ 48us) service_pool.cc:225] Handling call
1027 21:08:09.164848 (+4135548us) ts_tablet_manager.cc:1151] Deleting tablet c9efffa9766e46aca8eb286fc2ad48d2
1027 21:08:09.164856 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:09.256036 (+ 91180us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:09.302181 (+ 46145us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:09.303724 (+ 1543us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4135410}
I20251027 21:08:09.304239 10043 tablet_replica.cc:333] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:09.304462 10043 raft_consensus.cc:2243] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:09.306392 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet c9efffa9766e46aca8eb286fc2ad48d2 (table default.Table_97b01c1840c543838263ca4222695f54 [id=bea51a58fda64d40916f306c4aa8ab4e]) successfully deleted
I20251027 21:08:09.307781 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f41ebd90a534b6a9ee8d0886641646a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.310309 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d64a13667194c12864fa388eb88a380" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:09.319613 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d18ff6d8ddb465fbf479fdc7ebe7a05" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:09.320245 9772 raft_consensus.cc:3060] T 9fde4fd91943498d8888d1783005e492 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:09.324191 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet f1a931c36fec45029715b01754d0b161 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:09.327405 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 930) took 5047 ms. Trace:
I20251027 21:08:09.327490 10041 rpcz_store.cc:276] 1027 21:08:04.279540 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.279590 (+ 50us) service_pool.cc:225] Handling call
1027 21:08:08.958570 (+4678980us) ts_tablet_manager.cc:1151] Deleting tablet 96a680e8f16343b1908829a67e960b60
1027 21:08:08.958579 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:08.989565 (+ 30986us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:09.190511 (+200946us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:09.327382 (+136871us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65049,"tablet-delete.queue_time_us":4678823}
I20251027 21:08:09.328016 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 96a680e8f16343b1908829a67e960b60 (table default.Table_a46825cc888a4a0898d6db70b02e5ab6 [id=8579d3ff46d64c75b1da213fdee9cc1b]) successfully deleted
I20251027 21:08:09.308643 11254 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6e86ca5e83dd4df5b72f045e7b787e81: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251027 21:08:09.330174 10032 ts_tablet_manager.cc:1434] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.089s user 0.000s sys 0.003s
I20251027 21:08:09.331413 9310 master_service.cc:954] ReplaceTablet: received request to replace tablet 622fa1b3ff134f2d8a087ba13a85e597 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:09.332100 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5f6e0aaf15fe439c87306828e9a8cb1b" } new_table_name: "default.Table_020f4f2d75284ee0909dc0a9234113d8" modify_external_catalogs: true
I20251027 21:08:09.333580 9305 master_service.cc:954] ReplaceTablet: received request to replace tablet deb8b54e5e0f42bca4330d2160e5524a from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:09.304618 10043 raft_consensus.cc:2272] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:09.340792 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet b8c368698cfb40799571567e8bfdd8dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:09.334127 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_29fa885ad69b4de595605776e90aadde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:09.330538 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8dc358445526415d8cd4f1ac6779126d" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:09.341346 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29fa885ad69b4de595605776e90aadde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:09.342080 9859 leader_election.cc:343] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.342227 9859 leader_election.cc:304] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:09.342981 11234 raft_consensus.cc:2749] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:09.335122 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8b63d498ce8e4c3cafbefa650b657250 consensus state has no leader
I20251027 21:08:09.343436 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_df5f4de40f7f46728f38580d23124544" } modify_external_catalogs: true
I20251027 21:08:09.343649 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 931) took 5058 ms. Trace:
I20251027 21:08:09.343600 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_df5f4de40f7f46728f38580d23124544" } modify_external_catalogs: true
I20251027 21:08:09.343796 10041 rpcz_store.cc:276] 1027 21:08:04.285390 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.285449 (+ 59us) service_pool.cc:225] Handling call
1027 21:08:09.327891 (+5042442us) ts_tablet_manager.cc:1151] Deleting tablet c6a189aa22d048858e9f30bf4276859d
1027 21:08:09.343407 (+ 15516us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:09.343524 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5042285}
I20251027 21:08:09.344456 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 933) took 5030 ms. Trace:
I20251027 21:08:09.344596 10041 rpcz_store.cc:276] 1027 21:08:04.313784 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.313862 (+ 78us) service_pool.cc:225] Handling call
1027 21:08:09.344357 (+5030495us) ts_tablet_manager.cc:1151] Deleting tablet f9534e65d8bb4b3a88df934871792119
1027 21:08:09.344367 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:09.344441 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5030147}
I20251027 21:08:09.345075 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 934) took 4990 ms. Trace:
I20251027 21:08:09.345187 10041 rpcz_store.cc:276] 1027 21:08:04.354826 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.354877 (+ 51us) service_pool.cc:225] Handling call
1027 21:08:09.344980 (+4990103us) ts_tablet_manager.cc:1151] Deleting tablet 0ac1aed5f56844639b848f90c8eb5b6c
1027 21:08:09.344990 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:09.345061 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4989923}
I20251027 21:08:09.345639 10041 tablet_replica.cc:333] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:09.345870 10041 raft_consensus.cc:2243] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:09.346048 10041 raft_consensus.cc:2272] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:09.346732 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet c6a189aa22d048858e9f30bf4276859d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6a189aa22d048858e9f30bf4276859d already in progress: creating tablet
I20251027 21:08:09.336721 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_03f648dd39ab4c9cae71aa2c0121ced4" } new_table_name: "default.Table_45844e7e9f4947358be4c1ecdc66dc96" modify_external_catalogs: true
W20251027 21:08:09.346977 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet f9534e65d8bb4b3a88df934871792119 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9534e65d8bb4b3a88df934871792119 already in progress: creating tablet
I20251027 21:08:09.347038 11142 consensus_queue.cc:1048] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251027 21:08:09.347221 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 0ac1aed5f56844639b848f90c8eb5b6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ac1aed5f56844639b848f90c8eb5b6c already in progress: creating tablet
I20251027 21:08:09.338110 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4b21485c69fc40f180463005db09f2eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:09.347812 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b21485c69fc40f180463005db09f2eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.347998 9891 tablet_service.cc:1505] Processing CreateTablet for tablet ee90f271ec874109b233e9084e35a8f5 (DEFAULT_TABLE table=default.Table_5a743fc873a6414cab6c548e0041134c [id=3fec29f5687f4c91ae8f3a6439052d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.348387 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee90f271ec874109b233e9084e35a8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.339447 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f5602dbcd14b4fb5ae7190b6a0bd65bc" } modify_external_catalogs: true
I20251027 21:08:09.349629 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f5602dbcd14b4fb5ae7190b6a0bd65bc" } modify_external_catalogs: true
W20251027 21:08:09.350483 9568 consensus_peers.cc:597] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:09.350718 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07a9c3b7ef1548ec9d5ae9f20545f263" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:09.340051 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5443164d67cf445ca847b84c756635e6 consensus state has no leader
I20251027 21:08:09.351039 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07a9c3b7ef1548ec9d5ae9f20545f263" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:09.351154 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 03a02a8ec8d74ab6947a7c4b9098c692 (DEFAULT_TABLE table=default.Table_0feda3c71548490f8eae7b3f70073767 [id=ef03e107791849d598850239512c52d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.351293 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76bee322d0b94abe9166170733608853" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:09.351542 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03a02a8ec8d74ab6947a7c4b9098c692. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.351545 10040 ts_tablet_manager.cc:1929] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:09.351737 10040 log.cc:1199] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/32f80614e30d41128f9fb2e0ea165223
I20251027 21:08:09.351830 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59300e16b3e148a08a217fc24d1e572f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:09.353327 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 3a4fbbddfaff4d9280e8f750bbb0aab8 (DEFAULT_TABLE table=default.Table_fd9d066e8f9c42ee9c82bb11275d1b5c [id=e4bfa8656472457e8e1ac2bd5e1b3933]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:09.353509 9566 leader_election.cc:343] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.353741 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4fbbddfaff4d9280e8f750bbb0aab8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:09.353950 9566 leader_election.cc:343] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:09.354277 9566 leader_election.cc:343] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.354453 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_499a1282ff2b4f10918afa3d2c096d07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:09.354558 9566 leader_election.cc:343] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:09.355021 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_499a1282ff2b4f10918afa3d2c096d07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.355240 9903 tablet_service.cc:1505] Processing CreateTablet for tablet a2725a4587cf443f94fbd4829efb951a (DEFAULT_TABLE table=default.Table_c073b4449e4f49a1acb074b3130e3e3d [id=578b2028319843378c93b8e3f008b544]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.355355 11234 raft_consensus.cc:493] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.355585 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2725a4587cf443f94fbd4829efb951a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.355513 11234 raft_consensus.cc:515] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.355859 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 03a02a8ec8d74ab6947a7c4b9098c692 (DEFAULT_TABLE table=default.Table_0feda3c71548490f8eae7b3f70073767 [id=ef03e107791849d598850239512c52d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.356127 11234 leader_election.cc:290] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.356241 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03a02a8ec8d74ab6947a7c4b9098c692. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.359292 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 2da154dd3908414a98b53e450ff37994 (DEFAULT_TABLE table=default.Table_794b0b7aec3346f2bbcb4ad84958e797 [id=319376c04f1d4a999fc1c936910a886f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.359622 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da154dd3908414a98b53e450ff37994. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.336603 9312 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 59300e16b3e148a08a217fc24d1e572f (table default.Table_37d44f2ffd8b4816b4bff0b1960cd275 [id=a1dea3b74aa94995991d3123fbb2bf8d]). Expected version 1 got 0
I20251027 21:08:09.360715 9312 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet a9558d53008640739ac6fb45f1352fc0 (table default.Table_9821265dde764aee9571790dd10b4910 [id=c575fb3af44c46db8c198569241a425f]). Expected version 2 got 0
I20251027 21:08:09.360903 9312 catalog_manager.cc:5649] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67). New cstate: current_term: 1 leader_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:09.361279 9312 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 9fa39697360c46138ef4cd21e8c7a950 (table default.Table_b2a1b49649c54e5ab7a23fad7a157502 [id=434ed8c436c149ef9c7eb86ad252291d]). Expected version 4 got 0
I20251027 21:08:09.361457 9312 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet e6b58b1f4f63461aa96e416cae879be8 (table default.Table_f3c2de263e8849caaef87b570fc69459 [id=ef50ef2238804c98ad19c5b1c0b881c4]). Expected version 1 got 0
I20251027 21:08:09.361630 9312 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 93a3b22cdc1543aebcc7fe72f2f2c0ae (table default.Table_a04bcb2698ec467e8f85f4408ae2ca01 [id=8646c99b90414f4fa266cd7a31123d8e]). Expected version 5 got 0
I20251027 21:08:09.361951 9312 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 9be6b996df8145b9b0df1a2d2dc0ef03 (table default.Table_d09f1941d15f4517ac7561a66dd9b5e9 [id=b57564104d854b67a653c03de0650c09]). Expected version 1 got 0
I20251027 21:08:09.362002 9609 tablet_service.cc:1505] Processing CreateTablet for tablet ee90f271ec874109b233e9084e35a8f5 (DEFAULT_TABLE table=default.Table_5a743fc873a6414cab6c548e0041134c [id=3fec29f5687f4c91ae8f3a6439052d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.362236 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 17fcb982f4c84763bdaa449ab9682047 (DEFAULT_TABLE table=default.Table_10ff6b8aba2c40b2b4edd6466409d1f3 [id=78996749af464289a0d3b90640e75866]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.362371 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee90f271ec874109b233e9084e35a8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.362560 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17fcb982f4c84763bdaa449ab9682047. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:09.340265 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f543e8bcf70c4acab9880ee01af85562 consensus state has no leader
I20251027 21:08:09.363824 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8ebde3403af34f4589ee6222700b2283" } modify_external_catalogs: true
I20251027 21:08:09.363966 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8ebde3403af34f4589ee6222700b2283" } modify_external_catalogs: true
I20251027 21:08:09.351154 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 3a4fbbddfaff4d9280e8f750bbb0aab8 (DEFAULT_TABLE table=default.Table_fd9d066e8f9c42ee9c82bb11275d1b5c [id=e4bfa8656472457e8e1ac2bd5e1b3933]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.365171 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4fbbddfaff4d9280e8f750bbb0aab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.366343 9894 tablet_service.cc:1505] Processing CreateTablet for tablet e719aaba138842e58b4d46611bdf486d (DEFAULT_TABLE table=default.Table_19886f4eae9949fdb78d539336995823 [id=b3e91d7813774f8f855bc65c4fdb14cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.366461 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2d977e360b4ed29c9ed63a309783d6" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:09.366658 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e719aaba138842e58b4d46611bdf486d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:09.366895 9860 leader_election.cc:343] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.367354 10859 raft_consensus.cc:493] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.367437 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 17fcb982f4c84763bdaa449ab9682047 (DEFAULT_TABLE table=default.Table_10ff6b8aba2c40b2b4edd6466409d1f3 [id=78996749af464289a0d3b90640e75866]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.367542 10859 raft_consensus.cc:515] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.367805 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17fcb982f4c84763bdaa449ab9682047. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.368511 10859 leader_election.cc:290] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.368817 10041 ts_tablet_manager.cc:1916] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:09.369011 9915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "588e2f58f07b4a32a78dffd5b92ceb12" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:09.369189 9915 raft_consensus.cc:2468] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:09.369602 9568 leader_election.cc:304] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
W20251027 21:08:09.369920 9298 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 5443164d67cf445ca847b84c756635e6 (table default.Table_38143bba2f3d444aa0aa1eabb7bb1ff1 [id=62ba4f7526d74721a1fd3676a737fc01], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5443164d67cf445ca847b84c756635e6 consensus state has no leader
I20251027 21:08:09.370122 11097 raft_consensus.cc:2804] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:09.370253 11097 raft_consensus.cc:493] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:09.370375 11097 raft_consensus.cc:3060] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:09.372859 11097 raft_consensus.cc:515] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.373791 9605 tablet_service.cc:1505] Processing CreateTablet for tablet a2725a4587cf443f94fbd4829efb951a (DEFAULT_TABLE table=default.Table_c073b4449e4f49a1acb074b3130e3e3d [id=578b2028319843378c93b8e3f008b544]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.374203 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2725a4587cf443f94fbd4829efb951a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.375172 9902 tablet_service.cc:1505] Processing CreateTablet for tablet cf1751af4c774d078a4aede9d2bb8108 (DEFAULT_TABLE table=default.Table_6f18157ff71c4f37b28566ed2e696c74 [id=cf0b25c72b984df1bc0e139bfdb3f0b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.375501 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf1751af4c774d078a4aede9d2bb8108. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:09.378265 9566 consensus_peers.cc:597] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:09.379686 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 8d61afc9ce6b4b80b5e64259b9c04f85 (DEFAULT_TABLE table=default.Table_7fe12d37a07344b4b839c87a4b904c20 [id=71ff12fc440946e0b048072aa0c85c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.380004 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d61afc9ce6b4b80b5e64259b9c04f85. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:09.381027 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ae59aace74b48199219ef2252d4c7eb consensus state has no leader
I20251027 21:08:09.381340 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0668ac874959449eb530dd484f0c1d73" } modify_external_catalogs: true
I20251027 21:08:09.381434 9607 tablet_service.cc:1505] Processing CreateTablet for tablet cf1751af4c774d078a4aede9d2bb8108 (DEFAULT_TABLE table=default.Table_6f18157ff71c4f37b28566ed2e696c74 [id=cf0b25c72b984df1bc0e139bfdb3f0b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.381506 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0668ac874959449eb530dd484f0c1d73" } modify_external_catalogs: true
I20251027 21:08:09.381820 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf1751af4c774d078a4aede9d2bb8108. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.382890 10032 tablet_bootstrap.cc:492] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:09.386032 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 523d250f504e497094659e01666792f4 (DEFAULT_TABLE table=default.Table_3476d7c0b579463789510f1632676738 [id=1f8809a2c01a4b9eb498ef9cabe57a6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.386547 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 523d250f504e497094659e01666792f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.387275 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 8d61afc9ce6b4b80b5e64259b9c04f85 (DEFAULT_TABLE table=default.Table_7fe12d37a07344b4b839c87a4b904c20 [id=71ff12fc440946e0b048072aa0c85c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.387570 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d61afc9ce6b4b80b5e64259b9c04f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.394513 9603 tablet_service.cc:1505] Processing CreateTablet for tablet e719aaba138842e58b4d46611bdf486d (DEFAULT_TABLE table=default.Table_19886f4eae9949fdb78d539336995823 [id=b3e91d7813774f8f855bc65c4fdb14cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.394920 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e719aaba138842e58b4d46611bdf486d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.396682 10029 tablet_bootstrap.cc:492] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:09.396829 10029 ts_tablet_manager.cc:1403] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.211s user 0.007s sys 0.006s
I20251027 21:08:09.397691 10029 raft_consensus.cc:359] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.397876 10029 raft_consensus.cc:385] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:09.397958 10029 raft_consensus.cc:740] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:09.398237 10029 consensus_queue.cc:260] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.398846 10029 ts_tablet_manager.cc:1434] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:09.399154 10029 tablet_bootstrap.cc:492] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:09.402963 11234 raft_consensus.cc:493] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.403093 11234 raft_consensus.cc:515] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.403203 11097 leader_election.cc:290] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.403672 11234 leader_election.cc:290] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.404558 10032 tablet_bootstrap.cc:654] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:09.405484 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "588e2f58f07b4a32a78dffd5b92ceb12" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:09.405722 9909 raft_consensus.cc:3060] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:09.406183 9892 tablet_service.cc:1505] Processing CreateTablet for tablet a0a5f28107f143b09acc84ddfcf8425f (DEFAULT_TABLE table=default.Table_ab5de3d4fb7b41bf84ab54cbc5effb74 [id=49addd98987b451baf9532e9bb43dcae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.406531 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0a5f28107f143b09acc84ddfcf8425f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.408510 9909 raft_consensus.cc:2468] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:09.409758 9568 leader_election.cc:304] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:09.409761 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 61529c66ca42454ba65f0e4098e5e9eb (DEFAULT_TABLE table=default.Table_e74563f5a1ec4fbc9ddca17283f4bec2 [id=b013ad6eb8e249699e864a19d5f41db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.410960 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 60fc38dd62754475b790a7cd887488e2 (DEFAULT_TABLE table=default.Table_cd6e6c1476a64edbb448e788e60755fb [id=6ac49ff5642c4182a9bb9469fcd76961]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.411106 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_aa03fc53fae14398b298f577a92d8b5b" } modify_external_catalogs: true
I20251027 21:08:09.411314 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60fc38dd62754475b790a7cd887488e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.411283 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_aa03fc53fae14398b298f577a92d8b5b" } modify_external_catalogs: true
I20251027 21:08:09.414959 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 67009c2859ff46d988feb4e52ac8f47d (DEFAULT_TABLE table=default.Table_97d7c5edb9a54c508aadda3dcd8c1a99 [id=4b8c6674e0a948188fc893aac1413b2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.415334 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67009c2859ff46d988feb4e52ac8f47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.416661 9309 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet e6b58b1f4f63461aa96e416cae879be8 (table default.Table_f3c2de263e8849caaef87b570fc69459 [id=ef50ef2238804c98ad19c5b1c0b881c4]). Expected version 1 got 0
I20251027 21:08:09.410100 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61529c66ca42454ba65f0e4098e5e9eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.410655 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 67009c2859ff46d988feb4e52ac8f47d (DEFAULT_TABLE table=default.Table_97d7c5edb9a54c508aadda3dcd8c1a99 [id=4b8c6674e0a948188fc893aac1413b2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.418407 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67009c2859ff46d988feb4e52ac8f47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.418720 9602 tablet_service.cc:1505] Processing CreateTablet for tablet a0a5f28107f143b09acc84ddfcf8425f (DEFAULT_TABLE table=default.Table_ab5de3d4fb7b41bf84ab54cbc5effb74 [id=49addd98987b451baf9532e9bb43dcae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.419358 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0a5f28107f143b09acc84ddfcf8425f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.421180 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8691aa22fe84cf29893ed4e79d156cc" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:09.421789 9860 leader_election.cc:343] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.423117 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 918a4a19c64f4261b58bea03d1b8251e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:09.423744 9859 leader_election.cc:343] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.423854 9859 leader_election.cc:304] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:09.423987 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 60fc38dd62754475b790a7cd887488e2 (DEFAULT_TABLE table=default.Table_cd6e6c1476a64edbb448e788e60755fb [id=6ac49ff5642c4182a9bb9469fcd76961]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.424224 11234 raft_consensus.cc:2749] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:09.424355 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60fc38dd62754475b790a7cd887488e2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:09.424593 9859 leader_election.cc:343] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.429214 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 2da154dd3908414a98b53e450ff37994 (DEFAULT_TABLE table=default.Table_794b0b7aec3346f2bbcb4ad84958e797 [id=319376c04f1d4a999fc1c936910a886f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.429538 10040 ts_tablet_manager.cc:1950] T 32f80614e30d41128f9fb2e0ea165223 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:09.429620 11340 raft_consensus.cc:493] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.429775 11340 raft_consensus.cc:515] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.424683 9859 leader_election.cc:304] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:09.437034 11234 raft_consensus.cc:2749] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:09.443259 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a2eac284d1ea48ad9ecf74fa684dd769"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:09.443549 10041 ts_tablet_manager.cc:1929] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:09.443662 10041 log.cc:1199] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/fde0980aa64c4496861a662f7ed3ef75
W20251027 21:08:09.443841 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2eac284d1ea48ad9ecf74fa684dd769 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.443974 10041 ts_tablet_manager.cc:1950] T fde0980aa64c4496861a662f7ed3ef75 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:09.444446 9312 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_cf11a8c344c544c5ba54672fee38e4aa [id=71e02b21c9654e45a41d3e450ac3fbe1] alter complete (version 2)
I20251027 21:08:09.444828 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 937) took 5050 ms. Trace:
I20251027 21:08:09.444917 10041 rpcz_store.cc:276] 1027 21:08:04.394857 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.396143 (+ 1286us) service_pool.cc:225] Handling call
1027 21:08:09.345590 (+4949447us) ts_tablet_manager.cc:1151] Deleting tablet fde0980aa64c4496861a662f7ed3ef75
1027 21:08:09.345601 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:09.372160 (+ 26559us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:09.443423 (+ 71263us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:09.444805 (+ 1382us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4949254}
I20251027 21:08:09.445596 10029 tablet_bootstrap.cc:654] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:09.446223 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_34d1f3bf02564e879d5eb3aa607e98b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:09.446717 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34d1f3bf02564e879d5eb3aa607e98b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.449260 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet fde0980aa64c4496861a662f7ed3ef75 (table default.Table_096b7cf08dbf49279d8d5c11b3b687ab [id=20b788f5a2d14b31a58e1b29fa954951]) successfully deleted
I20251027 21:08:09.450456 11142 raft_consensus.cc:2804] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:09.450675 11142 raft_consensus.cc:697] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:09.451236 11142 consensus_queue.cc:237] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.453181 10859 raft_consensus.cc:493] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.453446 10859 raft_consensus.cc:515] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.455082 10859 leader_election.cc:290] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.454958 10041 tablet_replica.cc:333] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:09.456281 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 523d250f504e497094659e01666792f4 (DEFAULT_TABLE table=default.Table_3476d7c0b579463789510f1632676738 [id=1f8809a2c01a4b9eb498ef9cabe57a6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.460609 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 00c90ff84d3f4a7f92fda235416eed1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.461827 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d18ff6d8ddb465fbf479fdc7ebe7a05" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:09.462128 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "588e2f58f07b4a32a78dffd5b92ceb12" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:09.462416 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2d977e360b4ed29c9ed63a309783d6" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:09.462639 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "588e2f58f07b4a32a78dffd5b92ceb12" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:09.463052 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 523d250f504e497094659e01666792f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.463044 10041 raft_consensus.cc:2243] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251027 21:08:09.464035 9566 leader_election.cc:343] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.465525 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8691aa22fe84cf29893ed4e79d156cc" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:09.463094 11340 leader_election.cc:290] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:09.466308 10041 raft_consensus.cc:2272] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:09.469439 10041 ts_tablet_manager.cc:1916] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:09.470626 9750 tablet_service.cc:1505] Processing CreateTablet for tablet cf1751af4c774d078a4aede9d2bb8108 (DEFAULT_TABLE table=default.Table_6f18157ff71c4f37b28566ed2e696c74 [id=cf0b25c72b984df1bc0e139bfdb3f0b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.471130 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1043) took 2246 ms. Trace:
I20251027 21:08:09.471396 10040 rpcz_store.cc:276] 1027 21:08:07.185576 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.294729 (+109153us) service_pool.cc:225] Handling call
1027 21:08:08.910120 (+1615391us) ts_tablet_manager.cc:1151] Deleting tablet 32f80614e30d41128f9fb2e0ea165223
1027 21:08:08.910128 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:09.080967 (+170839us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:09.351389 (+270422us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:09.432160 (+ 80771us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1615285}
W20251027 21:08:09.472141 11370 log.cc:927] Time spent T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
I20251027 21:08:09.472918 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882c58a45d46402eabe96591a7c1de71" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:09.473161 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaccda3227f44a28a18d77f7bc7e6633. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.473210 10031 tablet_bootstrap.cc:492] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:09.473462 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf1751af4c774d078a4aede9d2bb8108. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.474624 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 523d250f504e497094659e01666792f4 (DEFAULT_TABLE table=default.Table_3476d7c0b579463789510f1632676738 [id=1f8809a2c01a4b9eb498ef9cabe57a6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.475473 9754 tablet_service.cc:1505] Processing CreateTablet for tablet a0a5f28107f143b09acc84ddfcf8425f (DEFAULT_TABLE table=default.Table_ab5de3d4fb7b41bf84ab54cbc5effb74 [id=49addd98987b451baf9532e9bb43dcae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.476593 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 784df834432244e8812d1802f9e1a1bc
I20251027 21:08:09.476593 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet ece28e82383540bea3457b1cbc65558b
I20251027 21:08:09.477133 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 67009c2859ff46d988feb4e52ac8f47d (DEFAULT_TABLE table=default.Table_97d7c5edb9a54c508aadda3dcd8c1a99 [id=4b8c6674e0a948188fc893aac1413b2f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:09.477306 9859 leader_election.cc:343] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.477412 9859 leader_election.cc:304] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:09.477861 11234 raft_consensus.cc:2749] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:09.478287 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 0e64d21abef342de86555e876f020787
W20251027 21:08:09.479183 9859 leader_election.cc:343] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.479300 9859 leader_election.cc:304] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:09.479518 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e24312671b4b4106a1790c61a5548a76
I20251027 21:08:09.479665 11234 raft_consensus.cc:2749] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:09.480275 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 32f80614e30d41128f9fb2e0ea165223 (table default.Table_a4ae12c512d645e58eadc2daa1bb7b0a [id=e1f4a77dfc0a497080c3c0ad4fe2e370]) successfully deleted
I20251027 21:08:09.481813 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet ece28e82383540bea3457b1cbc65558b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:09.482275 9738 tablet_service.cc:1505] Processing CreateTablet for tablet e719aaba138842e58b4d46611bdf486d (DEFAULT_TABLE table=default.Table_19886f4eae9949fdb78d539336995823 [id=b3e91d7813774f8f855bc65c4fdb14cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.485172 9753 tablet_service.cc:1505] Processing CreateTablet for tablet ee90f271ec874109b233e9084e35a8f5 (DEFAULT_TABLE table=default.Table_5a743fc873a6414cab6c548e0041134c [id=3fec29f5687f4c91ae8f3a6439052d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.485461 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 3a4fbbddfaff4d9280e8f750bbb0aab8 (DEFAULT_TABLE table=default.Table_fd9d066e8f9c42ee9c82bb11275d1b5c [id=e4bfa8656472457e8e1ac2bd5e1b3933]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.485992 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 17fcb982f4c84763bdaa449ab9682047 (DEFAULT_TABLE table=default.Table_10ff6b8aba2c40b2b4edd6466409d1f3 [id=78996749af464289a0d3b90640e75866]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.486320 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 03a02a8ec8d74ab6947a7c4b9098c692 (DEFAULT_TABLE table=default.Table_0feda3c71548490f8eae7b3f70073767 [id=ef03e107791849d598850239512c52d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.486596 9739 tablet_service.cc:1505] Processing CreateTablet for tablet 2da154dd3908414a98b53e450ff37994 (DEFAULT_TABLE table=default.Table_794b0b7aec3346f2bbcb4ad84958e797 [id=319376c04f1d4a999fc1c936910a886f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.499224 11234 raft_consensus.cc:493] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.499397 11234 raft_consensus.cc:515] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.503639 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet e24312671b4b4106a1790c61a5548a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.503118 9745 tablet_service.cc:1505] Processing CreateTablet for tablet a2725a4587cf443f94fbd4829efb951a (DEFAULT_TABLE table=default.Table_c073b4449e4f49a1acb074b3130e3e3d [id=578b2028319843378c93b8e3f008b544]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.504068 9611 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e64d21abef342de86555e876f020787 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.505015 10040 tablet_replica.cc:333] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:09.506192 10040 raft_consensus.cc:2243] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:09.507858 10040 raft_consensus.cc:2272] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:09.509183 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 952583fd7fde4b4baccdb98c01bdf9e1
I20251027 21:08:09.510646 10040 ts_tablet_manager.cc:1916] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:09.504276 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da154dd3908414a98b53e450ff37994. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.505061 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 784df834432244e8812d1802f9e1a1bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.518208 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94f2819dd0394530ae93754bdd1e482f" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:09.522054 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 8d61afc9ce6b4b80b5e64259b9c04f85 (DEFAULT_TABLE table=default.Table_7fe12d37a07344b4b839c87a4b904c20 [id=71ff12fc440946e0b048072aa0c85c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.522393 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 523d250f504e497094659e01666792f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.526218 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 60fc38dd62754475b790a7cd887488e2 (DEFAULT_TABLE table=default.Table_cd6e6c1476a64edbb448e788e60755fb [id=6ac49ff5642c4182a9bb9469fcd76961]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.526582 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60fc38dd62754475b790a7cd887488e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.529882 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 61529c66ca42454ba65f0e4098e5e9eb (DEFAULT_TABLE table=default.Table_e74563f5a1ec4fbc9ddca17283f4bec2 [id=b013ad6eb8e249699e864a19d5f41db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.530212 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61529c66ca42454ba65f0e4098e5e9eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.530385 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1050ac06e1e74d54ba4014b49cc84467. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.530660 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d61afc9ce6b4b80b5e64259b9c04f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.535732 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0d3203d12fc4823b3b3b1689f6bde62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.538282 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2044c73d56bf4434879dd65f35c5612b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.539173 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4fbbddfaff4d9280e8f750bbb0aab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.542177 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17fcb982f4c84763bdaa449ab9682047. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.542858 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04c1f18b611d45c9ae2c7b895240f545. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.545233 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0a5f28107f143b09acc84ddfcf8425f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.547451 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67009c2859ff46d988feb4e52ac8f47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.550879 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e719aaba138842e58b4d46611bdf486d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:09.513543 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f7fbffe7cd3c45d1b8fecec85f2bbf92 consensus state has no leader
I20251027 21:08:09.517031 9309 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet aaa8910d9fdb4146945621ec4ebf969e (table default.Table_d8da07f0725347338012d992737c5d04 [id=bb676562b97f48088767941239cfbf33]). Expected version 2 got 0
I20251027 21:08:09.553279 9309 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 9fde4fd91943498d8888d1783005e492 (table default.Table_87ac731a20bf4f6bbce8d82c1ad3bd97 [id=3a0c8b19df934d1580349b061fe0e8f5]). Expected version 1 got 0
I20251027 21:08:09.553540 9309 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 980eb6755aee4bcfb2ea44d84009021f (table default.Table_bb56319b6fe0405991a4d171477621e7 [id=42c8fedcabe0435dbaa11427141228ed]). Expected version 1 got 0
I20251027 21:08:09.554127 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03a02a8ec8d74ab6947a7c4b9098c692. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.508397 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 61529c66ca42454ba65f0e4098e5e9eb (DEFAULT_TABLE table=default.Table_e74563f5a1ec4fbc9ddca17283f4bec2 [id=b013ad6eb8e249699e864a19d5f41db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.554504 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61529c66ca42454ba65f0e4098e5e9eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.508875 10043 ts_tablet_manager.cc:1916] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:09.542742 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_be9e649e9750412b8350f0af57294c85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:09.544414 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f0938ced7d7147a8854fb3bb09a5bdbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:09.544368 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9b94589bffe54b1ab1cda98b7fbe273d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:09.544675 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_483a3db8dedc4021aa8b1430def3cc89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:09.544925 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4956622c69074c02b1659c0503f30547"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:09.545480 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_02661276431440f2a5613113911debd1" } new_table_name: "default.Table_7ab8221df3474d47b3bc2a3f1a934b84" modify_external_catalogs: true
I20251027 21:08:09.557499 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee90f271ec874109b233e9084e35a8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.558784 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_653e76a8cae24ff38c2eee3d9322aa6e" } new_table_name: "default.Table_50aedf7682e14774a74fc586b738034f" modify_external_catalogs: true
I20251027 21:08:09.561408 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 952583fd7fde4b4baccdb98c01bdf9e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:09.564530 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dbf46e4ce7714398b2c2019314e44578 consensus state has no leader
I20251027 21:08:09.568195 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c6bc5b44dd844823b76718c93f89a53d" } new_table_name: "default.Table_ae3b24400a3241a9b05e7c5fb016f8b5" modify_external_catalogs: true
I20251027 21:08:09.570518 9743 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e64d21abef342de86555e876f020787 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:09.570694 9743 tablet_service.cc:1552] Processing DeleteTablet for tablet e24312671b4b4106a1790c61a5548a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:08:09.571688 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b070d77b63ff4557883770df9ff4c55e consensus state has no leader
W20251027 21:08:09.575955 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e07797daedc47ee9dfb5daa36b15eef consensus state has no leader
I20251027 21:08:09.578332 10031 ts_tablet_manager.cc:1403] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.473s user 0.049s sys 0.063s
I20251027 21:08:09.580647 9920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94f2819dd0394530ae93754bdd1e482f" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
W20251027 21:08:09.581651 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b94589bffe54b1ab1cda98b7fbe273d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:09.582700 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4956622c69074c02b1659c0503f30547 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:09.583643 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be9e649e9750412b8350f0af57294c85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:09.584618 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_483a3db8dedc4021aa8b1430def3cc89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:09.585551 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0938ced7d7147a8854fb3bb09a5bdbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.587643 10031 raft_consensus.cc:359] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.587883 10031 raft_consensus.cc:385] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:09.587954 10031 raft_consensus.cc:740] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:09.588208 10031 consensus_queue.cc:260] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.588658 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4aae3c9852244881bf5286450425cf5d" } new_table_name: "default.Table_5030d0bdc78e4eb9979dbfc63446e63b" modify_external_catalogs: true
I20251027 21:08:09.588924 10031 ts_tablet_manager.cc:1434] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251027 21:08:09.589285 10031 tablet_bootstrap.cc:492] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:09.589377 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 784df834432244e8812d1802f9e1a1bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:09.590281 11177 raft_consensus.cc:493] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.593135 9309 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet aaa8910d9fdb4146945621ec4ebf969e (table default.Table_d8da07f0725347338012d992737c5d04 [id=bb676562b97f48088767941239cfbf33]). Expected version 2 got 0
W20251027 21:08:09.593153 9859 consensus_peers.cc:597] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:09.593374 9309 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 9fde4fd91943498d8888d1783005e492 (table default.Table_87ac731a20bf4f6bbce8d82c1ad3bd97 [id=3a0c8b19df934d1580349b061fe0e8f5]). Expected version 1 got 0
I20251027 21:08:09.593439 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet 952583fd7fde4b4baccdb98c01bdf9e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:09.594661 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet ece28e82383540bea3457b1cbc65558b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.596294 11234 leader_election.cc:290] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.596511 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet e10ae34eaaf3476594813de8e96c7e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.598258 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00c90ff84d3f4a7f92fda235416eed1d" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:09.598646 9859 leader_election.cc:343] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.601467 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00c90ff84d3f4a7f92fda235416eed1d" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:09.607875 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eeb667b3b10746dbad351c4c123ffd97 consensus state has no leader
I20251027 21:08:09.609692 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_88425ab3fcb841a383149d6747f24c4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:09.610249 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88425ab3fcb841a383149d6747f24c4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.590402 11177 raft_consensus.cc:515] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:09.602039 9860 leader_election.cc:343] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.610560 9860 leader_election.cc:304] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:09.610970 11177 leader_election.cc:290] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.611101 11177 raft_consensus.cc:2749] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:09.612103 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e64d21abef342de86555e876f020787" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:09.612486 9859 leader_election.cc:343] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.613791 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet 371aa8a57aee466ab01652d6061db4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.618079 9308 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 9fde4fd91943498d8888d1783005e492 (table default.Table_87ac731a20bf4f6bbce8d82c1ad3bd97 [id=3a0c8b19df934d1580349b061fe0e8f5]). Expected version 1 got 0
I20251027 21:08:09.618986 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e64d21abef342de86555e876f020787" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:09.619464 9860 leader_election.cc:343] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.619585 9860 leader_election.cc:304] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:09.619966 11234 raft_consensus.cc:2749] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:09.621806 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_27df2380f06a4b5aafc716e841c1b725"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:09.622373 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27df2380f06a4b5aafc716e841c1b725 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.624083 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ba13a89a33549af921d3b55023faf15 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.626371 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 04c579bb008448e9baa841135b6dbc61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:09.631213 11234 raft_consensus.cc:493] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.631338 11234 raft_consensus.cc:515] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:09.637075 11234 leader_election.cc:290] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.637665 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cb8ca3a50c148e6b602e949650353f8" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:09.638090 9859 leader_election.cc:343] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.639073 10029 tablet_bootstrap.cc:492] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:09.639194 10029 ts_tablet_manager.cc:1403] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.240s user 0.006s sys 0.008s
I20251027 21:08:09.639959 10029 raft_consensus.cc:359] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.640637 10029 raft_consensus.cc:385] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:09.640753 10029 raft_consensus.cc:740] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:09.641173 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d2d977e360b4ed29c9ed63a309783d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.641083 10029 consensus_queue.cc:260] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.643803 10041 ts_tablet_manager.cc:1929] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:09.643906 10041 log.cc:1199] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/1d64a13667194c12864fa388eb88a380
I20251027 21:08:09.644178 10041 ts_tablet_manager.cc:1950] T 1d64a13667194c12864fa388eb88a380 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:09.645313 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet ece28e82383540bea3457b1cbc65558b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.646539 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet 784df834432244e8812d1802f9e1a1bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.646785 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet e24312671b4b4106a1790c61a5548a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.647428 9894 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e64d21abef342de86555e876f020787 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.648139 11162 raft_consensus.cc:493] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.648254 11162 raft_consensus.cc:515] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.648825 11162 leader_election.cc:290] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.648926 11162 raft_consensus.cc:493] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.649012 11162 raft_consensus.cc:515] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.649461 11162 leader_election.cc:290] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.650070 10032 tablet_bootstrap.cc:492] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:09.650270 10032 ts_tablet_manager.cc:1403] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.267s user 0.004s sys 0.011s
I20251027 21:08:09.652014 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93a3b22cdc1543aebcc7fe72f2f2c0ae" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:09.652554 9859 leader_election.cc:343] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.652935 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42371496a7494e77a7bf1169de5b083a" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:09.654115 9859 leader_election.cc:343] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.661500 9311 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet f2710f265ae9485dafec389168328067 (table default.Table_eb44e5cb7bc24597a807d81c1207a162 [id=27d1cff8fc7940c9abb741c8c7d87e24]). Expected version 2 got 0
I20251027 21:08:09.661804 9311 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 90f96c21da9944f1ade05d35787d47e2 (table default.Table_cdf514bc48a4478da802d1bbfd0f1c71 [id=ec9aa8523bf44f2f9486b14f4cf7639c]). Expected version 2 got 0
I20251027 21:08:09.662617 9311 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet d672118ecd6f4094855adf106e5f7115 (table default.Table_1b066b58b2c649a2887ed91461f15d7c [id=887ea33c2fbe4573a1d7169dfd226567]). Expected version 2 got 0
I20251027 21:08:09.664650 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 588e2f58f07b4a32a78dffd5b92ceb12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.665148 9894 tablet_service.cc:1552] Processing DeleteTablet for tablet 952583fd7fde4b4baccdb98c01bdf9e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.666716 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42371496a7494e77a7bf1169de5b083a" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:09.666849 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 46d25c74d8414f549269c8105e6a2c99 (DEFAULT_TABLE table=default.Table_29fa885ad69b4de595605776e90aadde [id=3eb6510d05db47fabc5e5f6e2b877a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.666987 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93a3b22cdc1543aebcc7fe72f2f2c0ae" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:09.666716 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cb8ca3a50c148e6b602e949650353f8" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:09.667598 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46d25c74d8414f549269c8105e6a2c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.668015 10032 raft_consensus.cc:359] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.668233 10032 raft_consensus.cc:385] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:09.668289 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 939) took 5241 ms. Trace:
I20251027 21:08:09.668329 10032 raft_consensus.cc:740] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:09.668388 10041 rpcz_store.cc:276] 1027 21:08:04.426444 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.426512 (+ 68us) service_pool.cc:225] Handling call
1027 21:08:09.445200 (+5018688us) ts_tablet_manager.cc:1151] Deleting tablet 1d64a13667194c12864fa388eb88a380
1027 21:08:09.445212 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:09.580061 (+134849us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:09.643691 (+ 63630us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:09.668273 (+ 24582us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2947,"tablet-delete.queue_time_us":5018496}
I20251027 21:08:09.668617 10032 consensus_queue.cc:260] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.669413 10032 ts_tablet_manager.cc:1434] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20251027 21:08:09.669720 10032 tablet_bootstrap.cc:492] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:09.674600 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 945) took 5048 ms. Trace:
I20251027 21:08:09.674748 10041 rpcz_store.cc:276] 1027 21:08:04.626606 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.626665 (+ 59us) service_pool.cc:225] Handling call
1027 21:08:09.668743 (+5042078us) ts_tablet_manager.cc:1151] Deleting tablet 719321644ea3479ea4529a8a5d048754
1027 21:08:09.670706 (+ 1963us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:09.674579 (+ 3873us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5041918}
I20251027 21:08:09.675386 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 946) took 5045 ms. Trace:
I20251027 21:08:09.675529 10041 rpcz_store.cc:276] 1027 21:08:04.630023 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.630079 (+ 56us) service_pool.cc:225] Handling call
1027 21:08:09.675154 (+5045075us) ts_tablet_manager.cc:1151] Deleting tablet b83daa33ef0f45c6a78c70001e532ea2
1027 21:08:09.675284 (+ 130us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:09.675367 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5044867}
I20251027 21:08:09.676273 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 947) took 4889 ms. Trace:
I20251027 21:08:09.676404 10041 rpcz_store.cc:276] 1027 21:08:04.786751 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.786805 (+ 54us) service_pool.cc:225] Handling call
1027 21:08:09.675829 (+4889024us) ts_tablet_manager.cc:1151] Deleting tablet 1d1d4031a14e448ea5a60327bd37d003
1027 21:08:09.676173 (+ 344us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:09.676255 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4888844}
I20251027 21:08:09.677145 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 948) took 4883 ms. Trace:
W20251027 21:08:09.677440 9860 leader_election.cc:343] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.677873 9860 leader_election.cc:304] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:09.678356 9860 leader_election.cc:343] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.678444 9860 leader_election.cc:304] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:09.678733 9860 leader_election.cc:343] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.678831 9860 leader_election.cc:304] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:09.679014 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_58ec53ca465d4c2b82b50f300edba453" } modify_external_catalogs: true
I20251027 21:08:09.679214 11162 raft_consensus.cc:2749] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:09.679209 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_58ec53ca465d4c2b82b50f300edba453" } modify_external_catalogs: true
I20251027 21:08:09.679337 11162 raft_consensus.cc:2749] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:09.679426 11162 raft_consensus.cc:2749] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:09.679514 11162 consensus_queue.cc:1048] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:09.680045 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_00deabd570914e54b7fa9c4fc4eb02c7" } modify_external_catalogs: true
I20251027 21:08:09.680197 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_00deabd570914e54b7fa9c4fc4eb02c7" } modify_external_catalogs: true
I20251027 21:08:09.677269 10041 rpcz_store.cc:276] 1027 21:08:04.793853 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.793911 (+ 58us) service_pool.cc:225] Handling call
1027 21:08:09.676696 (+4882785us) ts_tablet_manager.cc:1151] Deleting tablet ee0c598904004dd18df058da483aa45c
1027 21:08:09.677057 (+ 361us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:09.677128 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4882619}
I20251027 21:08:09.680550 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f4d6cc291d944761aae3234c8ebdd6f1" } modify_external_catalogs: true
I20251027 21:08:09.680711 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f4d6cc291d944761aae3234c8ebdd6f1" } modify_external_catalogs: true
I20251027 21:08:09.682464 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_98fe733c6d594a04929bb89536b6c68f" } modify_external_catalogs: true
I20251027 21:08:09.682626 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_98fe733c6d594a04929bb89536b6c68f" } modify_external_catalogs: true
I20251027 21:08:09.683730 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_240c65b8e8b948eda2577237785b3507"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:09.689011 10041 tablet_replica.cc:333] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:09.689291 10041 raft_consensus.cc:2243] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:09.689535 10041 raft_consensus.cc:2272] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:09.690235 10041 ts_tablet_manager.cc:1916] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:09.691113 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_240c65b8e8b948eda2577237785b3507 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.692478 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da154dd3908414a98b53e450ff37994. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.693207 11234 raft_consensus.cc:493] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.693296 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 280ee23d80d64587b18811ff06340eee
I20251027 21:08:09.693869 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 2a9c6bf4f75144f487305520b3714b84
I20251027 21:08:09.695724 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2725a4587cf443f94fbd4829efb951a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.701936 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 280ee23d80d64587b18811ff06340eee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.693315 11234 raft_consensus.cc:515] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.702746 11234 leader_election.cc:290] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.703858 10031 tablet_bootstrap.cc:654] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:09.703924 9311 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet d672118ecd6f4094855adf106e5f7115 (table default.Table_1b066b58b2c649a2887ed91461f15d7c [id=887ea33c2fbe4573a1d7169dfd226567]). Expected version 2 got 0
I20251027 21:08:09.703975 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba13a89a33549af921d3b55023faf15" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:09.704149 9311 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 5cb8ca3a50c148e6b602e949650353f8 (table default.Table_b7607fa1c70d431d97f9e3f8a7c3d667 [id=080bf50609c045bd9b05f4e9144feac5]). Expected version 1 got 0
I20251027 21:08:09.704344 9311 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet f2710f265ae9485dafec389168328067 (table default.Table_eb44e5cb7bc24597a807d81c1207a162 [id=27d1cff8fc7940c9abb741c8c7d87e24]). Expected version 2 got 0
W20251027 21:08:09.704412 9859 leader_election.cc:343] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.704633 10029 ts_tablet_manager.cc:1434] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.065s user 0.001s sys 0.001s
I20251027 21:08:09.704953 10029 tablet_bootstrap.cc:492] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:09.705940 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a9c6bf4f75144f487305520b3714b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:09.706804 10029 tablet_bootstrap.cc:654] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:09.715252 10040 ts_tablet_manager.cc:1929] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:09.715350 10040 log.cc:1199] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/c9efffa9766e46aca8eb286fc2ad48d2
I20251027 21:08:09.715317 9920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882c58a45d46402eabe96591a7c1de71" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:09.715497 9920 raft_consensus.cc:2468] T 882c58a45d46402eabe96591a7c1de71 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:09.715693 10040 ts_tablet_manager.cc:1950] T c9efffa9766e46aca8eb286fc2ad48d2 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:09.715967 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1044) took 2460 ms. Trace:
I20251027 21:08:09.716048 10040 rpcz_store.cc:276] 1027 21:08:07.255101 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.294860 (+ 39759us) service_pool.cc:225] Handling call
1027 21:08:09.471957 (+2177097us) ts_tablet_manager.cc:1151] Deleting tablet c9efffa9766e46aca8eb286fc2ad48d2
1027 21:08:09.471989 (+ 32us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:09.700938 (+228949us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:09.715176 (+ 14238us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:09.715953 (+ 777us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2176998}
I20251027 21:08:09.716316 10040 tablet_replica.cc:333] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:09.716477 10040 raft_consensus.cc:2243] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:09.716794 10040 pending_rounds.cc:70] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
I20251027 21:08:09.716847 10040 pending_rounds.cc:77] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510678634565632 op_type: NO_OP noop_request { }
I20251027 21:08:09.716925 10040 raft_consensus.cc:2889] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:09.717012 10040 raft_consensus.cc:2272] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:09.717526 10040 ts_tablet_manager.cc:1916] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:09.720122 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 1d64a13667194c12864fa388eb88a380 (table default.Table_d11118d727fb4579a6465bd6deebaa43 [id=9df9c59afc184f9bb7494b154852d943]) successfully deleted
W20251027 21:08:09.721378 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 719321644ea3479ea4529a8a5d048754 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 719321644ea3479ea4529a8a5d048754 already in progress: creating tablet
W20251027 21:08:09.721884 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet b83daa33ef0f45c6a78c70001e532ea2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b83daa33ef0f45c6a78c70001e532ea2 already in progress: creating tablet
W20251027 21:08:09.722345 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 1d1d4031a14e448ea5a60327bd37d003 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d1d4031a14e448ea5a60327bd37d003 already in progress: creating tablet
W20251027 21:08:09.722756 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet ee0c598904004dd18df058da483aa45c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee0c598904004dd18df058da483aa45c already in progress: creating tablet
I20251027 21:08:09.724574 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 46d25c74d8414f549269c8105e6a2c99 (DEFAULT_TABLE table=default.Table_29fa885ad69b4de595605776e90aadde [id=3eb6510d05db47fabc5e5f6e2b877a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.725014 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet 280ee23d80d64587b18811ff06340eee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.725231 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a9c6bf4f75144f487305520b3714b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.725643 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 46d25c74d8414f549269c8105e6a2c99 (DEFAULT_TABLE table=default.Table_29fa885ad69b4de595605776e90aadde [id=3eb6510d05db47fabc5e5f6e2b877a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.726037 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 69097ed347dc4fe78640cfac973ed05e (DEFAULT_TABLE table=default.Table_e82b4a8ed42e40f29b491317173bd664 [id=707e6eace4754b31b159f179f06eb68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.728439 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7c87ac292e904db4af6b61b56d4ba295"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:09.728986 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c87ac292e904db4af6b61b56d4ba295 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.720377 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a9c6bf4f75144f487305520b3714b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.723368 9568 leader_election.cc:304] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:09.734699 11283 raft_consensus.cc:2804] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:09.734790 11283 raft_consensus.cc:493] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:09.734849 11283 raft_consensus.cc:3060] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:09.735884 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet bc387163e2d64532b87c32c5842a285a
I20251027 21:08:09.736536 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 53948dc94c334329bdf81f1eba94863c
I20251027 21:08:09.737564 11283 raft_consensus.cc:515] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.738564 11373 consensus_queue.cc:1048] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:09.739228 11373 consensus_queue.cc:1048] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:09.741549 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet bc387163e2d64532b87c32c5842a285a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.741923 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 53948dc94c334329bdf81f1eba94863c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.742538 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 9bc6e955bdd14bbc93f6a873c08ef6ba (DEFAULT_TABLE table=default.Table_27d4df68d5e7453295b814842d4089cd [id=585242e1adad4dbb9aaa1e107f5796aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.742918 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bc6e955bdd14bbc93f6a873c08ef6ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.744520 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet c9efffa9766e46aca8eb286fc2ad48d2 (table default.Table_97b01c1840c543838263ca4222695f54 [id=bea51a58fda64d40916f306c4aa8ab4e]) successfully deleted
I20251027 21:08:09.745386 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet bc387163e2d64532b87c32c5842a285a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:09.745729 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet 53948dc94c334329bdf81f1eba94863c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:09.746224 9745 tablet_service.cc:1505] Processing CreateTablet for tablet e22e1e3fb85b4a4a8d740c8ea5285aec (DEFAULT_TABLE table=default.Table_b42751988b8e445b9193301e2ef472e7 [id=ba9bd55cd9244527af50e12d097da1c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.738088 11283 leader_election.cc:290] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.746773 9739 tablet_service.cc:1505] Processing CreateTablet for tablet 2a9c6bf4f75144f487305520b3714b84 (DEFAULT_TABLE table=default.Table_58ec53ca465d4c2b82b50f300edba453 [id=a62b488fbbea44d8a5632724f1180259]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.747957 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 075934d4af2248d39a7652aa1af1c3e9 (DEFAULT_TABLE table=default.Table_4b21485c69fc40f180463005db09f2eb [id=b6d59513cfe54ec9abba69d9fb1ff9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.748288 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 075934d4af2248d39a7652aa1af1c3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.751799 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 78f9233dcd24498fb6a17ea81c527edb (DEFAULT_TABLE table=default.Table_3167089879c54e8ca0bd12fb186da304 [id=ca21287d9d3b413ca525d1d01adf0608]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.752207 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78f9233dcd24498fb6a17ea81c527edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.753173 9309 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 980eb6755aee4bcfb2ea44d84009021f (table default.Table_bb56319b6fe0405991a4d171477621e7 [id=42c8fedcabe0435dbaa11427141228ed]). Expected version 1 got 0
I20251027 21:08:09.754356 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9eb69e3772794bcf8f0662a43bed4bd1" } modify_external_catalogs: true
I20251027 21:08:09.754436 9598 tablet_service.cc:1505] Processing CreateTablet for tablet 0d0099e7b3bd403590ca5c9c73198853 (DEFAULT_TABLE table=default.Table_b431103b133e4f548240ad2a4b4f9110 [id=cfe76340b32e443e9fb84571e17644b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.754544 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9eb69e3772794bcf8f0662a43bed4bd1" } modify_external_catalogs: true
I20251027 21:08:09.754817 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0099e7b3bd403590ca5c9c73198853. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.758421 9604 tablet_service.cc:1505] Processing CreateTablet for tablet cf7b5a0b6cfd460da98558baf887093e (DEFAULT_TABLE table=default.Table_b1128d40cd914a9c9170d4d77fefa7cf [id=6618e796b8ba4aceb91e9dee70284281]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.758807 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf7b5a0b6cfd460da98558baf887093e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.761265 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet d042a15408484fbc9ee9666476b103d5
I20251027 21:08:09.761541 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba13a89a33549af921d3b55023faf15" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:09.762094 9860 leader_election.cc:343] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.762257 9860 leader_election.cc:304] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:09.762753 9608 tablet_service.cc:1552] Processing DeleteTablet for tablet d042a15408484fbc9ee9666476b103d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.762938 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_8d6356f22d714c699e68502a5cb1e71b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:09.762964 11234 consensus_queue.cc:1048] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251027 21:08:09.763438 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d6356f22d714c699e68502a5cb1e71b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.763592 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 781a162e17dd495b888f7e389c062013 (DEFAULT_TABLE table=default.Table_e6955caf15904f3bae744add8221897f [id=bd9615f2023e45688ae88cd7d323e61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.763923 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 781a162e17dd495b888f7e389c062013. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.764183 9747 tablet_service.cc:1505] Processing CreateTablet for tablet 69097ed347dc4fe78640cfac973ed05e (DEFAULT_TABLE table=default.Table_e82b4a8ed42e40f29b491317173bd664 [id=707e6eace4754b31b159f179f06eb68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.766265 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet b3abc5f78c4e41e29128e2d790319c75 with delete_type TABLET_DATA_DELETED (replaced by tablet ccd419ff40a94f9587b024ec87a2517e) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:09.767793 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_70172b5479b846bd884c62506727d518" } new_table_name: "default.Table_edf1a78f6fd44dcc9decf1dfe43840b4" modify_external_catalogs: true
W20251027 21:08:09.773916 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 54ad199e11dd46f4ac334cac76d070b4 consensus state has no leader
I20251027 21:08:09.774011 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e2e73046f34a4cb98d2a4cb5a48c4ca7" } new_table_name: "default.Table_335e44be85e54574ba1a2ba501ecc255" modify_external_catalogs: true
I20251027 21:08:09.776656 9751 tablet_service.cc:1505] Processing CreateTablet for tablet 9bc6e955bdd14bbc93f6a873c08ef6ba (DEFAULT_TABLE table=default.Table_27d4df68d5e7453295b814842d4089cd [id=585242e1adad4dbb9aaa1e107f5796aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.777233 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_42f66392d92c497db0755186643e6486" } new_table_name: "default.Table_33fa5ce10f0948348c85ec4d25f4535c" modify_external_catalogs: true
I20251027 21:08:09.784363 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f22296338b56467193b136c872ba30c4" } new_table_name: "default.Table_62364a3594a247078ee5736d6edf4a66" modify_external_catalogs: true
I20251027 21:08:09.785156 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 075934d4af2248d39a7652aa1af1c3e9 (DEFAULT_TABLE table=default.Table_4b21485c69fc40f180463005db09f2eb [id=b6d59513cfe54ec9abba69d9fb1ff9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.786341 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 78f9233dcd24498fb6a17ea81c527edb (DEFAULT_TABLE table=default.Table_3167089879c54e8ca0bd12fb186da304 [id=ca21287d9d3b413ca525d1d01adf0608]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.787536 11177 raft_consensus.cc:2749] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:09.788039 9744 tablet_service.cc:1552] Processing DeleteTablet for tablet 280ee23d80d64587b18811ff06340eee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:09.790117 11162 raft_consensus.cc:493] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.790263 11162 raft_consensus.cc:515] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.791112 10031 tablet_bootstrap.cc:492] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:09.791402 10031 ts_tablet_manager.cc:1403] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.202s user 0.003s sys 0.008s
I20251027 21:08:09.794224 9601 tablet_service.cc:1505] Processing CreateTablet for tablet f900efe8f5bc414fb54453a4d0ff4c18 (DEFAULT_TABLE table=default.Table_14d171d5f33e4baf9d9945618ae3e2d2 [id=187a98edf73e40d78c943bfdbb023189]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.794577 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f900efe8f5bc414fb54453a4d0ff4c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.795293 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9558d53008640739ac6fb45f1352fc0" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:09.795828 9860 leader_election.cc:343] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.723896 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 69097ed347dc4fe78640cfac973ed05e (DEFAULT_TABLE table=default.Table_e82b4a8ed42e40f29b491317173bd664 [id=707e6eace4754b31b159f179f06eb68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.798571 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69097ed347dc4fe78640cfac973ed05e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.799497 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a94891d2bdc54cdb84928a313c318421"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:09.800025 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a94891d2bdc54cdb84928a313c318421 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.800658 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 0d0099e7b3bd403590ca5c9c73198853 (DEFAULT_TABLE table=default.Table_b431103b133e4f548240ad2a4b4f9110 [id=cfe76340b32e443e9fb84571e17644b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.802500 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_67404e0500474119a04dbe1397e69f65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:09.803087 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67404e0500474119a04dbe1397e69f65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.803097 11234 raft_consensus.cc:493] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.803228 11234 raft_consensus.cc:515] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.803838 11234 leader_election.cc:290] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.804872 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 781a162e17dd495b888f7e389c062013 (DEFAULT_TABLE table=default.Table_e6955caf15904f3bae744add8221897f [id=bd9615f2023e45688ae88cd7d323e61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.805155 9741 tablet_service.cc:1552] Processing DeleteTablet for tablet d042a15408484fbc9ee9666476b103d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:08:09.805200 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0540fe4fe8744f79ee5e4d8d88f7436 consensus state has no leader
I20251027 21:08:09.805454 11234 raft_consensus.cc:493] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.805570 11234 raft_consensus.cc:515] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.806047 9755 tablet_service.cc:1505] Processing CreateTablet for tablet f900efe8f5bc414fb54453a4d0ff4c18 (DEFAULT_TABLE table=default.Table_14d171d5f33e4baf9d9945618ae3e2d2 [id=187a98edf73e40d78c943bfdbb023189]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.806221 11162 leader_election.cc:290] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.807580 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9be6b996df8145b9b0df1a2d2dc0ef03" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:09.809634 11142 raft_consensus.cc:493] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.809806 11142 raft_consensus.cc:515] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:09.810523 11142 leader_election.cc:290] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.812008 9744 tablet_service.cc:1505] Processing CreateTablet for tablet cf7b5a0b6cfd460da98558baf887093e (DEFAULT_TABLE table=default.Table_b1128d40cd914a9c9170d4d77fefa7cf [id=6618e796b8ba4aceb91e9dee70284281]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.806269 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 2a9c6bf4f75144f487305520b3714b84 (DEFAULT_TABLE table=default.Table_58ec53ca465d4c2b82b50f300edba453 [id=a62b488fbbea44d8a5632724f1180259]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.813840 11234 leader_election.cc:290] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.820452 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_23a1dab5b2d54b768282a93d7f1be443"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:09.820998 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23a1dab5b2d54b768282a93d7f1be443 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.822682 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_949c8006d42d47e09692d9b8e46045b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:09.822904 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_580942c54cc24c14b0849e1f7c209975" } new_table_name: "default.Table_42156f7dab5f4522aa9301297d4a10da" modify_external_catalogs: true
W20251027 21:08:09.823236 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_949c8006d42d47e09692d9b8e46045b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.824178 10041 ts_tablet_manager.cc:1929] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:09.825120 10041 log.cc:1199] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/ccf10ef559f74e2db28023fd520c8de9
I20251027 21:08:09.822682 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_bf4cbba7730e42ac8c88cc78b06da06d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:09.825773 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf4cbba7730e42ac8c88cc78b06da06d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:09.826177 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 25fbfcc28776483da266bb23c221f882 consensus state has no leader
I20251027 21:08:09.826174 11142 raft_consensus.cc:493] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.826308 11142 raft_consensus.cc:515] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.827008 11142 leader_election.cc:290] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.828778 10041 ts_tablet_manager.cc:1950] T ccf10ef559f74e2db28023fd520c8de9 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:09.826066 9609 tablet_service.cc:1505] Processing CreateTablet for tablet e22e1e3fb85b4a4a8d740c8ea5285aec (DEFAULT_TABLE table=default.Table_b42751988b8e445b9193301e2ef472e7 [id=ba9bd55cd9244527af50e12d097da1c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.829802 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e22e1e3fb85b4a4a8d740c8ea5285aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.831889 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_63b0ba7880994055983c8f60479f48df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:09.832336 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63b0ba7880994055983c8f60479f48df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:09.835907 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da23505b0fe742ddadfb829e37ca5499 consensus state has no leader
I20251027 21:08:09.836521 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69097ed347dc4fe78640cfac973ed05e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.836719 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a9c6bf4f75144f487305520b3714b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.837565 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9be6b996df8145b9b0df1a2d2dc0ef03" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:09.838033 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46d25c74d8414f549269c8105e6a2c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.820958 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46d25c74d8414f549269c8105e6a2c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.841470 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f900efe8f5bc414fb54453a4d0ff4c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.845278 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_345748a7ed204570983981e0356a3804"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:09.845826 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_345748a7ed204570983981e0356a3804 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.846405 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 781a162e17dd495b888f7e389c062013. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.847831 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 075934d4af2248d39a7652aa1af1c3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.850323 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78f9233dcd24498fb6a17ea81c527edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.859669 10032 tablet_bootstrap.cc:654] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:09.862636 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0099e7b3bd403590ca5c9c73198853. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.872879 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 949) took 5062 ms. Trace:
I20251027 21:08:09.873086 10041 rpcz_store.cc:276] 1027 21:08:04.810275 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.810531 (+ 256us) service_pool.cc:225] Handling call
1027 21:08:09.680348 (+4869817us) ts_tablet_manager.cc:1151] Deleting tablet ccf10ef559f74e2db28023fd520c8de9
1027 21:08:09.688963 (+ 8615us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:09.694801 (+ 5838us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:09.824046 (+129245us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:09.872855 (+ 48809us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4869626}
W20251027 21:08:09.873317 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cddb9e2e3a914c8ab4515cf4505333ee consensus state has no leader
I20251027 21:08:09.873670 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 950) took 4997 ms. Trace:
I20251027 21:08:09.873800 10041 rpcz_store.cc:276] 1027 21:08:04.876222 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:04.876283 (+ 61us) service_pool.cc:225] Handling call
1027 21:08:09.873548 (+4997265us) ts_tablet_manager.cc:1151] Deleting tablet 532ea47b15b544dfb05ddb2bdacf3953
1027 21:08:09.873559 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:09.873655 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4997075}
I20251027 21:08:09.724301 11392 tablet.cc:1722] T f2710f265ae9485dafec389168328067 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251027 21:08:09.729182 9566 leader_election.cc:343] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:09.876144 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 146974us
I20251027 21:08:09.876721 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bc6e955bdd14bbc93f6a873c08ef6ba. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:09.876902 9566 leader_election.cc:343] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:09.877316 9566 leader_election.cc:343] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.878778 10032 tablet_bootstrap.cc:492] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:09.878940 10032 ts_tablet_manager.cc:1403] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.209s user 0.006s sys 0.000s
I20251027 21:08:09.879685 10032 raft_consensus.cc:359] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.879907 10032 raft_consensus.cc:385] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:09.879989 10032 raft_consensus.cc:740] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:09.880281 10032 consensus_queue.cc:260] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.833364 9710 leader_election.cc:304] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:09.880575 11339 raft_consensus.cc:2749] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:09.880928 9710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.65:44531, user_credentials={real_user=slave}} blocked reactor thread for 47600.4us
I20251027 21:08:09.882373 9775 raft_consensus.cc:3060] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:09.893575 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e22e1e3fb85b4a4a8d740c8ea5285aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.895447 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882c58a45d46402eabe96591a7c1de71" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:09.895658 9909 raft_consensus.cc:3060] T 882c58a45d46402eabe96591a7c1de71 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:09.896385 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a9c6bf4f75144f487305520b3714b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.897914 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69097ed347dc4fe78640cfac973ed05e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.898854 9909 raft_consensus.cc:2468] T 882c58a45d46402eabe96591a7c1de71 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:09.902423 11396 tablet.cc:1722] T f2710f265ae9485dafec389168328067 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:09.904297 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf7b5a0b6cfd460da98558baf887093e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.904724 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8dce34af37324a659bd37f54cd0c379b" } modify_external_catalogs: true
I20251027 21:08:09.904951 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8dce34af37324a659bd37f54cd0c379b" } modify_external_catalogs: true
I20251027 21:08:09.905242 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7fe12d37a07344b4b839c87a4b904c20" } modify_external_catalogs: true
I20251027 21:08:09.905404 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7fe12d37a07344b4b839c87a4b904c20" } modify_external_catalogs: true
I20251027 21:08:09.906287 11405 tablet.cc:1722] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251027 21:08:09.907923 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882c58a45d46402eabe96591a7c1de71" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:09.909257 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cef2bfee222f45e6b6f20c6d63ff44d4" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:09.909730 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "420ccfaaa7f24ad0aea9fdfea24fc5dd" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:09.910981 9566 consensus_peers.cc:597] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:09.911295 9566 consensus_peers.cc:597] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:09.911597 9566 leader_election.cc:343] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.924263 9907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "420ccfaaa7f24ad0aea9fdfea24fc5dd" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:09.924436 9907 raft_consensus.cc:2468] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:09.925351 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet bc387163e2d64532b87c32c5842a285a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.926975 10032 ts_tablet_manager.cc:1434] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.048s user 0.001s sys 0.000s
I20251027 21:08:09.927316 10032 tablet_bootstrap.cc:492] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:09.928828 10032 tablet_bootstrap.cc:654] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:09.932919 9568 leader_election.cc:304] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:09.933058 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet ccf10ef559f74e2db28023fd520c8de9 (table default.Table_17e67d6acbdd470d9b351522b7908579 [id=bbdedeb45c464c47b96d078308141f9c]) successfully deleted
I20251027 21:08:09.933413 11283 raft_consensus.cc:2804] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:09.933548 11283 raft_consensus.cc:697] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:09.933774 11283 consensus_queue.cc:237] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.939894 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ef8a1ea1e15249948b532d0bd39d5bc8" } modify_external_catalogs: true
I20251027 21:08:09.940106 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ef8a1ea1e15249948b532d0bd39d5bc8" } modify_external_catalogs: true
I20251027 21:08:09.943624 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ad2a3d4173ca45948828915983300f00" } modify_external_catalogs: true
I20251027 21:08:09.943765 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ad2a3d4173ca45948828915983300f00" } modify_external_catalogs: true
I20251027 21:08:09.950659 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_158eb0214f8b455daffaa4f2b22fce78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:09.951205 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_158eb0214f8b455daffaa4f2b22fce78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.952332 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_56bd07e46cba4ff0a241b46adecf5fa3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:09.952808 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56bd07e46cba4ff0a241b46adecf5fa3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:09.929431 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 53948dc94c334329bdf81f1eba94863c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.929841 9898 tablet_service.cc:1505] Processing CreateTablet for tablet e22e1e3fb85b4a4a8d740c8ea5285aec (DEFAULT_TABLE table=default.Table_b42751988b8e445b9193301e2ef472e7 [id=ba9bd55cd9244527af50e12d097da1c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.959950 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e22e1e3fb85b4a4a8d740c8ea5285aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.930269 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 9bc6e955bdd14bbc93f6a873c08ef6ba (DEFAULT_TABLE table=default.Table_27d4df68d5e7453295b814842d4089cd [id=585242e1adad4dbb9aaa1e107f5796aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.963172 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bc6e955bdd14bbc93f6a873c08ef6ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.930675 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 075934d4af2248d39a7652aa1af1c3e9 (DEFAULT_TABLE table=default.Table_4b21485c69fc40f180463005db09f2eb [id=b6d59513cfe54ec9abba69d9fb1ff9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.931072 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 78f9233dcd24498fb6a17ea81c527edb (DEFAULT_TABLE table=default.Table_3167089879c54e8ca0bd12fb186da304 [id=ca21287d9d3b413ca525d1d01adf0608]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.966493 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78f9233dcd24498fb6a17ea81c527edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.931460 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 0d0099e7b3bd403590ca5c9c73198853 (DEFAULT_TABLE table=default.Table_b431103b133e4f548240ad2a4b4f9110 [id=cfe76340b32e443e9fb84571e17644b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.970674 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0099e7b3bd403590ca5c9c73198853. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.974002 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4956622c69074c02b1659c0503f30547" } modify_external_catalogs: true
I20251027 21:08:09.931576 9894 tablet_service.cc:1505] Processing CreateTablet for tablet cf7b5a0b6cfd460da98558baf887093e (DEFAULT_TABLE table=default.Table_b1128d40cd914a9c9170d4d77fefa7cf [id=6618e796b8ba4aceb91e9dee70284281]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.974200 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_4956622c69074c02b1659c0503f30547" } modify_external_catalogs: true
I20251027 21:08:09.974375 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf7b5a0b6cfd460da98558baf887093e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.931862 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet d042a15408484fbc9ee9666476b103d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:09.932267 9903 tablet_service.cc:1505] Processing CreateTablet for tablet 781a162e17dd495b888f7e389c062013 (DEFAULT_TABLE table=default.Table_e6955caf15904f3bae744add8221897f [id=bd9615f2023e45688ae88cd7d323e61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.933858 9899 tablet_service.cc:1505] Processing CreateTablet for tablet f900efe8f5bc414fb54453a4d0ff4c18 (DEFAULT_TABLE table=default.Table_14d171d5f33e4baf9d9945618ae3e2d2 [id=187a98edf73e40d78c943bfdbb023189]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.979009 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f900efe8f5bc414fb54453a4d0ff4c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.979651 9273 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu pid 9349
I20251027 21:08:09.981273 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_25435d2d53084e0aa737382bf0fa0e2c" } new_table_name: "default.Table_6b4342e6765145ceb4aae5006ab20f82" modify_external_catalogs: true
I20251027 21:08:09.981500 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 781a162e17dd495b888f7e389c062013. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.982323 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 075934d4af2248d39a7652aa1af1c3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:09.985596 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet befdd3a485a1426b8a3adf340d4eae4f
I20251027 21:08:09.986397 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 423047dfbf084bfcb881efeb0352f88d
I20251027 21:08:09.954031 11177 raft_consensus.cc:493] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.990661 11177 raft_consensus.cc:515] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.959008 11391 raft_consensus.cc:493] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:09.990976 11391 raft_consensus.cc:515] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:09.991848 11391 leader_election.cc:290] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.992610 11177 leader_election.cc:290] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:09.994405 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 8d61afc9ce6b4b80b5e64259b9c04f85
I20251027 21:08:09.994659 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9558d53008640739ac6fb45f1352fc0" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:09.995179 9859 leader_election.cc:343] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:09.995564 9859 consensus_peers.cc:597] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:09.996280 9859 leader_election.cc:343] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:09.996372 9859 leader_election.cc:304] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:09.997287 9751 tablet_service.cc:1505] Processing CreateTablet for tablet 103e1669203a45b387d9a086161a5d20 (DEFAULT_TABLE table=default.Table_88425ab3fcb841a383149d6747f24c4b [id=8bf361befef241be91cd9e051717975c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:09.997650 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 103e1669203a45b387d9a086161a5d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.003409 11177 raft_consensus.cc:2749] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:10.003939 10031 raft_consensus.cc:359] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.004164 10031 raft_consensus.cc:385] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:10.004267 10031 raft_consensus.cc:740] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:10.004489 10031 consensus_queue.cc:260] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.005014 10031 ts_tablet_manager.cc:1434] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.213s user 0.000s sys 0.002s
I20251027 21:08:10.005344 10031 tablet_bootstrap.cc:492] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:10.007616 10031 tablet_bootstrap.cc:654] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:10.008744 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c7d5f748f61a4201ad3c06b98ab1a958" } new_table_name: "default.Table_6c30fe76c38240b2b29a1269c6983a18" modify_external_catalogs: true
W20251027 21:08:10.019508 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d3a08a60c5e8461296ea8528eda90476 consensus state has no leader
W20251027 21:08:10.020905 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 83d6f067aef343ddaaa29b1a85e7f328 consensus state has no leader
I20251027 21:08:09.912510 10043 ts_tablet_manager.cc:1929] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:10.023198 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet ca14d51a66de4a0d8476117e0984d1e9
I20251027 21:08:10.024899 11373 raft_consensus.cc:493] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:10.025014 11373 raft_consensus.cc:515] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.026391 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d7fd481009d4474f9fa66bf669871624"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.026921 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7fd481009d4474f9fa66bf669871624 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:10.030038 9860 leader_election.cc:343] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.030186 9860 leader_election.cc:304] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:10.030781 11177 raft_consensus.cc:2749] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:10.021034 10043 log.cc:1199] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/04c579bb008448e9baa841135b6dbc61
I20251027 21:08:10.035264 10043 ts_tablet_manager.cc:1950] T 04c579bb008448e9baa841135b6dbc61 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:10.035319 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4bedb8cd144f4bb2854b9149070851a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.035954 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bedb8cd144f4bb2854b9149070851a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.035986 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 958) took 5006 ms. Trace:
I20251027 21:08:10.036062 10043 rpcz_store.cc:276] 1027 21:08:05.029623 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.029664 (+ 41us) service_pool.cc:225] Handling call
1027 21:08:09.304156 (+4274492us) ts_tablet_manager.cc:1151] Deleting tablet 04c579bb008448e9baa841135b6dbc61
1027 21:08:09.304164 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:09.573952 (+269788us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:09.912409 (+338457us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.035968 (+123559us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4274332}
I20251027 21:08:10.036365 10043 tablet_replica.cc:333] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:10.036533 10043 raft_consensus.cc:2243] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:10.036643 10043 raft_consensus.cc:2272] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:10.037078 10043 ts_tablet_manager.cc:1916] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:10.038295 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 2a9c6bf4f75144f487305520b3714b84 (DEFAULT_TABLE table=default.Table_58ec53ca465d4c2b82b50f300edba453 [id=a62b488fbbea44d8a5632724f1180259]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.038959 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a9c6bf4f75144f487305520b3714b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.050124 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_effb3e21eeb54ede9520b087b294612c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:09.914546 9566 leader_election.cc:343] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.024075 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e10ae34eaaf3476594813de8e96c7e77" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:10.050398 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 135861us
I20251027 21:08:10.025485 11373 leader_election.cc:290] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:10.050582 9566 leader_election.cc:343] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:10.050678 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_effb3e21eeb54ede9520b087b294612c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:10.050776 9566 consensus_peers.cc:597] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:10.050884 9566 consensus_peers.cc:597] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:10.051002 9568 leader_election.cc:343] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: cef2bfee222f45e6b6f20c6d63ff44d4
I20251027 21:08:10.051851 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_15aa76b39480416498545595ddf5cf1b" } new_table_name: "default.Table_a3198c3fe92746efbc0c218bb57042d8" modify_external_catalogs: true
I20251027 21:08:10.071530 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d3089b6fc3e64e108d93bdab2c49aaac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.072134 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3089b6fc3e64e108d93bdab2c49aaac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.074597 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_8e9ad7e433ba4538be8f764418dd70a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.075070 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e9ad7e433ba4538be8f764418dd70a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.075264 10043 ts_tablet_manager.cc:1929] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:10.075363 10043 log.cc:1199] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/bbb4116d1edd492893adfbf5fb3ddeef
I20251027 21:08:10.075649 10043 ts_tablet_manager.cc:1950] T bbb4116d1edd492893adfbf5fb3ddeef P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:10.076319 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 964) took 4855 ms. Trace:
I20251027 21:08:10.076392 10043 rpcz_store.cc:276] 1027 21:08:05.220622 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.220691 (+ 69us) service_pool.cc:225] Handling call
1027 21:08:10.036322 (+4815631us) ts_tablet_manager.cc:1151] Deleting tablet bbb4116d1edd492893adfbf5fb3ddeef
1027 21:08:10.036332 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.061785 (+ 25453us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.075168 (+ 13383us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.076302 (+ 1134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4815407}
I20251027 21:08:10.076649 10043 tablet_replica.cc:333] T d94c695320f24a7f9776c493e462ed24 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:10.076870 10043 raft_consensus.cc:2243] T d94c695320f24a7f9776c493e462ed24 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:10.077009 10043 raft_consensus.cc:2272] T d94c695320f24a7f9776c493e462ed24 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:10.077538 10043 ts_tablet_manager.cc:1916] T d94c695320f24a7f9776c493e462ed24 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:10.077916 9568 leader_election.cc:304] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:10.078224 9568 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 124609us
I20251027 21:08:10.078393 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_77b0da63638c4b85b08ba857ea63f5a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:10.078825 11373 raft_consensus.cc:2749] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:10.078928 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77b0da63638c4b85b08ba857ea63f5a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.079483 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 04c579bb008448e9baa841135b6dbc61 (table default.Table_fe308a2645c14e6c8efc7ce330b5add7 [id=73cba162a3af4c0fa5dfde81e1a81257]) successfully deleted
I20251027 21:08:10.079913 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet bbb4116d1edd492893adfbf5fb3ddeef (table default.Table_dfa3f51576fa4c689b2682b0e277df60 [id=088a35e69cc840c0b9386df033a885a8]) successfully deleted
I20251027 21:08:10.081624 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0c098e93345a4423888bb615af233dbe" } new_table_name: "default.Table_32bdd01af7424b478246a8ed1538480c" modify_external_catalogs: true
I20251027 21:08:10.082329 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_977d2d84338e4a73a6e069733e7157f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.082751 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_977d2d84338e4a73a6e069733e7157f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.083077 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3342d27362814c5da812391969037f20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.083490 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3342d27362814c5da812391969037f20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.084594 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_73d87eb46d0c436db67f3b5c9ccbb8db" } new_table_name: "default.Table_7d6b8d0ba5ad42fa9fdbcf13c7647d41" modify_external_catalogs: true
I20251027 21:08:10.110491 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f96c21da9944f1ade05d35787d47e2" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:10.110723 9632 raft_consensus.cc:2468] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 0.
I20251027 21:08:10.111140 9632 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3abc5f78c4e41e29128e2d790319c75" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:10.111277 9632 raft_consensus.cc:2468] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 0.
W20251027 21:08:10.112394 9860 leader_election.cc:343] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.112694 9860 leader_election.cc:304] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:10.112977 9860 leader_election.cc:304] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:10.113237 11391 raft_consensus.cc:2804] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:10.113310 11391 raft_consensus.cc:493] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:10.113359 11391 raft_consensus.cc:3060] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:10.116031 11391 raft_consensus.cc:515] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.116721 11391 leader_election.cc:290] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:10.116881 11391 raft_consensus.cc:2804] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:10.116945 11391 raft_consensus.cc:493] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:10.117018 11391 raft_consensus.cc:3060] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:10.119606 11391 raft_consensus.cc:515] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.120314 11391 leader_election.cc:290] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:10.121562 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f96c21da9944f1ade05d35787d47e2" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:10.121767 9909 raft_consensus.cc:2393] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51902b6b61ff4d5584ff21896e847e14 in current term 1: Already voted for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in this term.
I20251027 21:08:10.122350 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 2d343625a21344f7ad608aa560d3fb58 (DEFAULT_TABLE table=default.Table_7c87ac292e904db4af6b61b56d4ba295 [id=03e5cb0c32e54614be0afe06943da5d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.122737 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d343625a21344f7ad608aa560d3fb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.126621 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 103e1669203a45b387d9a086161a5d20 (DEFAULT_TABLE table=default.Table_88425ab3fcb841a383149d6747f24c4b [id=8bf361befef241be91cd9e051717975c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.126785 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 6ef4ab331ea04e33b53f8b32ea62830e (DEFAULT_TABLE table=default.Table_483a3db8dedc4021aa8b1430def3cc89 [id=b60c9db76a504778875915a009fbcf01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.126969 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 103e1669203a45b387d9a086161a5d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.127110 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ef4ab331ea04e33b53f8b32ea62830e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.133458 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 27753c58a3a44d9a901d20a8287b06b6 (DEFAULT_TABLE table=default.Table_34d1f3bf02564e879d5eb3aa607e98b3 [id=a603d31843fc4ab09703039a2c6b4306]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.133922 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27753c58a3a44d9a901d20a8287b06b6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:10.140929 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eaed784d81e74f9cb3514d3b55fdc0d1 consensus state has no leader
I20251027 21:08:10.142768 10032 tablet_bootstrap.cc:492] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:10.142999 10032 ts_tablet_manager.cc:1403] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.216s user 0.003s sys 0.007s
I20251027 21:08:10.143992 10032 raft_consensus.cc:359] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.144407 10032 raft_consensus.cc:385] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:10.144722 10032 raft_consensus.cc:740] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
W20251027 21:08:10.145088 9278 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 52828.6us
I20251027 21:08:10.145182 10032 consensus_queue.cc:260] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.146519 9413 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 21:08:10.158560 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f96c21da9944f1ade05d35787d47e2" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:10.158779 9630 raft_consensus.cc:3060] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:10.159524 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3abc5f78c4e41e29128e2d790319c75" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:10.159688 9631 raft_consensus.cc:3060] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:10.162885 10043 ts_tablet_manager.cc:1929] T d94c695320f24a7f9776c493e462ed24 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:10.163100 10043 log.cc:1199] T d94c695320f24a7f9776c493e462ed24 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/d94c695320f24a7f9776c493e462ed24
I20251027 21:08:10.163632 10043 ts_tablet_manager.cc:1950] T d94c695320f24a7f9776c493e462ed24 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
W20251027 21:08:10.165356 9566 consensus_peers.cc:597] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:10.171398 9894 tablet_service.cc:1505] Processing CreateTablet for tablet 1c71aff6f63847a6a1ca06f151be16e3 (DEFAULT_TABLE table=default.Table_27df2380f06a4b5aafc716e841c1b725 [id=c721f2e267fe4f26a58738e7c4a3d2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.172123 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c71aff6f63847a6a1ca06f151be16e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.172401 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 5e20fb964a4543ff9eda684acc8eb2bb (DEFAULT_TABLE table=default.Table_be9e649e9750412b8350f0af57294c85 [id=5e67a606c874479580d878a811bb1f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.172755 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e20fb964a4543ff9eda684acc8eb2bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.174439 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 61776cad160d49d48fdfff029ce3a1cc (DEFAULT_TABLE table=default.Table_a2eac284d1ea48ad9ecf74fa684dd769 [id=35ebd15e2cd04b3da335f8ef27f11b9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.174856 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61776cad160d49d48fdfff029ce3a1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.178498 9903 tablet_service.cc:1505] Processing CreateTablet for tablet 674520738b024e008dd41fa535c13a6e (DEFAULT_TABLE table=default.Table_f0938ced7d7147a8854fb3bb09a5bdbc [id=682b5ee5374f466e80710af78fc016e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.178608 9891 tablet_service.cc:1505] Processing CreateTablet for tablet b7e9c0bb583f438881c82f6ad308407e (DEFAULT_TABLE table=default.Table_499a1282ff2b4f10918afa3d2c096d07 [id=19eda38768d34dc0909d30ea6c553f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.178974 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 674520738b024e008dd41fa535c13a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.179186 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7e9c0bb583f438881c82f6ad308407e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.179452 9902 tablet_service.cc:1505] Processing CreateTablet for tablet ca14d51a66de4a0d8476117e0984d1e9 (DEFAULT_TABLE table=default.Table_4956622c69074c02b1659c0503f30547 [id=b8c4acbd59f84cf4be619ca9205ac877]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.179769 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca14d51a66de4a0d8476117e0984d1e9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:10.182896 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d715976e84754e31bfaf2cc9246342f4 consensus state has no leader
I20251027 21:08:10.184630 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet befdd3a485a1426b8a3adf340d4eae4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:10.185315 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 2b547446a11348669cc45e5b45a28c1d (DEFAULT_TABLE table=default.Table_240c65b8e8b948eda2577237785b3507 [id=32b951855708491d98d4e12d06b3c82a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.185758 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b547446a11348669cc45e5b45a28c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.186825 11391 raft_consensus.cc:493] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:10.186950 11391 raft_consensus.cc:515] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.188299 11391 leader_election.cc:290] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:10.189963 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_637357195ab4452dbc808d4d6a8d3c6c" } modify_external_catalogs: true
I20251027 21:08:10.190126 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_637357195ab4452dbc808d4d6a8d3c6c" } modify_external_catalogs: true
I20251027 21:08:10.190896 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_afddc510687a430dad9e6988b4cc2328"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.191359 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afddc510687a430dad9e6988b4cc2328 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.171398 9905 tablet_service.cc:1505] Processing CreateTablet for tablet d6377c8d469c4a39bc1789804486a3d2 (DEFAULT_TABLE table=default.Table_9b94589bffe54b1ab1cda98b7fbe273d [id=3ea2549000f4482ebeb7fb3941aced5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.202524 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6377c8d469c4a39bc1789804486a3d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.029999 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 2d343625a21344f7ad608aa560d3fb58 (DEFAULT_TABLE table=default.Table_7c87ac292e904db4af6b61b56d4ba295 [id=03e5cb0c32e54614be0afe06943da5d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.048132 9751 tablet_service.cc:1505] Processing CreateTablet for tablet 2b547446a11348669cc45e5b45a28c1d (DEFAULT_TABLE table=default.Table_240c65b8e8b948eda2577237785b3507 [id=32b951855708491d98d4e12d06b3c82a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.048637 9736 tablet_service.cc:1505] Processing CreateTablet for tablet d6377c8d469c4a39bc1789804486a3d2 (DEFAULT_TABLE table=default.Table_9b94589bffe54b1ab1cda98b7fbe273d [id=3ea2549000f4482ebeb7fb3941aced5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.048946 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 1c71aff6f63847a6a1ca06f151be16e3 (DEFAULT_TABLE table=default.Table_27df2380f06a4b5aafc716e841c1b725 [id=c721f2e267fe4f26a58738e7c4a3d2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.049201 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 61776cad160d49d48fdfff029ce3a1cc (DEFAULT_TABLE table=default.Table_a2eac284d1ea48ad9ecf74fa684dd769 [id=35ebd15e2cd04b3da335f8ef27f11b9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.203496 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b547446a11348669cc45e5b45a28c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.049502 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 674520738b024e008dd41fa535c13a6e (DEFAULT_TABLE table=default.Table_f0938ced7d7147a8854fb3bb09a5bdbc [id=682b5ee5374f466e80710af78fc016e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.086316 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f96c21da9944f1ade05d35787d47e2" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:10.203771 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 674520738b024e008dd41fa535c13a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.204522 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d343625a21344f7ad608aa560d3fb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.209995 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6377c8d469c4a39bc1789804486a3d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.089565 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 27753c58a3a44d9a901d20a8287b06b6 (DEFAULT_TABLE table=default.Table_34d1f3bf02564e879d5eb3aa607e98b3 [id=a603d31843fc4ab09703039a2c6b4306]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.210304 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61776cad160d49d48fdfff029ce3a1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.103588 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 5e20fb964a4543ff9eda684acc8eb2bb (DEFAULT_TABLE table=default.Table_be9e649e9750412b8350f0af57294c85 [id=5e67a606c874479580d878a811bb1f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.103739 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet 423047dfbf084bfcb881efeb0352f88d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:10.210676 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27753c58a3a44d9a901d20a8287b06b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.103876 9748 tablet_service.cc:1505] Processing CreateTablet for tablet b7e9c0bb583f438881c82f6ad308407e (DEFAULT_TABLE table=default.Table_499a1282ff2b4f10918afa3d2c096d07 [id=19eda38768d34dc0909d30ea6c553f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.210942 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7e9c0bb583f438881c82f6ad308407e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:10.211185 9566 leader_election.cc:343] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:10.212169 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57ef07c764664deaa4b2e9bcae7e7973 consensus state has no leader
I20251027 21:08:10.212755 10032 ts_tablet_manager.cc:1434] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.070s user 0.000s sys 0.002s
I20251027 21:08:10.104040 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet befdd3a485a1426b8a3adf340d4eae4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:10.104110 9746 tablet_service.cc:1552] Processing DeleteTablet for tablet ca14d51a66de4a0d8476117e0984d1e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:10.104143 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d61afc9ce6b4b80b5e64259b9c04f85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:10.104354 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 59b6818d29624da5a49dd1a7d20994c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:10.104507 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e10ae34eaaf3476594813de8e96c7e77" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:10.214596 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c71aff6f63847a6a1ca06f151be16e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.105077 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3abc5f78c4e41e29128e2d790319c75" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:10.105631 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f96c21da9944f1ade05d35787d47e2" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:10.217262 10032 tablet_bootstrap.cc:492] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:10.218319 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e20fb964a4543ff9eda684acc8eb2bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.218729 10032 tablet_bootstrap.cc:654] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:10.229326 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_04741665fd574abfada5d7d96a5fe910" } modify_external_catalogs: true
I20251027 21:08:10.229540 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_04741665fd574abfada5d7d96a5fe910" } modify_external_catalogs: true
I20251027 21:08:10.229908 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a48067b40324c79b1ee4eb431ce4a46" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:10.232542 9566 consensus_peers.cc:597] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:10.233037 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_98e35c35ab94438a8db3dbd106f9f265"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:10.233464 9631 raft_consensus.cc:2468] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 1.
W20251027 21:08:10.233646 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98e35c35ab94438a8db3dbd106f9f265 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:10.233927 9859 leader_election.cc:343] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.234050 9859 leader_election.cc:304] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:10.234328 10029 tablet_bootstrap.cc:492] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:10.234441 10029 ts_tablet_manager.cc:1403] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.530s user 0.014s sys 0.004s
I20251027 21:08:10.234441 11177 raft_consensus.cc:2749] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:10.234843 9860 leader_election.cc:343] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.235008 9860 leader_election.cc:304] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:10.236209 11177 raft_consensus.cc:2804] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:10.236450 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 963) took 5015 ms. Trace:
I20251027 21:08:10.236537 10043 rpcz_store.cc:276] 1027 21:08:05.220544 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.234783 (+ 14239us) service_pool.cc:225] Handling call
1027 21:08:10.076609 (+4841826us) ts_tablet_manager.cc:1151] Deleting tablet d94c695320f24a7f9776c493e462ed24
1027 21:08:10.076618 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.139727 (+ 63109us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.162756 (+ 23029us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.236431 (+ 73675us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4612700}
I20251027 21:08:10.236848 10043 tablet_replica.cc:333] T 83f5fbf5757a49df93ab949b7657c61f P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:10.236653 11177 raft_consensus.cc:697] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Becoming Leader. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Running, Role: LEADER
I20251027 21:08:10.237020 10043 raft_consensus.cc:2243] T 83f5fbf5757a49df93ab949b7657c61f P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251027 21:08:10.237112 9859 leader_election.cc:343] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.237157 10043 raft_consensus.cc:2272] T 83f5fbf5757a49df93ab949b7657c61f P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:10.237221 11177 consensus_queue.cc:237] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:10.237630 9859 leader_election.cc:343] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.238528 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a48067b40324c79b1ee4eb431ce4a46" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:10.238598 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f96c21da9944f1ade05d35787d47e2" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:10.238767 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_bb56319b6fe0405991a4d171477621e7" } modify_external_catalogs: true
I20251027 21:08:10.238914 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_bb56319b6fe0405991a4d171477621e7" } modify_external_catalogs: true
I20251027 21:08:10.238922 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3abc5f78c4e41e29128e2d790319c75" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:10.238952 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c6736558d45d4ba291581bb4b39a2208" } modify_external_catalogs: true
I20251027 21:08:10.239042 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c6736558d45d4ba291581bb4b39a2208" } modify_external_catalogs: true
W20251027 21:08:10.239555 9711 consensus_peers.cc:597] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbb4116d1edd492893adfbf5fb3ddeef. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:10.240370 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_59f5f9751881492e9c237e5926a96350" } new_table_name: "default.Table_a9013ed918dd437fa1348bb7e164ae01" modify_external_catalogs: true
I20251027 21:08:10.240561 9740 tablet_service.cc:1505] Processing CreateTablet for tablet d801c325bfeb4ef2b748b73b422c2036 (DEFAULT_TABLE table=default.Table_345748a7ed204570983981e0356a3804 [id=79df8aca58b14af7981c3debbdc03801]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.240658 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 1f370cb981aa43e9acf5104baca26653 (DEFAULT_TABLE table=default.Table_8d6356f22d714c699e68502a5cb1e71b [id=4ffd10f2ee814c85b55a18a9f76e6024]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.240909 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d801c325bfeb4ef2b748b73b422c2036. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.241060 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f370cb981aa43e9acf5104baca26653. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.241341 9752 tablet_service.cc:1505] Processing CreateTablet for tablet cb5e2c82ef0f482984e77e1e64f47d1f (DEFAULT_TABLE table=default.Table_949c8006d42d47e09692d9b8e46045b4 [id=4c91ccbecde2471b9fb188d548df1af4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.241643 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb5e2c82ef0f482984e77e1e64f47d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.241896 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_cc31649fd9f544e985ed00eb483164ea" } new_table_name: "default.Table_59b396d426544bb7800dacba4e8212f0" modify_external_catalogs: true
I20251027 21:08:10.242919 11400 raft_consensus.cc:493] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:10.243031 11400 raft_consensus.cc:515] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.243557 11400 leader_election.cc:290] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:10.246928 11391 raft_consensus.cc:493] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:10.247102 11391 raft_consensus.cc:515] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.247983 11391 leader_election.cc:290] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:10.247998 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95446c4333424f10a71829ce32d857e4" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:10.248327 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c67c5cb3b244feb8478f9594b834fbf" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:10.248744 9860 leader_election.cc:343] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:10.248936 9860 leader_election.cc:343] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.250886 9311 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_eb44e5cb7bc24597a807d81c1207a162 [id=27d1cff8fc7940c9abb741c8c7d87e24] alter complete (version 2)
I20251027 21:08:10.251446 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 6ef4ab331ea04e33b53f8b32ea62830e (DEFAULT_TABLE table=default.Table_483a3db8dedc4021aa8b1430def3cc89 [id=b60c9db76a504778875915a009fbcf01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.251798 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ef4ab331ea04e33b53f8b32ea62830e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.256594 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 6f590a2796b8403da2daa72ec2668b45 (DEFAULT_TABLE table=default.Table_63b0ba7880994055983c8f60479f48df [id=546edcd3299946e5a510c7be9224be31]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.257027 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f590a2796b8403da2daa72ec2668b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.257640 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 5cf3dd1b5f4b4b4ba112df78bf0df8f3 (DEFAULT_TABLE table=default.Table_a94891d2bdc54cdb84928a313c318421 [id=065a47e896814ef08b9e6ae660591380]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.257957 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf3dd1b5f4b4b4ba112df78bf0df8f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.258985 9748 tablet_service.cc:1505] Processing CreateTablet for tablet 6f590a2796b8403da2daa72ec2668b45 (DEFAULT_TABLE table=default.Table_63b0ba7880994055983c8f60479f48df [id=546edcd3299946e5a510c7be9224be31]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.258984 9894 tablet_service.cc:1505] Processing CreateTablet for tablet f042212dbf3748a291b720a8f6e942a0 (DEFAULT_TABLE table=default.Table_bf4cbba7730e42ac8c88cc78b06da06d [id=aa47aa694a494cc5b84d65e93f9f93b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.259287 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f590a2796b8403da2daa72ec2668b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.259279 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f042212dbf3748a291b720a8f6e942a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.262374 9903 tablet_service.cc:1552] Processing DeleteTablet for tablet ca14d51a66de4a0d8476117e0984d1e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:10.262853 9888 tablet_service.cc:1505] Processing CreateTablet for tablet e356ff5b06274740a4a118bcea2c996c (DEFAULT_TABLE table=default.Table_67404e0500474119a04dbe1397e69f65 [id=0402134098914826ba87086fd8514438]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.263242 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e356ff5b06274740a4a118bcea2c996c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.266311 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d61afc9ce6b4b80b5e64259b9c04f85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:10.274461 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 5cf3dd1b5f4b4b4ba112df78bf0df8f3 (DEFAULT_TABLE table=default.Table_a94891d2bdc54cdb84928a313c318421 [id=065a47e896814ef08b9e6ae660591380]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.274914 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf3dd1b5f4b4b4ba112df78bf0df8f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.144204 10040 ts_tablet_manager.cc:1929] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:10.280642 10040 log.cc:1199] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/67bb12de9f044c73ae9ab26d10498c90
I20251027 21:08:10.281034 10040 ts_tablet_manager.cc:1950] T 67bb12de9f044c73ae9ab26d10498c90 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:10.281056 9742 tablet_service.cc:1505] Processing CreateTablet for tablet f042212dbf3748a291b720a8f6e942a0 (DEFAULT_TABLE table=default.Table_bf4cbba7730e42ac8c88cc78b06da06d [id=aa47aa694a494cc5b84d65e93f9f93b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.281397 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1045) took 3026 ms. Trace:
I20251027 21:08:10.281483 10040 rpcz_store.cc:276] 1027 21:08:07.255230 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.294960 (+ 39730us) service_pool.cc:225] Handling call
1027 21:08:09.716274 (+2421314us) ts_tablet_manager.cc:1151] Deleting tablet 67bb12de9f044c73ae9ab26d10498c90
1027 21:08:09.716282 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.014457 (+298175us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.144095 (+129638us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.281381 (+137286us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2421207}
I20251027 21:08:10.281772 10040 tablet_replica.cc:333] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:10.171672 11317 raft_consensus.cc:493] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:10.281951 10040 raft_consensus.cc:2243] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:10.281993 11317 raft_consensus.cc:515] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.282312 10040 raft_consensus.cc:2272] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:10.282356 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f042212dbf3748a291b720a8f6e942a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.282608 11317 leader_election.cc:290] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:10.103039 9741 tablet_service.cc:1505] Processing CreateTablet for tablet ca14d51a66de4a0d8476117e0984d1e9 (DEFAULT_TABLE table=default.Table_4956622c69074c02b1659c0503f30547 [id=b8c4acbd59f84cf4be619ca9205ac877]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.282987 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca14d51a66de4a0d8476117e0984d1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.284111 9891 tablet_service.cc:1505] Processing CreateTablet for tablet d801c325bfeb4ef2b748b73b422c2036 (DEFAULT_TABLE table=default.Table_345748a7ed204570983981e0356a3804 [id=79df8aca58b14af7981c3debbdc03801]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.284446 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d801c325bfeb4ef2b748b73b422c2036. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.290638 9630 raft_consensus.cc:2468] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 1.
I20251027 21:08:10.291126 9860 leader_election.cc:304] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
W20251027 21:08:10.291499 9710 leader_election.cc:343] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 04c579bb008448e9baa841135b6dbc61
I20251027 21:08:10.291929 11391 raft_consensus.cc:2804] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:10.292145 11391 raft_consensus.cc:697] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Becoming Leader. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Running, Role: LEADER
I20251027 21:08:10.292434 11391 consensus_queue.cc:237] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.294500 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 2b3ce782c7044f04be019fa4558ad6fd (DEFAULT_TABLE table=default.Table_4bedb8cd144f4bb2854b9149070851a0 [id=821b273263194e088cd4829b1285a9d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.294731 9903 tablet_service.cc:1505] Processing CreateTablet for tablet 675f72e099be439b8d63101a40008cd7 (DEFAULT_TABLE table=default.Table_d7fd481009d4474f9fa66bf669871624 [id=ed7c4600c7ec4326ac0d51c69333e89b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.294878 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b3ce782c7044f04be019fa4558ad6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.295040 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 675f72e099be439b8d63101a40008cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.298273 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 1f370cb981aa43e9acf5104baca26653 (DEFAULT_TABLE table=default.Table_8d6356f22d714c699e68502a5cb1e71b [id=4ffd10f2ee814c85b55a18a9f76e6024]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.298766 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f370cb981aa43e9acf5104baca26653. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.301177 9902 tablet_service.cc:1505] Processing CreateTablet for tablet f1c2c72c99314d4eb5d5c752bdda4021 (DEFAULT_TABLE table=default.Table_158eb0214f8b455daffaa4f2b22fce78 [id=52a77f7f90d843249c3641f134d51ca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.301673 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1c2c72c99314d4eb5d5c752bdda4021. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.303977 9311 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet fe94124b239e442dbf644e4a483615fc (table default.Table_59f5f9751881492e9c237e5926a96350 [id=13f784ef7e0b4ede81d4d11e5e2a4138]). Expected version 1 got 0
I20251027 21:08:10.304005 9894 tablet_service.cc:1505] Processing CreateTablet for tablet 037afde2aeea4eddaa67b4ff69eaaad6 (DEFAULT_TABLE table=default.Table_23a1dab5b2d54b768282a93d7f1be443 [id=562d9ea4d5b94f8693374315bd45be27]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.304373 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 037afde2aeea4eddaa67b4ff69eaaad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.307281 9891 tablet_service.cc:1505] Processing CreateTablet for tablet cb5e2c82ef0f482984e77e1e64f47d1f (DEFAULT_TABLE table=default.Table_949c8006d42d47e09692d9b8e46045b4 [id=4c91ccbecde2471b9fb188d548df1af4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.307215 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b3acc4069419400c9cf22f7450e14d03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:10.307688 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb5e2c82ef0f482984e77e1e64f47d1f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:10.307796 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3acc4069419400c9cf22f7450e14d03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.309136 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_116d0f9e18e840409b5a038c264e39cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.309577 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_116d0f9e18e840409b5a038c264e39cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:10.310791 9710 consensus_peers.cc:597] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 -> Peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Couldn't send request to peer 51902b6b61ff4d5584ff21896e847e14. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbb4116d1edd492893adfbf5fb3ddeef. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:10.311014 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet 423047dfbf084bfcb881efeb0352f88d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:10.311616 9741 tablet_service.cc:1505] Processing CreateTablet for tablet 037afde2aeea4eddaa67b4ff69eaaad6 (DEFAULT_TABLE table=default.Table_23a1dab5b2d54b768282a93d7f1be443 [id=562d9ea4d5b94f8693374315bd45be27]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.312063 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 037afde2aeea4eddaa67b4ff69eaaad6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:10.321309 9859 leader_election.cc:343] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.321442 9859 leader_election.cc:304] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:10.321668 11177 raft_consensus.cc:2749] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:10.322011 9859 leader_election.cc:343] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.322273 10029 raft_consensus.cc:359] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.322458 10029 raft_consensus.cc:385] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251027 21:08:10.322456 9859 leader_election.cc:343] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.322502 10029 raft_consensus.cc:740] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:10.322674 10029 consensus_queue.cc:260] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.323022 10029 ts_tablet_manager.cc:1434] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.089s user 0.002s sys 0.000s
I20251027 21:08:10.323360 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 67bb12de9f044c73ae9ab26d10498c90 (table default.Table_0ce16ec9dc8b44eab80cedc074577cba [id=c22fee46f319480e88ab7745f6a95126]) successfully deleted
I20251027 21:08:10.323503 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c67c5cb3b244feb8478f9594b834fbf" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:10.323840 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95446c4333424f10a71829ce32d857e4" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:10.323951 9736 tablet_service.cc:1505] Processing CreateTablet for tablet f1c2c72c99314d4eb5d5c752bdda4021 (DEFAULT_TABLE table=default.Table_158eb0214f8b455daffaa4f2b22fce78 [id=52a77f7f90d843249c3641f134d51ca2]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:10.324393 9859 leader_election.cc:343] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.324481 9859 leader_election.cc:304] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:10.325012 9859 leader_election.cc:343] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.325114 9859 leader_election.cc:304] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:10.325170 11177 raft_consensus.cc:2749] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:10.325419 11177 raft_consensus.cc:2749] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:10.325693 9711 leader_election.cc:343] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 04c579bb008448e9baa841135b6dbc61
I20251027 21:08:10.332105 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 980eb6755aee4bcfb2ea44d84009021f
I20251027 21:08:10.334805 10029 tablet_bootstrap.cc:492] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:10.335937 10043 ts_tablet_manager.cc:1916] T 83f5fbf5757a49df93ab949b7657c61f P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:10.337857 9711 leader_election.cc:304] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:10.338232 11317 raft_consensus.cc:2749] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:10.326750 10031 tablet_bootstrap.cc:492] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:10.338624 9754 tablet_service.cc:1552] Processing DeleteTablet for tablet 980eb6755aee4bcfb2ea44d84009021f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:10.328336 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 2aafb2563ae34be58b4b3ee6635b8d7c (DEFAULT_TABLE table=default.Table_56bd07e46cba4ff0a241b46adecf5fa3 [id=a73bb5e84bfe427fb6eb83827d025a13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.338979 9741 tablet_service.cc:1505] Processing CreateTablet for tablet 2b3ce782c7044f04be019fa4558ad6fd (DEFAULT_TABLE table=default.Table_4bedb8cd144f4bb2854b9149070851a0 [id=821b273263194e088cd4829b1285a9d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.328660 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 675f72e099be439b8d63101a40008cd7 (DEFAULT_TABLE table=default.Table_d7fd481009d4474f9fa66bf669871624 [id=ed7c4600c7ec4326ac0d51c69333e89b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.327194 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1c2c72c99314d4eb5d5c752bdda4021. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.341395 9312 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 9fa39697360c46138ef4cd21e8c7a950 (table default.Table_b2a1b49649c54e5ab7a23fad7a157502 [id=434ed8c436c149ef9c7eb86ad252291d]). Expected version 4 got 0
I20251027 21:08:10.341730 9312 catalog_manager.cc:5649] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:10.342437 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aafb2563ae34be58b4b3ee6635b8d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.346691 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b3ce782c7044f04be019fa4558ad6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.348446 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 675f72e099be439b8d63101a40008cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.353281 10029 tablet_bootstrap.cc:654] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:10.330049 9734 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:40722 (request call id 160) took 1541 ms. Trace:
I20251027 21:08:10.353467 9734 rpcz_store.cc:276] 1027 21:08:08.680567 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.680714 (+ 147us) service_pool.cc:225] Handling call
1027 21:08:08.832700 (+151986us) spinlock_profiling.cc:243] Waited 24.8 ms on lock 0x616000137254. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d8fee1200 00007f8d93d9166f 00007f8d8e1b7c36 00007f8d8e1b7965 00007f8d8afe627e 00007f8d8afe515e 00007f8d8afeb395 00007f8d8afebcbf 00007f8d8afebae8 0000000000a8867c
1027 21:08:08.995649 (+162949us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x6160000c3d54. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d8fee1200 00007f8d93d9166f 00007f8d8e1b7c36 00007f8d8e1b7965 00007f8d8afe627e 00007f8d8afe515e 00007f8d8afeb395 00007f8d8afebcbf 00007f8d8afebae8 0000000000a8867c
1027 21:08:09.974415 (+978766us) spinlock_profiling.cc:243] Waited 81.3 ms on lock 0x6160001b2854. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d8fee1200 00007f8d93d9166f 00007f8d8e1b7c36 00007f8d8e1b7965 00007f8d8afe627e 00007f8d8afe515e 00007f8d8afeb395 00007f8d8afebcbf 00007f8d8afebae8 0000000000a8867c
1027 21:08:10.222125 (+247710us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":616993024}
I20251027 21:08:10.331756 9737 tablet_service.cc:1505] Processing CreateTablet for tablet e356ff5b06274740a4a118bcea2c996c (DEFAULT_TABLE table=default.Table_67404e0500474119a04dbe1397e69f65 [id=0402134098914826ba87086fd8514438]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.355664 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e356ff5b06274740a4a118bcea2c996c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.338671 10031 ts_tablet_manager.cc:1403] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.333s user 0.002s sys 0.003s
I20251027 21:08:10.357331 10031 raft_consensus.cc:359] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:10.357589 10031 raft_consensus.cc:385] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:10.357676 10031 raft_consensus.cc:740] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:10.357901 10031 consensus_queue.cc:260] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:10.335083 10040 ts_tablet_manager.cc:1916] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:10.362954 9566 consensus_peers.cc:597] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:10.363610 11392 tablet.cc:1722] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:08:10.367895 11405 tablet.cc:1722] T 9fde4fd91943498d8888d1783005e492 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:08:10.371670 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 2aafb2563ae34be58b4b3ee6635b8d7c (DEFAULT_TABLE table=default.Table_56bd07e46cba4ff0a241b46adecf5fa3 [id=a73bb5e84bfe427fb6eb83827d025a13]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:10.371762 9298 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 57ef07c764664deaa4b2e9bcae7e7973 (table default.Table_7d6b8d0ba5ad42fa9fdbcf13c7647d41 [id=a541c8d30a9540f5a7d4dd98a5d3739a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 57ef07c764664deaa4b2e9bcae7e7973 consensus state has no leader
I20251027 21:08:10.372051 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aafb2563ae34be58b4b3ee6635b8d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.372275 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 980eb6755aee4bcfb2ea44d84009021f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:10.373667 10029 tablet_bootstrap.cc:492] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:10.373804 10029 ts_tablet_manager.cc:1403] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20251027 21:08:10.374497 10029 raft_consensus.cc:359] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.374734 10029 raft_consensus.cc:385] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:10.374835 10029 raft_consensus.cc:740] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:10.375097 10029 consensus_queue.cc:260] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.375600 10029 ts_tablet_manager.cc:1434] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:10.375885 10029 tablet_bootstrap.cc:492] T f4187b2f66f24055b1df170f1062cc88 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:10.377545 10029 tablet_bootstrap.cc:654] T f4187b2f66f24055b1df170f1062cc88 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:10.378016 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 2d343625a21344f7ad608aa560d3fb58 (DEFAULT_TABLE table=default.Table_7c87ac292e904db4af6b61b56d4ba295 [id=03e5cb0c32e54614be0afe06943da5d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.386487 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d343625a21344f7ad608aa560d3fb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.378485 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 103e1669203a45b387d9a086161a5d20 (DEFAULT_TABLE table=default.Table_88425ab3fcb841a383149d6747f24c4b [id=8bf361befef241be91cd9e051717975c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.387598 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 103e1669203a45b387d9a086161a5d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.378897 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 6ef4ab331ea04e33b53f8b32ea62830e (DEFAULT_TABLE table=default.Table_483a3db8dedc4021aa8b1430def3cc89 [id=b60c9db76a504778875915a009fbcf01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.388916 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ef4ab331ea04e33b53f8b32ea62830e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.379362 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 27753c58a3a44d9a901d20a8287b06b6 (DEFAULT_TABLE table=default.Table_34d1f3bf02564e879d5eb3aa607e98b3 [id=a603d31843fc4ab09703039a2c6b4306]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.389608 11177 raft_consensus.cc:493] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:10.389703 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27753c58a3a44d9a901d20a8287b06b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.389755 11177 raft_consensus.cc:515] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.379410 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 2b547446a11348669cc45e5b45a28c1d (DEFAULT_TABLE table=default.Table_240c65b8e8b948eda2577237785b3507 [id=32b951855708491d98d4e12d06b3c82a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.390504 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b547446a11348669cc45e5b45a28c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.379439 9599 tablet_service.cc:1505] Processing CreateTablet for tablet d6377c8d469c4a39bc1789804486a3d2 (DEFAULT_TABLE table=default.Table_9b94589bffe54b1ab1cda98b7fbe273d [id=3ea2549000f4482ebeb7fb3941aced5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.391480 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6377c8d469c4a39bc1789804486a3d2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:10.410935 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe94124b239e442dbf644e4a483615fc consensus state has no leader
I20251027 21:08:10.413452 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_18c39792431548bf80997615fa3dd505"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.414040 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18c39792431548bf80997615fa3dd505 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.416138 9311 master_service.cc:954] ReplaceTablet: received request to replace tablet 824ff6913b0142d5889c75aa8e3cba3e from {username='slave'} at 127.0.0.1:41710
W20251027 21:08:10.416600 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4bc5fbdbbea0406a961e1a4df51b8762 consensus state has no leader
I20251027 21:08:10.416903 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b221c4f03f1e49d0aab44626822ea9ae" } new_table_name: "default.Table_401adedf60a04e5bb4caa5d680aecbd5" modify_external_catalogs: true
I20251027 21:08:10.429844 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 44804ae1c85749cb9286777e7b5c5707
I20251027 21:08:10.432466 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet befdd3a485a1426b8a3adf340d4eae4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:10.432543 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet 423047dfbf084bfcb881efeb0352f88d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:10.432771 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 674520738b024e008dd41fa535c13a6e (DEFAULT_TABLE table=default.Table_f0938ced7d7147a8854fb3bb09a5bdbc [id=682b5ee5374f466e80710af78fc016e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.433234 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 674520738b024e008dd41fa535c13a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.434394 9607 tablet_service.cc:1505] Processing CreateTablet for tablet ca14d51a66de4a0d8476117e0984d1e9 (DEFAULT_TABLE table=default.Table_4956622c69074c02b1659c0503f30547 [id=b8c4acbd59f84cf4be619ca9205ac877]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.434764 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca14d51a66de4a0d8476117e0984d1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.436661 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b0918d68920b4aa5b1fbcd706612d409"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.437180 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0918d68920b4aa5b1fbcd706612d409 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.438508 9601 tablet_service.cc:1505] Processing CreateTablet for tablet b7e9c0bb583f438881c82f6ad308407e (DEFAULT_TABLE table=default.Table_499a1282ff2b4f10918afa3d2c096d07 [id=19eda38768d34dc0909d30ea6c553f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.438910 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7e9c0bb583f438881c82f6ad308407e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.442457 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 61776cad160d49d48fdfff029ce3a1cc (DEFAULT_TABLE table=default.Table_a2eac284d1ea48ad9ecf74fa684dd769 [id=35ebd15e2cd04b3da335f8ef27f11b9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.442881 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61776cad160d49d48fdfff029ce3a1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.444065 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 93e8b973b39244be9de1107b14d6d93f
I20251027 21:08:10.446367 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 1c71aff6f63847a6a1ca06f151be16e3 (DEFAULT_TABLE table=default.Table_27df2380f06a4b5aafc716e841c1b725 [id=c721f2e267fe4f26a58738e7c4a3d2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.446739 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c71aff6f63847a6a1ca06f151be16e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.448729 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 5e20fb964a4543ff9eda684acc8eb2bb (DEFAULT_TABLE table=default.Table_be9e649e9750412b8350f0af57294c85 [id=5e67a606c874479580d878a811bb1f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.449159 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e20fb964a4543ff9eda684acc8eb2bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.449735 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_146c25bb53a14c978f7015defb5a6ae0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.450423 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_146c25bb53a14c978f7015defb5a6ae0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.459731 11142 raft_consensus.cc:493] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:10.459861 11142 raft_consensus.cc:515] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:10.460330 11142 leader_election.cc:290] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:10.464533 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c97cba61e08d42a7bdb3f51db0a63ec8
I20251027 21:08:10.465158 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01e9db9eaf8e41c6a2ee13b9ed127421" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:10.465371 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f3248ba254bd46baad5416e8b608e8c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.466058 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3248ba254bd46baad5416e8b608e8c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.466941 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_93bb0a51ff97439e8ebcb08268bd2462" } new_table_name: "default.Table_f24ffb28a47c46b7998af4df9cad118e" modify_external_catalogs: true
I20251027 21:08:10.471865 11177 leader_election.cc:290] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:10.472483 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7425b6b0184e4d0b871777b25d904ac4" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:10.472550 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d61afc9ce6b4b80b5e64259b9c04f85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:10.475498 9860 leader_election.cc:343] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.478921 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 44804ae1c85749cb9286777e7b5c5707 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:10.484843 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet c97cba61e08d42a7bdb3f51db0a63ec8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:10.485656 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_1f15260477b0409794ec2c672c472be3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.486191 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f15260477b0409794ec2c672c472be3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.490566 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet 93e8b973b39244be9de1107b14d6d93f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:10.491717 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet ca14d51a66de4a0d8476117e0984d1e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:10.492553 9603 tablet_service.cc:1505] Processing CreateTablet for tablet f042212dbf3748a291b720a8f6e942a0 (DEFAULT_TABLE table=default.Table_bf4cbba7730e42ac8c88cc78b06da06d [id=aa47aa694a494cc5b84d65e93f9f93b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.492899 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f042212dbf3748a291b720a8f6e942a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.500447 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 5cf3dd1b5f4b4b4ba112df78bf0df8f3 (DEFAULT_TABLE table=default.Table_a94891d2bdc54cdb84928a313c318421 [id=065a47e896814ef08b9e6ae660591380]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.500541 9603 tablet_service.cc:1505] Processing CreateTablet for tablet e356ff5b06274740a4a118bcea2c996c (DEFAULT_TABLE table=default.Table_67404e0500474119a04dbe1397e69f65 [id=0402134098914826ba87086fd8514438]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.500828 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf3dd1b5f4b4b4ba112df78bf0df8f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.501019 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e356ff5b06274740a4a118bcea2c996c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.501144 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 6f590a2796b8403da2daa72ec2668b45 (DEFAULT_TABLE table=default.Table_63b0ba7880994055983c8f60479f48df [id=546edcd3299946e5a510c7be9224be31]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.501539 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f590a2796b8403da2daa72ec2668b45. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:10.502983 9566 leader_election.cc:343] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.508769 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_10376e17301944fe9c799d6a87ff1f34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.509313 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10376e17301944fe9c799d6a87ff1f34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:10.509941 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24e6f57c27c743a2a260b52d264d92ea consensus state has no leader
I20251027 21:08:10.512583 9308 catalog_manager.cc:5649] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67). New cstate: current_term: 1 leader_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:10.514557 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_df2d8d09ba3147afb5c9271da5e23d45" } new_table_name: "default.Table_fc2bfcea6bae460a85d55717d23d3539" modify_external_catalogs: true
I20251027 21:08:10.516018 9308 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 90f96c21da9944f1ade05d35787d47e2 (table default.Table_cdf514bc48a4478da802d1bbfd0f1c71 [id=ec9aa8523bf44f2f9486b14f4cf7639c]). Expected version 2 got 0
I20251027 21:08:10.517096 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 824ff6913b0142d5889c75aa8e3cba3e
I20251027 21:08:10.517575 9311 catalog_manager.cc:6719] ReplaceTablet: tablet 824ff6913b0142d5889c75aa8e3cba3e deleted and replaced by tablet 107c608f9d904d86b74e2ad682e1bdc7
I20251027 21:08:10.518483 10032 tablet_bootstrap.cc:492] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:10.518648 10032 ts_tablet_manager.cc:1403] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.301s user 0.013s sys 0.018s
I20251027 21:08:10.519249 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet 824ff6913b0142d5889c75aa8e3cba3e with delete_type TABLET_DATA_DELETED (replaced by tablet 107c608f9d904d86b74e2ad682e1bdc7) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:10.519819 10032 raft_consensus.cc:359] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.520026 10032 raft_consensus.cc:385] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:10.520094 10032 raft_consensus.cc:740] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:10.520437 10032 consensus_queue.cc:260] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.521023 10032 ts_tablet_manager.cc:1434] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:10.521313 10032 tablet_bootstrap.cc:492] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:10.522228 9311 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 9fa39697360c46138ef4cd21e8c7a950 (table default.Table_b2a1b49649c54e5ab7a23fad7a157502 [id=434ed8c436c149ef9c7eb86ad252291d]). Expected version 4 got 0
I20251027 21:08:10.522559 9311 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet aaa8910d9fdb4146945621ec4ebf969e (table default.Table_d8da07f0725347338012d992737c5d04 [id=bb676562b97f48088767941239cfbf33]). Expected version 2 got 0
I20251027 21:08:10.522696 9311 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 67cb71ce43244d8ab9e71ead6bb315c9 (table default.Table_6327cacf4ed54d56b522291ad9594784 [id=3eb54d245ec64ee48022c541c2e49b17]). Expected version 2 got 0
I20251027 21:08:10.523083 10032 tablet_bootstrap.cc:654] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:10.524871 9309 master_service.cc:954] ReplaceTablet: received request to replace tablet c548d2413a5940c8b114ced667b4dc41 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:10.527746 9309 master_service.cc:954] ReplaceTablet: received request to replace tablet 9c2d713d72fd4b5ca34cb3e828c75761 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:10.529026 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_406cb517a4d34f0c921987c19227668b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.529542 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_406cb517a4d34f0c921987c19227668b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.536809 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2d8e45c40e9243faac5ee9568774bd62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.537326 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d8e45c40e9243faac5ee9568774bd62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:10.542644 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6fe4618b580745428d8809010d4e64fe consensus state has no leader
W20251027 21:08:10.542980 9566 consensus_peers.cc:597] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:10.545341 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_cb55ac1fcbad4075b84ec473bf89aa2b" } modify_external_catalogs: true
I20251027 21:08:10.545502 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_cb55ac1fcbad4075b84ec473bf89aa2b" } modify_external_catalogs: true
I20251027 21:08:10.547933 9602 tablet_service.cc:1505] Processing CreateTablet for tablet d801c325bfeb4ef2b748b73b422c2036 (DEFAULT_TABLE table=default.Table_345748a7ed204570983981e0356a3804 [id=79df8aca58b14af7981c3debbdc03801]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.548331 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d801c325bfeb4ef2b748b73b422c2036. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.558202 11373 raft_consensus.cc:493] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:10.558339 11373 raft_consensus.cc:515] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.558876 11373 leader_election.cc:290] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:10.567983 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 037afde2aeea4eddaa67b4ff69eaaad6 (DEFAULT_TABLE table=default.Table_23a1dab5b2d54b768282a93d7f1be443 [id=562d9ea4d5b94f8693374315bd45be27]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.568035 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 2b3ce782c7044f04be019fa4558ad6fd (DEFAULT_TABLE table=default.Table_4bedb8cd144f4bb2854b9149070851a0 [id=821b273263194e088cd4829b1285a9d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.568397 9604 tablet_service.cc:1505] Processing CreateTablet for tablet f1c2c72c99314d4eb5d5c752bdda4021 (DEFAULT_TABLE table=default.Table_158eb0214f8b455daffaa4f2b22fce78 [id=52a77f7f90d843249c3641f134d51ca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.568502 9595 tablet_service.cc:1552] Processing DeleteTablet for tablet c97cba61e08d42a7bdb3f51db0a63ec8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:10.568744 9598 tablet_service.cc:1505] Processing CreateTablet for tablet 675f72e099be439b8d63101a40008cd7 (DEFAULT_TABLE table=default.Table_d7fd481009d4474f9fa66bf669871624 [id=ed7c4600c7ec4326ac0d51c69333e89b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.569052 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 2aafb2563ae34be58b4b3ee6635b8d7c (DEFAULT_TABLE table=default.Table_56bd07e46cba4ff0a241b46adecf5fa3 [id=a73bb5e84bfe427fb6eb83827d025a13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.569248 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet 980eb6755aee4bcfb2ea44d84009021f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:10.569298 9595 tablet_service.cc:1552] Processing DeleteTablet for tablet 824ff6913b0142d5889c75aa8e3cba3e with delete_type TABLET_DATA_DELETED (replaced by tablet 107c608f9d904d86b74e2ad682e1bdc7) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:10.569401 9608 tablet_service.cc:1552] Processing DeleteTablet for tablet 44804ae1c85749cb9286777e7b5c5707 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:10.569550 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet 93e8b973b39244be9de1107b14d6d93f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:10.568464 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 037afde2aeea4eddaa67b4ff69eaaad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.570125 11373 consensus_queue.cc:1048] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:10.570626 9602 tablet_service.cc:1505] Processing CreateTablet for tablet cb5e2c82ef0f482984e77e1e64f47d1f (DEFAULT_TABLE table=default.Table_949c8006d42d47e09692d9b8e46045b4 [id=4c91ccbecde2471b9fb188d548df1af4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.570915 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb5e2c82ef0f482984e77e1e64f47d1f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:10.570921 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fdc215e390ca4b93bdee902f959843fb consensus state has no leader
I20251027 21:08:10.571262 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b3ce782c7044f04be019fa4558ad6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.574429 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aafb2563ae34be58b4b3ee6635b8d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.574985 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 675f72e099be439b8d63101a40008cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.575390 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1c2c72c99314d4eb5d5c752bdda4021. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.580436 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7425b6b0184e4d0b871777b25d904ac4" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:10.581180 10031 ts_tablet_manager.cc:1434] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.225s user 0.000s sys 0.002s
I20251027 21:08:10.581704 11317 raft_consensus.cc:493] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:10.581856 11317 raft_consensus.cc:515] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.587045 11317 leader_election.cc:290] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:10.587296 9311 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_87ac731a20bf4f6bbce8d82c1ad3bd97 [id=3a0c8b19df934d1580349b061fe0e8f5] alter complete (version 1)
I20251027 21:08:10.588156 9741 tablet_service.cc:1552] Processing DeleteTablet for tablet 44804ae1c85749cb9286777e7b5c5707 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:10.588343 9745 tablet_service.cc:1552] Processing DeleteTablet for tablet c97cba61e08d42a7bdb3f51db0a63ec8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:10.589910 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 93e8b973b39244be9de1107b14d6d93f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:08:10.590672 9859 leader_election.cc:343] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.590812 9859 leader_election.cc:304] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:10.592860 9748 tablet_service.cc:1552] Processing DeleteTablet for tablet 824ff6913b0142d5889c75aa8e3cba3e with delete_type TABLET_DATA_DELETED (replaced by tablet 107c608f9d904d86b74e2ad682e1bdc7) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:10.599069 9309 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 9fa39697360c46138ef4cd21e8c7a950 (table default.Table_b2a1b49649c54e5ab7a23fad7a157502 [id=434ed8c436c149ef9c7eb86ad252291d]). Expected version 4 got 0
I20251027 21:08:10.599323 11400 raft_consensus.cc:2749] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:10.600256 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet 9351a369dc1e4ac49f6ad419eab3a4c5 with delete_type TABLET_DATA_DELETED (replaced by tablet d5f4f3990da34f5e8f0385595f457427) from {username='slave'} at 127.0.0.1:34644
W20251027 21:08:10.610527 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 532ea47b15b544dfb05ddb2bdacf3953 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 532ea47b15b544dfb05ddb2bdacf3953 already in progress: creating tablet
I20251027 21:08:10.611024 10041 tablet_replica.cc:333] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:10.611363 10041 raft_consensus.cc:2243] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:10.611828 9568 leader_election.cc:304] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:10.612176 11373 raft_consensus.cc:2804] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:10.612262 11373 raft_consensus.cc:493] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:10.612327 11373 raft_consensus.cc:3060] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:10.612435 9568 leader_election.cc:304] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:10.613184 11097 raft_consensus.cc:2749] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:10.613257 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet d94c695320f24a7f9776c493e462ed24 (table default.Table_3da5a86d9f104d1ba39726b5a6dcec8a [id=da40d7d63ce348dd8a275cdd54f8f018]) successfully deleted
I20251027 21:08:10.620621 11373 raft_consensus.cc:515] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.621513 11373 leader_election.cc:290] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:10.611634 10041 raft_consensus.cc:2272] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:10.622404 10041 ts_tablet_manager.cc:1916] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:10.623059 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c3c17096bae64f378a84096c3b6f5ab5" } new_table_name: "default.Table_42823df1b61240fa9dc0e59703660dd5" modify_external_catalogs: true
I20251027 21:08:10.624595 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6b4342e6765145ceb4aae5006ab20f82" } new_table_name: "default.Table_e249dcc0941a494f85fe35df603156b3" modify_external_catalogs: true
I20251027 21:08:10.630571 10041 ts_tablet_manager.cc:1929] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:10.630963 10041 log.cc:1199] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/5d18ff6d8ddb465fbf479fdc7ebe7a05
I20251027 21:08:10.631402 10041 ts_tablet_manager.cc:1950] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:10.632387 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 958) took 5416 ms. Trace:
I20251027 21:08:10.632580 10041 rpcz_store.cc:276] 1027 21:08:05.215521 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.221039 (+ 5518us) service_pool.cc:225] Handling call
1027 21:08:10.610955 (+5389916us) ts_tablet_manager.cc:1151] Deleting tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05
1027 21:08:10.610971 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.626600 (+ 15629us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.630462 (+ 3862us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.632362 (+ 1900us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5389731}
I20251027 21:08:10.633208 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 (table default.Table_f99fba6d68834ef4b426a23a2ca11c9e [id=e6e865b72bbc4d01867f19308bdd343b]) successfully deleted
I20251027 21:08:10.633641 10041 tablet_replica.cc:333] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
W20251027 21:08:10.635828 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1a48067b40324c79b1ee4eb431ce4a46 consensus state has no leader
I20251027 21:08:10.636564 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f2e9f61a51bd4e1796f6bda56f48a6a1" } modify_external_catalogs: true
I20251027 21:08:10.636734 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f2e9f61a51bd4e1796f6bda56f48a6a1" } modify_external_catalogs: true
W20251027 21:08:10.637192 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d3a08a60c5e8461296ea8528eda90476 consensus state has no leader
I20251027 21:08:10.639871 10041 raft_consensus.cc:2243] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:10.640264 10041 raft_consensus.cc:2272] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:10.640981 10041 ts_tablet_manager.cc:1916] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:10.647964 10041 ts_tablet_manager.cc:1929] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:10.648144 10041 log.cc:1199] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/d94c695320f24a7f9776c493e462ed24
I20251027 21:08:10.648164 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 1f370cb981aa43e9acf5104baca26653 (DEFAULT_TABLE table=default.Table_8d6356f22d714c699e68502a5cb1e71b [id=4ffd10f2ee814c85b55a18a9f76e6024]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.648569 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f370cb981aa43e9acf5104baca26653. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.648593 10041 ts_tablet_manager.cc:1950] T d94c695320f24a7f9776c493e462ed24 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:10.652102 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet cef2bfee222f45e6b6f20c6d63ff44d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:08:10.656796 9711 leader_election.cc:343] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 59b6818d29624da5a49dd1a7d20994c6
W20251027 21:08:10.656984 9711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 63031.3us
W20251027 21:08:10.657720 9710 leader_election.cc:343] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 59b6818d29624da5a49dd1a7d20994c6
I20251027 21:08:10.658037 9710 leader_election.cc:304] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:10.662230 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b7607fa1c70d431d97f9e3f8a7c3d667" } modify_external_catalogs: true
I20251027 21:08:10.662411 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b7607fa1c70d431d97f9e3f8a7c3d667" } modify_external_catalogs: true
I20251027 21:08:10.665020 10031 tablet_bootstrap.cc:492] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:08:10.665429 9710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.65:44531, user_credentials={real_user=slave}} blocked reactor thread for 70293us
I20251027 21:08:10.665951 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67cb71ce43244d8ab9e71ead6bb315c9" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:10.666657 9566 leader_election.cc:343] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.667526 11317 raft_consensus.cc:2749] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:10.668334 10043 ts_tablet_manager.cc:1929] T 83f5fbf5757a49df93ab949b7657c61f P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:10.668447 10043 log.cc:1199] T 83f5fbf5757a49df93ab949b7657c61f P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/83f5fbf5757a49df93ab949b7657c61f
I20251027 21:08:10.668816 10043 ts_tablet_manager.cc:1950] T 83f5fbf5757a49df93ab949b7657c61f P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:10.669550 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1005) took 5095 ms. Trace:
I20251027 21:08:10.669637 10043 rpcz_store.cc:276] 1027 21:08:05.574493 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.277680 (+703187us) service_pool.cc:225] Handling call
1027 21:08:10.236804 (+3959124us) ts_tablet_manager.cc:1151] Deleting tablet 83f5fbf5757a49df93ab949b7657c61f
1027 21:08:10.236813 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.658909 (+422096us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.668244 (+ 9335us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.669533 (+ 1289us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3958948}
I20251027 21:08:10.669993 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1006) took 5095 ms. Trace:
I20251027 21:08:10.670068 10043 rpcz_store.cc:276] 1027 21:08:05.574581 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.277874 (+703293us) service_pool.cc:225] Handling call
1027 21:08:10.669890 (+4392016us) ts_tablet_manager.cc:1151] Deleting tablet 8649a3b611cc48ff8832c0fdeb128efe
1027 21:08:10.669899 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.669980 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4391888}
I20251027 21:08:10.670971 10043 tablet_replica.cc:333] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:10.671157 10043 raft_consensus.cc:2243] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:10.671427 10043 raft_consensus.cc:2272] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:10.672037 10043 ts_tablet_manager.cc:1916] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:10.675184 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "420ccfaaa7f24ad0aea9fdfea24fc5dd" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:10.675655 9566 leader_election.cc:343] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.678117 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e35dbbb1f48a4d2998d79cff38d71088" } modify_external_catalogs: true
I20251027 21:08:10.678309 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e35dbbb1f48a4d2998d79cff38d71088" } modify_external_catalogs: true
I20251027 21:08:10.682457 10031 tablet_bootstrap.cc:654] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:10.682883 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet f954daa93d5a42bfabfbf8fd4b72172b
I20251027 21:08:10.684092 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet f954daa93d5a42bfabfbf8fd4b72172b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:10.686913 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 957) took 5481 ms. Trace:
I20251027 21:08:10.687217 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet f954daa93d5a42bfabfbf8fd4b72172b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:10.692111 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0c303276539a43feb144ecfce1f3d172"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.692720 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c303276539a43feb144ecfce1f3d172 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.693373 9733 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:40722 (request call id 161) took 1640 ms. Trace:
I20251027 21:08:10.693470 9733 rpcz_store.cc:276] 1027 21:08:09.052467 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.052569 (+ 102us) service_pool.cc:225] Handling call
1027 21:08:09.940919 (+888350us) spinlock_profiling.cc:243] Waited 32.8 ms on lock 0x616000104f30. stack: 00007f8d840479f6 00007f8d840477b0 00007f8d830233ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f8d91269829 00007f8d93d91478 00007f8d8e1b7c36 00007f8d8e1b7965 00007f8d8afe627e 00007f8d8afe515e 00007f8d8afeb395 00007f8d8afebcbf 00007f8d8afebae8 0000000000a8867c
1027 21:08:10.693344 (+752425us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":75428608}
I20251027 21:08:10.694229 10041 rpcz_store.cc:276] 1027 21:08:05.205518 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:05.234621 (+ 29103us) service_pool.cc:225] Handling call
1027 21:08:10.633592 (+5398971us) ts_tablet_manager.cc:1151] Deleting tablet d94c695320f24a7f9776c493e462ed24
1027 21:08:10.633603 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.644661 (+ 11058us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.647881 (+ 3220us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.686891 (+ 39010us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5936,"tablet-delete.queue_time_us":5171321}
I20251027 21:08:10.694762 10041 tablet_replica.cc:333] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:10.694907 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet d94c695320f24a7f9776c493e462ed24 (table default.Table_3da5a86d9f104d1ba39726b5a6dcec8a [id=da40d7d63ce348dd8a275cdd54f8f018]) successfully deleted
I20251027 21:08:10.695003 10041 raft_consensus.cc:2243] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:10.695307 10041 raft_consensus.cc:2272] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:10.696295 10041 ts_tablet_manager.cc:1916] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:10.691197 10032 tablet_bootstrap.cc:492] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:10.691990 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67cb71ce43244d8ab9e71ead6bb315c9" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:10.697398 9925 raft_consensus.cc:2468] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:10.697877 10032 ts_tablet_manager.cc:1403] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.177s user 0.003s sys 0.000s
I20251027 21:08:10.698974 10032 raft_consensus.cc:359] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.714576 11142 raft_consensus.cc:493] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:10.714745 11142 raft_consensus.cc:515] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:10.715327 11142 leader_election.cc:290] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:10.715420 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_52c44a2a153c46ec823f80c2c4fbb600" } new_table_name: "default.Table_3e82113b8b1c4fba9d5124d41550bdac" modify_external_catalogs: true
I20251027 21:08:10.715865 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a23d9519c070420782525d2a7397288c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.716331 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a23d9519c070420782525d2a7397288c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.717633 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet cb95618d9fd54a4f8121b43e717d5616 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:10.724596 11419 raft_consensus.cc:493] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:10.724749 11419 raft_consensus.cc:515] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:10.725332 11419 leader_election.cc:290] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:10.725914 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f775a7e42bdc4910bdb604b26a28a7f4" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:10.727327 9566 leader_election.cc:343] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.727658 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet f954daa93d5a42bfabfbf8fd4b72172b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:10.727718 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76e4fb2bea7f4a4696fe9f6bb892bccc" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:10.728243 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc1cb2149b5f49e8a1f78d820a74cea7 consensus state has no leader
W20251027 21:08:10.728238 9566 leader_election.cc:343] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.728994 10041 ts_tablet_manager.cc:1929] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:10.741262 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_aef78f959ff04f84a9f5d8d736c7f596"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:10.741348 10032 raft_consensus.cc:385] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:10.741407 10041 log.cc:1199] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/83f5fbf5757a49df93ab949b7657c61f
I20251027 21:08:10.741447 10032 raft_consensus.cc:740] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:10.741816 10041 ts_tablet_manager.cc:1950] T 83f5fbf5757a49df93ab949b7657c61f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
W20251027 21:08:10.741825 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aef78f959ff04f84a9f5d8d736c7f596 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.742642 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1001) took 5163 ms. Trace:
I20251027 21:08:10.742729 10041 rpcz_store.cc:276] 1027 21:08:05.579110 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.273581 (+694471us) service_pool.cc:225] Handling call
1027 21:08:10.694711 (+4421130us) ts_tablet_manager.cc:1151] Deleting tablet 83f5fbf5757a49df93ab949b7657c61f
1027 21:08:10.694722 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.699576 (+ 4854us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.728883 (+ 29307us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.742623 (+ 13740us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4400816}
I20251027 21:08:10.743101 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1002) took 5163 ms. Trace:
I20251027 21:08:10.743112 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 83f5fbf5757a49df93ab949b7657c61f (table default.Table_50b757fa69504e57872e1ecaee13583c [id=2719caa93b0945c59e6ad4d6f1e9e717]) successfully deleted
I20251027 21:08:10.743168 10041 rpcz_store.cc:276] 1027 21:08:05.579186 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.273804 (+694618us) service_pool.cc:225] Handling call
1027 21:08:10.742982 (+4469178us) ts_tablet_manager.cc:1151] Deleting tablet 8649a3b611cc48ff8832c0fdeb128efe
1027 21:08:10.742995 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.743087 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4445445}
I20251027 21:08:10.743422 10041 tablet_replica.cc:333] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:10.743599 10041 raft_consensus.cc:2243] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:10.743750 10041 raft_consensus.cc:2272] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:10.744026 10032 consensus_queue.cc:260] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.744261 10041 ts_tablet_manager.cc:1916] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:10.744345 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 22dbfa50996b47269983c107cc399000
I20251027 21:08:10.745065 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 0166ba1e634747cd866fabc4e9acf1b5
I20251027 21:08:10.745668 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_dbd96c7982f647c6990d16cbec0a6a04" } new_table_name: "default.Table_ad1b7cd233624251b8fe7c077dfb57f0" modify_external_catalogs: true
I20251027 21:08:10.746379 10032 ts_tablet_manager.cc:1434] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.048s user 0.001s sys 0.000s
W20251027 21:08:10.746958 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 8649a3b611cc48ff8832c0fdeb128efe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8649a3b611cc48ff8832c0fdeb128efe already in progress: creating tablet
I20251027 21:08:10.747200 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 0166ba1e634747cd866fabc4e9acf1b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:10.747284 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet 0166ba1e634747cd866fabc4e9acf1b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:10.747386 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 22dbfa50996b47269983c107cc399000 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:10.747462 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet 22dbfa50996b47269983c107cc399000 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:10.747526 9919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "420ccfaaa7f24ad0aea9fdfea24fc5dd" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:10.747691 9919 raft_consensus.cc:3060] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:10.747984 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b7e7e18292df4076a52ea8b99c94f901"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.748463 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7e7e18292df4076a52ea8b99c94f901 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.749142 9902 tablet_service.cc:1505] Processing CreateTablet for tablet d91fced778784b65be2486e24426d1b5 (DEFAULT_TABLE table=default.Table_8e9ad7e433ba4538be8f764418dd70a2 [id=84cfa52eb6934fbf80d2027aa63e0c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.749465 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d91fced778784b65be2486e24426d1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.750725 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 83f5fbf5757a49df93ab949b7657c61f (table default.Table_50b757fa69504e57872e1ecaee13583c [id=2719caa93b0945c59e6ad4d6f1e9e717]) successfully deleted
I20251027 21:08:10.751140 9919 raft_consensus.cc:2468] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
W20251027 21:08:10.751781 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 8649a3b611cc48ff8832c0fdeb128efe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8649a3b611cc48ff8832c0fdeb128efe already in progress: creating tablet
I20251027 21:08:10.751941 10041 ts_tablet_manager.cc:1929] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:10.752020 10041 log.cc:1199] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/6f1d1bd52556463fa306e51eb253af40
I20251027 21:08:10.752363 10041 ts_tablet_manager.cc:1950] T 6f1d1bd52556463fa306e51eb253af40 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:10.753007 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1003) took 5173 ms. Trace:
I20251027 21:08:10.753093 10041 rpcz_store.cc:276] 1027 21:08:05.579266 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.273975 (+694709us) service_pool.cc:225] Handling call
1027 21:08:10.743383 (+4469408us) ts_tablet_manager.cc:1151] Deleting tablet 6f1d1bd52556463fa306e51eb253af40
1027 21:08:10.743392 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.748930 (+ 5538us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.751845 (+ 2915us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.752992 (+ 1147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4442382}
I20251027 21:08:10.753420 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 6f1d1bd52556463fa306e51eb253af40 (table default.Table_9b06fbc2554c4780a754ed9cfb964fa9 [id=47a1de450a474742ae41e277716f5381]) successfully deleted
I20251027 21:08:10.754354 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3c794af372fc4ed8ab15aecdf30067f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:10.754724 10032 tablet_bootstrap.cc:492] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:08:10.754809 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c794af372fc4ed8ab15aecdf30067f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.755812 9891 tablet_service.cc:1505] Processing CreateTablet for tablet eb8a89a572314d72a7d1ce77e6754abb (DEFAULT_TABLE table=default.Table_d3089b6fc3e64e108d93bdab2c49aaac [id=583c5e9d61e64173ae6799c3683a2d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.756182 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb8a89a572314d72a7d1ce77e6754abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.756163 9742 tablet_service.cc:1505] Processing CreateTablet for tablet eb8a89a572314d72a7d1ce77e6754abb (DEFAULT_TABLE table=default.Table_d3089b6fc3e64e108d93bdab2c49aaac [id=583c5e9d61e64173ae6799c3683a2d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.756520 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb8a89a572314d72a7d1ce77e6754abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.758119 9737 tablet_service.cc:1505] Processing CreateTablet for tablet d91fced778784b65be2486e24426d1b5 (DEFAULT_TABLE table=default.Table_8e9ad7e433ba4538be8f764418dd70a2 [id=84cfa52eb6934fbf80d2027aa63e0c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.758522 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d91fced778784b65be2486e24426d1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.759827 11425 raft_consensus.cc:493] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:10.759979 11425 raft_consensus.cc:515] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.760833 11425 leader_election.cc:290] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:10.760913 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe94124b239e442dbf644e4a483615fc" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:10.761368 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe94124b239e442dbf644e4a483615fc" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:10.761763 9859 leader_election.cc:343] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:10.761965 9860 leader_election.cc:343] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.762038 9860 leader_election.cc:304] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:10.763756 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1042) took 4099 ms. Trace:
I20251027 21:08:10.763918 10041 rpcz_store.cc:276] 1027 21:08:06.664689 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.888321 (+223632us) service_pool.cc:225] Handling call
1027 21:08:10.763620 (+3875299us) ts_tablet_manager.cc:1151] Deleting tablet c548d2413a5940c8b114ced667b4dc41
1027 21:08:10.763631 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.763737 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3875112}
I20251027 21:08:10.764038 11425 raft_consensus.cc:2749] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:10.764431 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet c548d2413a5940c8b114ced667b4dc41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c548d2413a5940c8b114ced667b4dc41 already in progress: creating tablet
I20251027 21:08:10.764750 10041 tablet_replica.cc:333] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:10.765033 10041 raft_consensus.cc:2243] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251027 21:08:10.765300 9568 consensus_peers.cc:597] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d64a13667194c12864fa388eb88a380. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:10.765946 10041 raft_consensus.cc:2272] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:10.766626 10041 ts_tablet_manager.cc:1916] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:10.768936 11412 consensus_queue.cc:1048] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:10.770969 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_1100e497c4ed4782824ba1c9e058fdbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:10.770969 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0e46a559dc894535889e6d10f3a67414"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.771731 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1100e497c4ed4782824ba1c9e058fdbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:10.771768 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e46a559dc894535889e6d10f3a67414 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:10.773133 9568 consensus_peers.cc:597] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:10.773821 11412 consensus_queue.cc:1048] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251027 21:08:10.776742 9568 leader_election.cc:343] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 01e9db9eaf8e41c6a2ee13b9ed127421
I20251027 21:08:10.776899 9568 leader_election.cc:304] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:10.777272 11412 raft_consensus.cc:2749] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:10.770388 10032 tablet_bootstrap.cc:654] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:10.779307 11392 tablet.cc:1722] T 9fa39697360c46138ef4cd21e8c7a950 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251027 21:08:10.783883 9740 tablet_service.cc:1505] Processing CreateTablet for tablet c6a3cf8372184825a18d734ce8f94d62 (DEFAULT_TABLE table=default.Table_effb3e21eeb54ede9520b087b294612c [id=93e27e936ccd4fd792e8a2f169e3dd68]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.784412 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a3cf8372184825a18d734ce8f94d62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.786222 9902 tablet_service.cc:1505] Processing CreateTablet for tablet f130520beb774d718496d98727a68c1a (DEFAULT_TABLE table=default.Table_977d2d84338e4a73a6e069733e7157f5 [id=00a032835a16405c9eeaee42d9637022]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.783883 9742 tablet_service.cc:1505] Processing CreateTablet for tablet f130520beb774d718496d98727a68c1a (DEFAULT_TABLE table=default.Table_977d2d84338e4a73a6e069733e7157f5 [id=00a032835a16405c9eeaee42d9637022]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.788450 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f130520beb774d718496d98727a68c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.789523 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 641699af1428453194656ee84b96e031 (DEFAULT_TABLE table=default.Table_3342d27362814c5da812391969037f20 [id=da04f8ff74a14fce8759857ef0da2ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.789824 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 641699af1428453194656ee84b96e031. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.789791 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 641699af1428453194656ee84b96e031 (DEFAULT_TABLE table=default.Table_3342d27362814c5da812391969037f20 [id=da04f8ff74a14fce8759857ef0da2ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.790212 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 641699af1428453194656ee84b96e031. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.789791 9891 tablet_service.cc:1505] Processing CreateTablet for tablet c6a3cf8372184825a18d734ce8f94d62 (DEFAULT_TABLE table=default.Table_effb3e21eeb54ede9520b087b294612c [id=93e27e936ccd4fd792e8a2f169e3dd68]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.794586 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a3cf8372184825a18d734ce8f94d62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.794584 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 9dc603374d7c461c934db1a593e32cda (DEFAULT_TABLE table=default.Table_77b0da63638c4b85b08ba857ea63f5a9 [id=2d335c276cd9427bbf64dced90b97f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.795367 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 9dc603374d7c461c934db1a593e32cda (DEFAULT_TABLE table=default.Table_77b0da63638c4b85b08ba857ea63f5a9 [id=2d335c276cd9427bbf64dced90b97f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:10.795673 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dc603374d7c461c934db1a593e32cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.798070 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f130520beb774d718496d98727a68c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.801198 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dc603374d7c461c934db1a593e32cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:10.806973 10041 ts_tablet_manager.cc:1929] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:10.807082 10041 log.cc:1199] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/4c67c5cb3b244feb8478f9594b834fbf
I20251027 21:08:10.808877 10041 ts_tablet_manager.cc:1950] T 4c67c5cb3b244feb8478f9594b834fbf P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:10.810168 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1043) took 4141 ms. Trace:
I20251027 21:08:10.810350 10041 rpcz_store.cc:276] 1027 21:08:06.668402 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.888518 (+220116us) service_pool.cc:225] Handling call
1027 21:08:10.764703 (+3876185us) ts_tablet_manager.cc:1151] Deleting tablet 4c67c5cb3b244feb8478f9594b834fbf
1027 21:08:10.764713 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.803874 (+ 39161us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.806885 (+ 3011us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.810126 (+ 3241us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3876067}
I20251027 21:08:10.810886 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1044) took 4120 ms. Trace:
I20251027 21:08:10.811014 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 4c67c5cb3b244feb8478f9594b834fbf (table default.Table_e49807b03fc64f669376118d2ee01918 [id=642101fc044642829702e0ac1341329f]) successfully deleted
I20251027 21:08:10.811019 10041 rpcz_store.cc:276] 1027 21:08:06.690418 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.888629 (+198211us) service_pool.cc:225] Handling call
1027 21:08:10.810770 (+3922141us) ts_tablet_manager.cc:1151] Deleting tablet 17a19ce92da8448bba471f94e3ee4647
1027 21:08:10.810779 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.810868 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3922052}
I20251027 21:08:10.811338 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1045) took 4089 ms. Trace:
W20251027 21:08:10.811405 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 17a19ce92da8448bba471f94e3ee4647 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17a19ce92da8448bba471f94e3ee4647 already in progress: creating tablet
I20251027 21:08:10.811409 10041 rpcz_store.cc:276] 1027 21:08:06.721599 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.888720 (+167121us) service_pool.cc:225] Handling call
1027 21:08:10.811244 (+3922524us) ts_tablet_manager.cc:1151] Deleting tablet 35b99f8c57e847b1848cbdea9e50da11
1027 21:08:10.811252 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.811322 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3922441}
I20251027 21:08:10.811697 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1046) took 4085 ms. Trace:
W20251027 21:08:10.811882 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 35b99f8c57e847b1848cbdea9e50da11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35b99f8c57e847b1848cbdea9e50da11 already in progress: creating tablet
I20251027 21:08:10.820084 11412 consensus_queue.cc:1048] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:10.824496 10041 rpcz_store.cc:276] 1027 21:08:06.726682 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.888805 (+162123us) service_pool.cc:225] Handling call
1027 21:08:10.811598 (+3922793us) ts_tablet_manager.cc:1151] Deleting tablet 933fb952d66d4cb1ade70c334fbbb3e8
1027 21:08:10.811605 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.811671 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3922713}
I20251027 21:08:10.825063 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1047) took 3989 ms. Trace:
W20251027 21:08:10.825145 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 933fb952d66d4cb1ade70c334fbbb3e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 933fb952d66d4cb1ade70c334fbbb3e8 already in progress: creating tablet
I20251027 21:08:10.825325 10041 rpcz_store.cc:276] 1027 21:08:06.835786 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.888891 (+ 53105us) service_pool.cc:225] Handling call
1027 21:08:10.824929 (+3936038us) ts_tablet_manager.cc:1151] Deleting tablet 4a621d5246f14997a2f2952ab640efc8
1027 21:08:10.824941 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.825045 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3935954}
I20251027 21:08:10.825680 10041 tablet_replica.cc:333] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:10.825747 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_557acd3dc3344fff87c5b92798fcfa66" } new_table_name: "default.Table_3f998bc22cc84917b46975ebd9a461e0" modify_external_catalogs: true
I20251027 21:08:10.825937 10041 raft_consensus.cc:2243] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:10.826090 10041 raft_consensus.cc:2272] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:10.826669 10041 ts_tablet_manager.cc:1916] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:10.826949 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 5cb8ca3a50c148e6b602e949650353f8
I20251027 21:08:10.827273 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 953b137f14a44955ae41ab503d04b43c
I20251027 21:08:10.831782 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 953b137f14a44955ae41ab503d04b43c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:10.832018 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 5cb8ca3a50c148e6b602e949650353f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:08:10.832935 9566 consensus_peers.cc:597] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:10.834713 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 4a621d5246f14997a2f2952ab640efc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a621d5246f14997a2f2952ab640efc8 already in progress: creating tablet
I20251027 21:08:10.837276 10041 ts_tablet_manager.cc:1929] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:10.837461 10041 log.cc:1199] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/8dc358445526415d8cd4f1ac6779126d
I20251027 21:08:10.837859 10041 ts_tablet_manager.cc:1950] T 8dc358445526415d8cd4f1ac6779126d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:10.841039 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1048) took 4005 ms. Trace:
I20251027 21:08:10.841173 10041 rpcz_store.cc:276] 1027 21:08:06.835890 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.888974 (+ 53084us) service_pool.cc:225] Handling call
1027 21:08:10.825639 (+3936665us) ts_tablet_manager.cc:1151] Deleting tablet 8dc358445526415d8cd4f1ac6779126d
1027 21:08:10.825647 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.831329 (+ 5682us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.837175 (+ 5846us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.841019 (+ 3844us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3936590}
I20251027 21:08:10.841650 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1049) took 4003 ms. Trace:
I20251027 21:08:10.841785 10041 rpcz_store.cc:276] 1027 21:08:06.838373 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.889052 (+ 50679us) service_pool.cc:225] Handling call
1027 21:08:10.841533 (+3952481us) ts_tablet_manager.cc:1151] Deleting tablet db2735604dc14039810df4ecd330c671
1027 21:08:10.841542 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.841633 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3952391}
I20251027 21:08:10.842181 10041 tablet_replica.cc:333] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:10.842507 10041 raft_consensus.cc:2243] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:10.842991 10041 raft_consensus.cc:2272] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:10.843544 10041 ts_tablet_manager.cc:1916] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:10.843824 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 8dc358445526415d8cd4f1ac6779126d (table default.Table_1b463a3e59984a129fc2824928138b26 [id=7422045718af4223aadf1927576e0b53]) successfully deleted
W20251027 21:08:10.844026 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet db2735604dc14039810df4ecd330c671 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db2735604dc14039810df4ecd330c671 already in progress: creating tablet
I20251027 21:08:10.848524 9568 leader_election.cc:304] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:10.848838 11412 raft_consensus.cc:2804] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:10.848903 11412 raft_consensus.cc:493] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:10.848956 11412 raft_consensus.cc:3060] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:10.845983 11425 raft_consensus.cc:493] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:10.846655 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet 953b137f14a44955ae41ab503d04b43c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:10.849231 11425 raft_consensus.cc:515] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.850258 11425 leader_election.cc:290] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:10.850370 11425 raft_consensus.cc:493] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:10.850358 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9351a369dc1e4ac49f6ad419eab3a4c5" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:10.850473 11425 raft_consensus.cc:515] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:10.850857 9860 leader_election.cc:343] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.851464 11425 leader_election.cc:290] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:10.851439 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "952583fd7fde4b4baccdb98c01bdf9e1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:10.851854 9860 leader_election.cc:343] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.851948 11412 raft_consensus.cc:515] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.852105 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9351a369dc1e4ac49f6ad419eab3a4c5" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:10.852203 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "952583fd7fde4b4baccdb98c01bdf9e1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:10.852546 9859 leader_election.cc:343] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.852632 9859 leader_election.cc:304] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:10.852945 11412 leader_election.cc:290] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:10.853168 9859 leader_election.cc:343] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.853267 9859 leader_election.cc:304] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:10.853857 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67cb71ce43244d8ab9e71ead6bb315c9" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:10.854307 11234 raft_consensus.cc:2749] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:10.854308 9566 leader_election.cc:343] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.854494 11234 raft_consensus.cc:2749] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:10.846928 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet 5cb8ca3a50c148e6b602e949650353f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:10.855106 9568 leader_election.cc:304] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:10.855659 11412 raft_consensus.cc:2804] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:10.856455 11412 raft_consensus.cc:697] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:10.856788 11412 consensus_queue.cc:237] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.858760 10032 tablet_bootstrap.cc:492] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:10.858935 10032 ts_tablet_manager.cc:1403] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.112s user 0.003s sys 0.000s
I20251027 21:08:10.859223 10041 ts_tablet_manager.cc:1929] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:10.861502 11392 tablet.cc:1722] T 45568792d7954b498581ce69fb611948 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:08:10.861627 10043 ts_tablet_manager.cc:1929] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:10.861721 10043 log.cc:1199] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/6f1d1bd52556463fa306e51eb253af40
I20251027 21:08:10.862069 10043 ts_tablet_manager.cc:1950] T 6f1d1bd52556463fa306e51eb253af40 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:10.862069 10041 log.cc:1199] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/f6789dc2f7cb4a3c826ac507d5c9a239
I20251027 21:08:10.862445 10041 ts_tablet_manager.cc:1950] T f6789dc2f7cb4a3c826ac507d5c9a239 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:10.863365 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1050) took 4010 ms. Trace:
I20251027 21:08:10.863518 10041 rpcz_store.cc:276] 1027 21:08:06.852897 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.889142 (+ 36245us) service_pool.cc:225] Handling call
1027 21:08:10.842100 (+3952958us) ts_tablet_manager.cc:1151] Deleting tablet f6789dc2f7cb4a3c826ac507d5c9a239
1027 21:08:10.842111 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.854118 (+ 12007us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.859135 (+ 5017us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.863344 (+ 4209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3952871}
I20251027 21:08:10.866541 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet f6789dc2f7cb4a3c826ac507d5c9a239 (table default.Table_ab9ae94ee52942d588d846e58da2e533 [id=19f6e48bb56848afb9b5982c825b093e]) successfully deleted
I20251027 21:08:10.859725 10032 raft_consensus.cc:359] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.867343 10032 raft_consensus.cc:385] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:10.867434 10032 raft_consensus.cc:740] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:10.867272 10041 tablet_replica.cc:333] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:10.867808 10041 raft_consensus.cc:2243] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:10.868223 10041 pending_rounds.cc:70] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678: Trying to abort 1 pending ops.
I20251027 21:08:10.868324 10041 pending_rounds.cc:77] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510687325245440 op_type: NO_OP noop_request { }
I20251027 21:08:10.868455 10041 raft_consensus.cc:2889] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:10.869513 10041 raft_consensus.cc:2272] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:10.868002 10032 consensus_queue.cc:260] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.870738 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7975dedb2ea34a1aa8d4bd689438585c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.871362 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7975dedb2ea34a1aa8d4bd689438585c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.871760 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7c14877bc32245418a930cf51449c007"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.872191 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c14877bc32245418a930cf51449c007 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.874048 10032 ts_tablet_manager.cc:1434] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20251027 21:08:10.875025 10032 tablet_bootstrap.cc:492] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:10.877111 10032 tablet_bootstrap.cc:654] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:10.874706 10041 ts_tablet_manager.cc:1916] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:10.886109 11234 raft_consensus.cc:493] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:10.887248 11234 raft_consensus.cc:515] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.887809 11234 leader_election.cc:290] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:10.888945 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2752b86151a6428c88bfaac1ff72cfb2" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:10.889786 9859 leader_election.cc:343] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:10.903790 10041 ts_tablet_manager.cc:1929] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:10.905146 10041 log.cc:1199] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/7292bc6c22ca4ab5b8f53a69520833ba
I20251027 21:08:10.905761 10041 ts_tablet_manager.cc:1950] T 7292bc6c22ca4ab5b8f53a69520833ba P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:10.906538 10032 tablet_bootstrap.cc:492] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:10.906777 10032 ts_tablet_manager.cc:1403] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.001s
I20251027 21:08:10.907574 10032 raft_consensus.cc:359] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.907797 10032 raft_consensus.cc:385] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:10.907917 10032 raft_consensus.cc:740] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:10.908195 10032 consensus_queue.cc:260] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.908874 10032 ts_tablet_manager.cc:1434] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:10.909518 10032 tablet_bootstrap.cc:492] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:10.912438 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1051) took 4059 ms. Trace:
I20251027 21:08:10.912578 10041 rpcz_store.cc:276] 1027 21:08:06.852973 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.889228 (+ 36255us) service_pool.cc:225] Handling call
1027 21:08:10.867222 (+3977994us) ts_tablet_manager.cc:1151] Deleting tablet 7292bc6c22ca4ab5b8f53a69520833ba
1027 21:08:10.867233 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.899211 (+ 31978us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.903691 (+ 4480us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.912405 (+ 8714us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3328,"tablet-delete.queue_time_us":3977902}
I20251027 21:08:10.913033 10041 tablet_replica.cc:333] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:10.913260 10041 raft_consensus.cc:2243] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:10.913457 10041 raft_consensus.cc:2272] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:10.913554 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 7292bc6c22ca4ab5b8f53a69520833ba (table default.Table_4658a756ae8149cf9e248c65a102bdb6 [id=0a31c72c314e4e5ab0655d9b853c941c]) successfully deleted
I20251027 21:08:10.913996 10041 ts_tablet_manager.cc:1916] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:10.920449 10041 ts_tablet_manager.cc:1929] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:10.925088 10032 tablet_bootstrap.cc:654] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:10.927049 10041 log.cc:1199] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/07a9c3b7ef1548ec9d5ae9f20545f263
I20251027 21:08:10.927495 10041 ts_tablet_manager.cc:1950] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:10.928236 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1052) took 4075 ms. Trace:
I20251027 21:08:10.928371 10041 rpcz_store.cc:276] 1027 21:08:06.853036 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.889317 (+ 36281us) service_pool.cc:225] Handling call
1027 21:08:10.912987 (+4023670us) ts_tablet_manager.cc:1151] Deleting tablet 07a9c3b7ef1548ec9d5ae9f20545f263
1027 21:08:10.912997 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.917023 (+ 4026us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.920356 (+ 3333us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.928218 (+ 7862us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4023579}
I20251027 21:08:10.928802 10041 tablet_replica.cc:333] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:10.929031 10041 raft_consensus.cc:2243] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:10.929986 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 07a9c3b7ef1548ec9d5ae9f20545f263 (table default.Table_a7632420526d409290924efb41972ee1 [id=4a103f97fc964eaba06cd48d4ba854a6]) successfully deleted
I20251027 21:08:10.930394 10041 pending_rounds.cc:70] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678: Trying to abort 1 pending ops.
I20251027 21:08:10.930497 10041 pending_rounds.cc:77] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510692814147584 op_type: NO_OP noop_request { }
I20251027 21:08:10.930868 10041 raft_consensus.cc:2889] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:10.931006 10041 raft_consensus.cc:2272] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:10.933036 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b7c9623ff35e4ae9a2ead45bef605e49 consensus state has no leader
I20251027 21:08:10.933789 10041 ts_tablet_manager.cc:1916] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:10.938431 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a04bcb2698ec467e8f85f4408ae2ca01" } new_table_name: "default.Table_c5a552fcdf214416a4b9217758c32b80" modify_external_catalogs: true
I20251027 21:08:10.938508 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_fb6cfb8df03347efbf6dd0002e05bf86" } new_table_name: "default.Table_98dbd9ef5d9840fb9b0d6fa7463da36c" modify_external_catalogs: true
I20251027 21:08:10.943109 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f846038ed8334d998a32d5775f2b0403" } modify_external_catalogs: true
I20251027 21:08:10.943368 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f846038ed8334d998a32d5775f2b0403" } modify_external_catalogs: true
I20251027 21:08:10.943606 10032 tablet_bootstrap.cc:492] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:10.943804 10032 ts_tablet_manager.cc:1403] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.005s
I20251027 21:08:10.944016 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9695e56a968e424c84c3dd11d12d5fb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.944512 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9695e56a968e424c84c3dd11d12d5fb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.944583 10032 raft_consensus.cc:359] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.944749 10032 raft_consensus.cc:385] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:10.944793 10032 raft_consensus.cc:740] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:10.945034 10032 consensus_queue.cc:260] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.948879 10032 ts_tablet_manager.cc:1434] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20251027 21:08:10.949725 10032 tablet_bootstrap.cc:492] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:10.951156 10032 tablet_bootstrap.cc:654] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:10.952175 9310 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 93d2f7df14a74731aea84927f0b7ed45 (table default.Table_d52fc4f0f55b4bab9edbfff9297f8728 [id=c24a9debcdfc413ba606517b997cd19c]). Expected version 4 got 0
I20251027 21:08:10.958127 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet e7df6f0acec34d399aa25b78c3ed7c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:10.958899 10041 ts_tablet_manager.cc:1929] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:10.958997 10041 log.cc:1199] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/b3abc5f78c4e41e29128e2d790319c75
I20251027 21:08:10.959370 10041 ts_tablet_manager.cc:1950] T b3abc5f78c4e41e29128e2d790319c75 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:10.960135 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1053) took 4107 ms. Trace:
I20251027 21:08:10.960223 10041 rpcz_store.cc:276] 1027 21:08:06.853117 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.889409 (+ 36292us) service_pool.cc:225] Handling call
1027 21:08:10.928756 (+4039347us) ts_tablet_manager.cc:1151] Deleting tablet b3abc5f78c4e41e29128e2d790319c75
1027 21:08:10.928765 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.937259 (+ 8494us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.958815 (+ 21556us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.960118 (+ 1303us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4039265}
I20251027 21:08:10.960534 10041 tablet_replica.cc:333] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:10.960716 10041 raft_consensus.cc:2243] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:10.960870 10041 raft_consensus.cc:2272] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:10.961226 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet b3abc5f78c4e41e29128e2d790319c75 (table default.Table_3cb6bc9f33ea40aa9d58e82f709c64b7 [id=b9930c5b513749ac905895ebca813733]) successfully deleted
I20251027 21:08:10.964074 11234 raft_consensus.cc:493] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:10.964188 11234 raft_consensus.cc:515] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:10.964758 11234 leader_election.cc:290] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:10.965298 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28a2e85021394f9d8a9ea524959b52a1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:10.965927 9859 leader_election.cc:343] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:10.966806 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2e5b6912c484517afd48f8070455da8 consensus state has no leader
I20251027 21:08:10.967120 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c26460a4ac0742d997bcc2dc9d88ce5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.967582 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c26460a4ac0742d997bcc2dc9d88ce5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.967466 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4cca34ef6c5f47e285603a77b24501bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:10.968044 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4cca34ef6c5f47e285603a77b24501bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:10.968513 10041 ts_tablet_manager.cc:1916] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:10.972313 9306 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_b2a1b49649c54e5ab7a23fad7a157502 [id=434ed8c436c149ef9c7eb86ad252291d] alter complete (version 4)
I20251027 21:08:10.972779 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_bb9c3894cb5d40738ca8be284b1d1e25" } new_table_name: "default.Table_4ba26495af344e3cb082a4b781f1b208" modify_external_catalogs: true
I20251027 21:08:10.978078 9731 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:40722 (request call id 162) took 1660 ms. Trace:
I20251027 21:08:10.979789 11439 tablet.cc:1722] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251027 21:08:10.997175 10041 ts_tablet_manager.cc:1929] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:10.997310 10041 log.cc:1199] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/a2d26784bbb24ae3893103802c41affb
I20251027 21:08:10.997716 10041 ts_tablet_manager.cc:1950] T a2d26784bbb24ae3893103802c41affb P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.002372 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1054) took 4149 ms. Trace:
I20251027 21:08:11.002496 10041 rpcz_store.cc:276] 1027 21:08:06.853179 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.889490 (+ 36311us) service_pool.cc:225] Handling call
1027 21:08:10.960493 (+4071003us) ts_tablet_manager.cc:1151] Deleting tablet a2d26784bbb24ae3893103802c41affb
1027 21:08:10.960501 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.987569 (+ 27068us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.997071 (+ 9502us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.002344 (+ 5273us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4070924}
I20251027 21:08:11.002867 10041 tablet_replica.cc:333] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.003054 10041 raft_consensus.cc:2243] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.003091 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet a2d26784bbb24ae3893103802c41affb (table default.Table_ecf1a7cbc64b46579e08c39312812361 [id=54ab773e30384ae98a0b0539284c85b4]) successfully deleted
I20251027 21:08:11.003202 10041 raft_consensus.cc:2272] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.003736 10041 ts_tablet_manager.cc:1916] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.020097 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet 2752b86151a6428c88bfaac1ff72cfb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:11.023463 10041 ts_tablet_manager.cc:1929] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.023582 10041 log.cc:1199] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/7425b6b0184e4d0b871777b25d904ac4
I20251027 21:08:11.023898 10041 ts_tablet_manager.cc:1950] T 7425b6b0184e4d0b871777b25d904ac4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.024622 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1064) took 3755 ms. Trace:
I20251027 21:08:11.024704 10041 rpcz_store.cc:276] 1027 21:08:07.269470 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.269750 (+ 280us) service_pool.cc:225] Handling call
1027 21:08:11.002806 (+3733056us) ts_tablet_manager.cc:1151] Deleting tablet 7425b6b0184e4d0b871777b25d904ac4
1027 21:08:11.002819 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.020362 (+ 17543us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.023372 (+ 3010us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.024604 (+ 1232us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3732885}
I20251027 21:08:11.025013 10041 tablet_replica.cc:333] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.025128 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 7425b6b0184e4d0b871777b25d904ac4 (table default.Table_92cfd00027b045f3b151deec3558ece6 [id=479548b373ef42b4ad8683a9dcfb3ddc]) successfully deleted
W20251027 21:08:11.026546 9859 consensus_peers.cc:597] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:11.030423 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_352763845d1f4fb2a97474f8c82883b7" } modify_external_catalogs: true
I20251027 21:08:11.030606 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_352763845d1f4fb2a97474f8c82883b7" } modify_external_catalogs: true
I20251027 21:08:11.037648 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0e46a559dc894535889e6d10f3a67414" } modify_external_catalogs: true
I20251027 21:08:11.037914 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0e46a559dc894535889e6d10f3a67414" } modify_external_catalogs: true
I20251027 21:08:11.025185 10041 raft_consensus.cc:2243] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.039981 10041 raft_consensus.cc:2272] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.040572 10041 ts_tablet_manager.cc:1916] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:11.041922 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93a3b22cdc1543aebcc7fe72f2f2c0ae consensus state has no leader
W20251027 21:08:10.862360 9568 leader_election.cc:343] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: f775a7e42bdc4910bdb604b26a28a7f4
I20251027 21:08:11.055510 9568 leader_election.cc:304] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:11.055886 9568 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 193532us
I20251027 21:08:11.056874 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1007) took 5482 ms. Trace:
I20251027 21:08:11.057755 9915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67cb71ce43244d8ab9e71ead6bb315c9" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:11.057986 9915 raft_consensus.cc:3060] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:11.058229 10041 ts_tablet_manager.cc:1929] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.058318 10041 log.cc:1199] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/588e2f58f07b4a32a78dffd5b92ceb12
I20251027 21:08:11.058442 9607 tablet_service.cc:1505] Processing CreateTablet for tablet d91fced778784b65be2486e24426d1b5 (DEFAULT_TABLE table=default.Table_8e9ad7e433ba4538be8f764418dd70a2 [id=84cfa52eb6934fbf80d2027aa63e0c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.058619 10041 ts_tablet_manager.cc:1950] T 588e2f58f07b4a32a78dffd5b92ceb12 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.058856 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d91fced778784b65be2486e24426d1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.059270 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1065) took 3789 ms. Trace:
I20251027 21:08:11.059342 10041 rpcz_store.cc:276] 1027 21:08:07.269704 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.270778 (+ 1074us) service_pool.cc:225] Handling call
1027 21:08:11.024975 (+3754197us) ts_tablet_manager.cc:1151] Deleting tablet 588e2f58f07b4a32a78dffd5b92ceb12
1027 21:08:11.024983 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.046725 (+ 21742us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.058118 (+ 11393us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.059252 (+ 1134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3754022}
I20251027 21:08:11.059617 10041 tablet_replica.cc:333] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.059806 10041 raft_consensus.cc:2243] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.059927 10041 raft_consensus.cc:2272] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.060382 10041 ts_tablet_manager.cc:1916] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.060650 9915 raft_consensus.cc:2468] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:10.863564 10029 tablet_bootstrap.cc:492] T f4187b2f66f24055b1df170f1062cc88 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:11.061532 10029 ts_tablet_manager.cc:1403] T f4187b2f66f24055b1df170f1062cc88 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.686s user 0.023s sys 0.016s
I20251027 21:08:11.062284 10029 raft_consensus.cc:359] T f4187b2f66f24055b1df170f1062cc88 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.062460 10029 raft_consensus.cc:385] T f4187b2f66f24055b1df170f1062cc88 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.062512 10029 raft_consensus.cc:740] T f4187b2f66f24055b1df170f1062cc88 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.062690 10029 consensus_queue.cc:260] T f4187b2f66f24055b1df170f1062cc88 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.062798 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 9dc603374d7c461c934db1a593e32cda (DEFAULT_TABLE table=default.Table_77b0da63638c4b85b08ba857ea63f5a9 [id=2d335c276cd9427bbf64dced90b97f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.063114 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dc603374d7c461c934db1a593e32cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.063180 10029 ts_tablet_manager.cc:1434] T f4187b2f66f24055b1df170f1062cc88 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:11.063446 10029 tablet_bootstrap.cc:492] T 7425b6b0184e4d0b871777b25d904ac4 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:11.065078 10029 tablet_bootstrap.cc:654] T 7425b6b0184e4d0b871777b25d904ac4 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:11.065897 11412 raft_consensus.cc:2749] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:11.067147 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 22dbfa50996b47269983c107cc399000 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:11.067320 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet 0166ba1e634747cd866fabc4e9acf1b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:11.067613 9566 consensus_peers.cc:597] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:11.068101 9611 tablet_service.cc:1505] Processing CreateTablet for tablet eb8a89a572314d72a7d1ce77e6754abb (DEFAULT_TABLE table=default.Table_d3089b6fc3e64e108d93bdab2c49aaac [id=583c5e9d61e64173ae6799c3683a2d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.068423 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb8a89a572314d72a7d1ce77e6754abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.069887 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 588e2f58f07b4a32a78dffd5b92ceb12 (table default.Table_7a3ebf1ecf35428495d2d7bb5540530a [id=c4838f30ee9f4aec96e6c73b4ce15c7b]) successfully deleted
I20251027 21:08:10.987987 11437 raft_consensus.cc:493] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.070384 11437 raft_consensus.cc:515] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.070811 9598 tablet_service.cc:1505] Processing CreateTablet for tablet f130520beb774d718496d98727a68c1a (DEFAULT_TABLE table=default.Table_977d2d84338e4a73a6e069733e7157f5 [id=00a032835a16405c9eeaee42d9637022]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.071146 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f130520beb774d718496d98727a68c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.071177 11437 leader_election.cc:290] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:11.003988 10031 tablet_bootstrap.cc:492] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:11.012408 11317 raft_consensus.cc:493] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.071403 11317 raft_consensus.cc:515] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:11.071931 11317 leader_election.cc:290] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:11.072115 10031 ts_tablet_manager.cc:1403] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.407s user 0.002s sys 0.009s
I20251027 21:08:11.072847 10031 raft_consensus.cc:359] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:11.072873 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cef2bfee222f45e6b6f20c6d63ff44d4" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:11.072988 10031 raft_consensus.cc:385] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.073035 10031 raft_consensus.cc:740] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.073035 9635 raft_consensus.cc:2468] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
W20251027 21:08:11.073294 9710 leader_election.cc:343] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: a26ded6012174348b7295fc7c0e05e6e
I20251027 21:08:11.073431 10031 consensus_queue.cc:260] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:08:11.073771 9711 leader_election.cc:343] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: a26ded6012174348b7295fc7c0e05e6e
I20251027 21:08:11.073856 9711 leader_election.cc:304] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:11.074385 9610 tablet_service.cc:1505] Processing CreateTablet for tablet c6a3cf8372184825a18d734ce8f94d62 (DEFAULT_TABLE table=default.Table_effb3e21eeb54ede9520b087b294612c [id=93e27e936ccd4fd792e8a2f169e3dd68]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.074534 9710 leader_election.cc:304] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:11.074829 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 641699af1428453194656ee84b96e031 (DEFAULT_TABLE table=default.Table_3342d27362814c5da812391969037f20 [id=da04f8ff74a14fce8759857ef0da2ad3]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:11.074895 9711 leader_election.cc:343] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: cef2bfee222f45e6b6f20c6d63ff44d4
I20251027 21:08:11.075119 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet 5cb8ca3a50c148e6b602e949650353f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:10.888540 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2752b86151a6428c88bfaac1ff72cfb2" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:11.076926 11317 raft_consensus.cc:2749] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:11.077041 11449 tablet.cc:1722] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:08:11.077129 11317 raft_consensus.cc:2804] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:11.077181 11317 raft_consensus.cc:493] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:11.077231 11317 raft_consensus.cc:3060] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:10.966207 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28a2e85021394f9d8a9ea524959b52a1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:11.027125 9630 raft_consensus.cc:1275] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Refusing update from remote peer 0fbd567f929a4f6d8e4d78feef7ed678: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251027 21:08:11.079905 9860 leader_election.cc:343] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.080029 9860 leader_election.cc:304] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:11.080132 11317 raft_consensus.cc:515] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
W20251027 21:08:11.080394 9860 leader_election.cc:343] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.080513 9860 leader_election.cc:304] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:11.080919 11425 raft_consensus.cc:2749] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:11.080961 11234 consensus_queue.cc:1048] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:11.080925 11391 raft_consensus.cc:2749] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:11.081496 11317 leader_election.cc:290] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:11.082316 10031 ts_tablet_manager.cc:1434] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
W20251027 21:08:11.082603 9711 leader_election.cc:343] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: cef2bfee222f45e6b6f20c6d63ff44d4
I20251027 21:08:11.082644 10031 tablet_bootstrap.cc:492] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:11.084244 10031 tablet_bootstrap.cc:654] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:11.084470 11234 raft_consensus.cc:493] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.084597 11234 raft_consensus.cc:515] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.085178 11234 leader_election.cc:290] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.086158 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cef2bfee222f45e6b6f20c6d63ff44d4" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:11.086309 9630 raft_consensus.cc:3060] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:11.087549 11449 tablet.cc:1722] T 45568792d7954b498581ce69fb611948 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:08:11.089236 9630 raft_consensus.cc:2468] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:11.096010 10041 ts_tablet_manager.cc:1929] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.096114 10041 log.cc:1199] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/f4187b2f66f24055b1df170f1062cc88
I20251027 21:08:11.057030 10043 rpcz_store.cc:276] 1027 21:08:05.574669 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.278004 (+703335us) service_pool.cc:225] Handling call
1027 21:08:10.670932 (+4392928us) ts_tablet_manager.cc:1151] Deleting tablet 6f1d1bd52556463fa306e51eb253af40
1027 21:08:10.670940 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:10.858261 (+187321us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:10.861543 (+ 3282us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.056850 (+195307us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4392840}
I20251027 21:08:11.096503 11449 tablet.cc:1722] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:11.097165 10041 ts_tablet_manager.cc:1950] T f4187b2f66f24055b1df170f1062cc88 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.097962 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1062) took 3829 ms. Trace:
I20251027 21:08:11.023744 11446 tablet.cc:1722] T 45568792d7954b498581ce69fb611948 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:08:11.085726 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb95618d9fd54a4f8121b43e717d5616" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:11.098660 9859 leader_election.cc:343] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.098052 10041 rpcz_store.cc:276] 1027 21:08:07.268967 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.271017 (+ 2050us) service_pool.cc:225] Handling call
1027 21:08:11.059578 (+3788561us) ts_tablet_manager.cc:1151] Deleting tablet f4187b2f66f24055b1df170f1062cc88
1027 21:08:11.059587 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.085290 (+ 25703us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.095923 (+ 10633us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.097943 (+ 2020us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3788403}
I20251027 21:08:11.099095 10041 tablet_replica.cc:333] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.099287 10041 raft_consensus.cc:2243] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.099431 10041 raft_consensus.cc:2272] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.099897 10041 ts_tablet_manager.cc:1916] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.099987 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet f4187b2f66f24055b1df170f1062cc88 (table default.Table_d276ca9d745f41a58deb7846600ddcc0 [id=a23a34526a1b46db9ce948189242dc02]) successfully deleted
I20251027 21:08:11.089025 9731 rpcz_store.cc:276] 1027 21:08:09.317215 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.317370 (+ 155us) service_pool.cc:225] Handling call
1027 21:08:10.978050 (+1660680us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251027 21:08:11.095264 9710 leader_election.cc:304] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:11.101223 11317 raft_consensus.cc:2804] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:11.101472 11317 raft_consensus.cc:697] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:11.101881 11317 consensus_queue.cc:237] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:11.113296 10041 ts_tablet_manager.cc:1929] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.113410 10041 log.cc:1199] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/30ce703650c94c06b93bce976cb5d1fd
I20251027 21:08:11.113739 10041 ts_tablet_manager.cc:1950] T 30ce703650c94c06b93bce976cb5d1fd P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.096836 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb95618d9fd54a4f8121b43e717d5616" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:11.097615 11392 tablet.cc:1722] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251027 21:08:11.102732 9568 consensus_peers.cc:597] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:11.102912 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet 953b137f14a44955ae41ab503d04b43c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:11.106233 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a3cf8372184825a18d734ce8f94d62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.118716 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1063) took 3849 ms. Trace:
I20251027 21:08:11.113493 11412 raft_consensus.cc:493] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.118880 10043 tablet_replica.cc:333] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:11.118801 10041 rpcz_store.cc:276] 1027 21:08:07.269226 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.271222 (+ 1996us) service_pool.cc:225] Handling call
1027 21:08:11.099040 (+3827818us) ts_tablet_manager.cc:1151] Deleting tablet 30ce703650c94c06b93bce976cb5d1fd
1027 21:08:11.099049 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.108543 (+ 9494us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.113222 (+ 4679us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.118694 (+ 5472us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3827122}
I20251027 21:08:11.118846 11412 raft_consensus.cc:515] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.119171 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1068) took 3583 ms. Trace:
I20251027 21:08:11.119179 10043 raft_consensus.cc:2243] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.119380 10043 raft_consensus.cc:2272] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.119452 11412 leader_election.cc:290] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.119248 10041 rpcz_store.cc:276] 1027 21:08:07.535560 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.535612 (+ 52us) service_pool.cc:225] Handling call
1027 21:08:11.119067 (+3583455us) ts_tablet_manager.cc:1151] Deleting tablet a24dbceaa95e4f43a5ea416a20e7041e
1027 21:08:11.119075 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.119157 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3583203}
I20251027 21:08:11.119794 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 30ce703650c94c06b93bce976cb5d1fd (table default.Table_cfb36d57b28d49f88f50a05650a40b49 [id=b82c052176d94b1292aa47473d08a68c]) successfully deleted
I20251027 21:08:11.119846 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1067) took 3584 ms. Trace:
I20251027 21:08:11.119922 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb95618d9fd54a4f8121b43e717d5616" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:11.119908 10041 rpcz_store.cc:276] 1027 21:08:07.535388 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.543821 (+ 8433us) service_pool.cc:225] Handling call
1027 21:08:11.119757 (+3575936us) ts_tablet_manager.cc:1151] Deleting tablet 473cc9f814c943a8967d92e58ed7dfd2
1027 21:08:11.119765 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.119832 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3573726}
I20251027 21:08:11.120164 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1066) took 3584 ms. Trace:
W20251027 21:08:11.120244 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet a24dbceaa95e4f43a5ea416a20e7041e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a24dbceaa95e4f43a5ea416a20e7041e already in progress: creating tablet
I20251027 21:08:11.120229 10041 rpcz_store.cc:276] 1027 21:08:07.535277 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.551189 (+ 15912us) service_pool.cc:225] Handling call
1027 21:08:11.120082 (+3568893us) ts_tablet_manager.cc:1151] Deleting tablet 050e9442041d495496bfa7ce25ac6a88
1027 21:08:11.120089 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.120150 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3568723}
W20251027 21:08:11.120386 9566 leader_election.cc:343] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:11.120424 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 473cc9f814c943a8967d92e58ed7dfd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 473cc9f814c943a8967d92e58ed7dfd2 already in progress: creating tablet
I20251027 21:08:11.120468 10041 tablet_replica.cc:333] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.120637 10041 raft_consensus.cc:2243] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.120760 10041 raft_consensus.cc:2272] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:11.120766 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 050e9442041d495496bfa7ce25ac6a88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 050e9442041d495496bfa7ce25ac6a88 already in progress: creating tablet
I20251027 21:08:11.120973 9634 raft_consensus.cc:2468] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 0.
I20251027 21:08:11.121224 10041 ts_tablet_manager.cc:1916] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.121249 10043 ts_tablet_manager.cc:1916] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.121541 9860 leader_election.cc:304] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:11.121788 11400 raft_consensus.cc:2804] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:11.121851 11400 raft_consensus.cc:493] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:11.121909 11400 raft_consensus.cc:3060] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:11.121987 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 641699af1428453194656ee84b96e031. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.124478 11400 raft_consensus.cc:515] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.125476 11400 leader_election.cc:290] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.125633 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb95618d9fd54a4f8121b43e717d5616" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:11.129118 9859 leader_election.cc:343] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.129923 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb95618d9fd54a4f8121b43e717d5616" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:11.130088 9634 raft_consensus.cc:3060] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:11.130525 9568 leader_election.cc:304] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:11.130803 10032 tablet_bootstrap.cc:492] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:11.130904 10032 ts_tablet_manager.cc:1403] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.181s user 0.020s sys 0.020s
W20251027 21:08:11.130951 9568 leader_election.cc:343] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 76e4fb2bea7f4a4696fe9f6bb892bccc
I20251027 21:08:11.131037 9568 leader_election.cc:304] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:11.131384 11142 raft_consensus.cc:2804] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:11.131484 11142 raft_consensus.cc:697] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:11.131511 10032 raft_consensus.cc:359] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.131644 10032 raft_consensus.cc:385] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.131681 10032 raft_consensus.cc:740] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.131703 11142 consensus_queue.cc:237] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.131844 10032 consensus_queue.cc:260] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.132282 10032 ts_tablet_manager.cc:1434] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251027 21:08:11.132408 11419 raft_consensus.cc:2749] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:11.132597 10032 tablet_bootstrap.cc:492] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:11.132858 9634 raft_consensus.cc:2468] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 1.
I20251027 21:08:11.133217 9860 leader_election.cc:304] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:11.133414 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b1f06adb64714ba5818da1b0c060fd16" } modify_external_catalogs: true
I20251027 21:08:11.133579 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b1f06adb64714ba5818da1b0c060fd16" } modify_external_catalogs: true
I20251027 21:08:11.133637 11425 raft_consensus.cc:2804] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:11.133756 11425 raft_consensus.cc:697] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Becoming Leader. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Running, Role: LEADER
I20251027 21:08:11.133958 11425 consensus_queue.cc:237] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.134061 10032 tablet_bootstrap.cc:654] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:11.135131 9568 consensus_peers.cc:597] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07a9c3b7ef1548ec9d5ae9f20545f263. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:11.135296 9568 consensus_peers.cc:597] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6789dc2f7cb4a3c826ac507d5c9a239. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:11.135517 9568 consensus_peers.cc:597] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7292bc6c22ca4ab5b8f53a69520833ba. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:11.137082 11391 raft_consensus.cc:493] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.137195 11391 raft_consensus.cc:515] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.137734 11391 leader_election.cc:290] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.138062 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cdbde6272af147829ecadd67d634f049" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:11.138178 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cdbde6272af147829ecadd67d634f049" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:11.138602 9859 leader_election.cc:343] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:11.139150 9860 leader_election.cc:343] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.139241 9860 leader_election.cc:304] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:11.139545 11391 raft_consensus.cc:2749] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:11.140900 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 6f1d1bd52556463fa306e51eb253af40 (table default.Table_9b06fbc2554c4780a754ed9cfb964fa9 [id=47a1de450a474742ae41e277716f5381]) successfully deleted
I20251027 21:08:11.141726 9911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb95618d9fd54a4f8121b43e717d5616" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:11.142472 9568 leader_election.cc:304] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:11.143301 11411 raft_consensus.cc:2749] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:11.151180 10041 ts_tablet_manager.cc:1929] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.151296 10041 log.cc:1199] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/9ed4f9c61ad54e1292281b30bcd19533
I20251027 21:08:11.157819 10041 ts_tablet_manager.cc:1950] T 9ed4f9c61ad54e1292281b30bcd19533 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.158591 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1069) took 3601 ms. Trace:
I20251027 21:08:11.158699 10041 rpcz_store.cc:276] 1027 21:08:07.557547 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.557597 (+ 50us) service_pool.cc:225] Handling call
1027 21:08:11.120417 (+3562820us) ts_tablet_manager.cc:1151] Deleting tablet 9ed4f9c61ad54e1292281b30bcd19533
1027 21:08:11.120436 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.124238 (+ 3802us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.151082 (+ 26844us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.158572 (+ 7490us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3562586}
I20251027 21:08:11.159003 10041 tablet_replica.cc:333] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.159140 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 9ed4f9c61ad54e1292281b30bcd19533 (table default.Table_796614d1dc4148699dfa07316ea8febf [id=5f4093d9346248dba86e86cf2fbacd92]) successfully deleted
W20251027 21:08:11.178277 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07b8fd24b8af4f8db2a401c26cc65058 consensus state has no leader
I20251027 21:08:11.181233 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet b6c4fc648fa04d9aa4a10b5b6b07cd21 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:11.187947 10031 tablet_bootstrap.cc:492] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:11.188203 10031 ts_tablet_manager.cc:1403] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.106s user 0.000s sys 0.003s
I20251027 21:08:11.188994 10031 raft_consensus.cc:359] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:11.189316 10031 raft_consensus.cc:385] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.189476 10031 raft_consensus.cc:740] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.189781 10031 consensus_queue.cc:260] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:11.191722 10031 ts_tablet_manager.cc:1434] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:08:11.195531 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b2a1b49649c54e5ab7a23fad7a157502" } new_table_name: "default.Table_fe1d774fd34f41fa916f26ced6fb7612" modify_external_catalogs: true
I20251027 21:08:11.197587 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b431103b133e4f548240ad2a4b4f9110" } modify_external_catalogs: true
I20251027 21:08:11.197738 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b431103b133e4f548240ad2a4b4f9110" } modify_external_catalogs: true
W20251027 21:08:11.207568 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8cccd253ae2a46328dc9f767b8e9ee97 consensus state has no leader
I20251027 21:08:11.217674 10041 raft_consensus.cc:2243] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:11.195659 9737 tablet_service.cc:1552] Processing DeleteTablet for tablet 01e9db9eaf8e41c6a2ee13b9ed127421 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:11.218323 10041 pending_rounds.cc:70] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678: Trying to abort 1 pending ops.
I20251027 21:08:11.218374 10041 pending_rounds.cc:77] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510696486592512 op_type: NO_OP noop_request { }
I20251027 21:08:11.218477 10041 raft_consensus.cc:2889] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:11.218552 10041 raft_consensus.cc:2272] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.218731 10031 tablet_bootstrap.cc:492] T f775a7e42bdc4910bdb604b26a28a7f4 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:11.229591 10041 ts_tablet_manager.cc:1916] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.235107 10040 ts_tablet_manager.cc:1929] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:11.235235 10040 log.cc:1199] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/8b30052ff19744a9bf4bfaaa1bc77b36
I20251027 21:08:11.235553 10040 ts_tablet_manager.cc:1950] T 8b30052ff19744a9bf4bfaaa1bc77b36 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:11.238811 10041 ts_tablet_manager.cc:1929] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:11.238945 10041 log.cc:1199] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/cb95618d9fd54a4f8121b43e717d5616
I20251027 21:08:11.239297 10041 ts_tablet_manager.cc:1950] T cb95618d9fd54a4f8121b43e717d5616 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.240101 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1070) took 3671 ms. Trace:
I20251027 21:08:11.240350 10041 rpcz_store.cc:276] 1027 21:08:07.568160 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.568208 (+ 48us) service_pool.cc:225] Handling call
1027 21:08:11.158956 (+3590748us) ts_tablet_manager.cc:1151] Deleting tablet cb95618d9fd54a4f8121b43e717d5616
1027 21:08:11.158967 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.218085 (+ 59118us) spinlock_profiling.cc:243] Waited 58.5 ms on lock 0x616000195754. stack: 00007fc04d4329f6 00007fc04d4327b0 00007fc04c40e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc0592fd100 00007fc05a64aae3 00007fc05d20b287 00007fc05d23e2d4 00007fc05d21ed70 00007fc05d21eafd 0000000000a8867c 00007fc04d478d7f 00007fc04d47c04b 00007fc04d47bec8
1027 21:08:11.234564 (+ 16479us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.238717 (+ 4153us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.240081 (+ 1364us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":134500736,"tablet-delete.queue_time_us":3590513}
I20251027 21:08:11.240803 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1071) took 3657 ms. Trace:
I20251027 21:08:11.240861 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet cb95618d9fd54a4f8121b43e717d5616 (table default.Table_f2775123ffc8464eaf6d5e7d4ee692b0 [id=3298547a152045fdb51d360f4af22108]) successfully deleted
I20251027 21:08:11.240887 10041 rpcz_store.cc:276] 1027 21:08:07.583205 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.585172 (+ 1967us) service_pool.cc:225] Handling call
1027 21:08:11.240679 (+3655507us) ts_tablet_manager.cc:1151] Deleting tablet c2992034f072482382a4a593fb7f7aaa
1027 21:08:11.240689 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.240787 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3655208}
I20251027 21:08:11.241204 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1072) took 3641 ms. Trace:
W20251027 21:08:11.241284 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet c2992034f072482382a4a593fb7f7aaa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2992034f072482382a4a593fb7f7aaa already in progress: creating tablet
I20251027 21:08:11.241279 10041 rpcz_store.cc:276] 1027 21:08:07.600033 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.648209 (+ 48176us) service_pool.cc:225] Handling call
1027 21:08:11.241113 (+3592904us) ts_tablet_manager.cc:1151] Deleting tablet c0e2772d66404d74891e0893c49a66e0
1027 21:08:11.241121 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.241189 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3592627}
I20251027 21:08:11.241547 10041 tablet_replica.cc:333] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
W20251027 21:08:11.241654 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet c0e2772d66404d74891e0893c49a66e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0e2772d66404d74891e0893c49a66e0 already in progress: creating tablet
I20251027 21:08:11.241747 10041 raft_consensus.cc:2243] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.241919 10041 raft_consensus.cc:2272] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.242508 10041 ts_tablet_manager.cc:1916] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.248095 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet bd6152f7ac8a4123872098955b8837ac
I20251027 21:08:11.248157 9308 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 791376e559fb443cbd142c5724aa8a5e
I20251027 21:08:11.248502 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_67fb7650bd3745438426fc06514bcc9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:11.248777 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_db3806bdc5c2498294b20102a1767b93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:11.249094 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67fb7650bd3745438426fc06514bcc9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:11.249202 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db3806bdc5c2498294b20102a1767b93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.250188 9737 tablet_service.cc:1552] Processing DeleteTablet for tablet bd6152f7ac8a4123872098955b8837ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:11.246939 10032 tablet_bootstrap.cc:492] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:11.253849 10032 ts_tablet_manager.cc:1403] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.121s user 0.015s sys 0.023s
I20251027 21:08:11.254587 10032 raft_consensus.cc:359] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.254772 10032 raft_consensus.cc:385] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.254828 10032 raft_consensus.cc:740] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.255023 10032 consensus_queue.cc:260] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.255708 10032 ts_tablet_manager.cc:1434] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:11.246091 11425 raft_consensus.cc:493] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.255887 11425 raft_consensus.cc:515] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.256065 10032 tablet_bootstrap.cc:492] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:11.256762 11425 leader_election.cc:290] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.257736 10032 tablet_bootstrap.cc:654] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:11.253481 10041 ts_tablet_manager.cc:1929] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.250762 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet bd6152f7ac8a4123872098955b8837ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:11.261234 10041 log.cc:1199] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/27bd3b2a011648b2b8f2f92944513751
I20251027 21:08:11.261695 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "842c1a1298234d9ea2719bab29c54ae3" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:11.262050 10041 ts_tablet_manager.cc:1950] T 27bd3b2a011648b2b8f2f92944513751 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
W20251027 21:08:11.262423 9860 leader_election.cc:343] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.262918 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1073) took 3162 ms. Trace:
I20251027 21:08:11.263021 10041 rpcz_store.cc:276] 1027 21:08:08.100774 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.100994 (+ 220us) service_pool.cc:225] Handling call
1027 21:08:11.241510 (+3140516us) ts_tablet_manager.cc:1151] Deleting tablet 27bd3b2a011648b2b8f2f92944513751
1027 21:08:11.241518 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.245676 (+ 4158us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.253344 (+ 7668us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.262895 (+ 9551us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3140349}
I20251027 21:08:11.263517 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 27bd3b2a011648b2b8f2f92944513751 (table default.Table_4d2a5a0a0ac24d4bb907b89a6bee1aaf [id=8f16323b12524c71924a38207824dae4]) successfully deleted
I20251027 21:08:11.265100 10043 ts_tablet_manager.cc:1929] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.265209 10043 log.cc:1199] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/cef2bfee222f45e6b6f20c6d63ff44d4
I20251027 21:08:11.270381 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet bd6152f7ac8a4123872098955b8837ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:11.272188 10043 ts_tablet_manager.cc:1950] T cef2bfee222f45e6b6f20c6d63ff44d4 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:11.273613 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1046) took 4018 ms. Trace:
I20251027 21:08:11.273718 10040 rpcz_store.cc:276] 1027 21:08:07.255311 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.295069 (+ 39758us) service_pool.cc:225] Handling call
1027 21:08:10.281733 (+2986664us) ts_tablet_manager.cc:1151] Deleting tablet 8b30052ff19744a9bf4bfaaa1bc77b36
1027 21:08:10.281741 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.164696 (+882955us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.235009 (+ 70313us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.273594 (+ 38585us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2986550}
I20251027 21:08:11.274190 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 8b30052ff19744a9bf4bfaaa1bc77b36 (table default.Table_5dc88891b75b4f1ca7b3ffba0d2b1fda [id=b46f1e4f1e6d4bd8b4a3edd1d6787612]) successfully deleted
I20251027 21:08:11.274518 10040 tablet_replica.cc:333] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:11.274742 10040 raft_consensus.cc:2243] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.275061 10040 raft_consensus.cc:2272] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.275926 10040 ts_tablet_manager.cc:1916] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.277437 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1008) took 5571 ms. Trace:
I20251027 21:08:11.277657 10043 rpcz_store.cc:276] 1027 21:08:05.705996 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.278094 (+572098us) service_pool.cc:225] Handling call
1027 21:08:11.118828 (+4840734us) ts_tablet_manager.cc:1151] Deleting tablet cef2bfee222f45e6b6f20c6d63ff44d4
1027 21:08:11.118842 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.124602 (+ 5760us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.264980 (+140378us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.277390 (+ 12410us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4840413}
I20251027 21:08:11.278060 11412 consensus_queue.cc:1048] T 882c58a45d46402eabe96591a7c1de71 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:11.278759 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet cef2bfee222f45e6b6f20c6d63ff44d4 (table default.Table_c3f0afa8e76249b69b9fbe4b3f67d020 [id=43518e3f60164df7a812afb41f6c4936]) successfully deleted
I20251027 21:08:11.281702 10043 tablet_replica.cc:333] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:11.281937 10043 raft_consensus.cc:2243] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.282131 10043 raft_consensus.cc:2272] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.282761 10043 ts_tablet_manager.cc:1916] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:11.288614 9273 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu pid 9349: Timed out: CheckLeaks RPC to 127.9.14.125:46313 timed out after 1.000s (SENT)
I20251027 21:08:11.288769 9273 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu with pid 9349
I20251027 21:08:11.291815 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_258950ebb36249149222ca71cb3278e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:11.292299 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_258950ebb36249149222ca71cb3278e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.292601 11411 raft_consensus.cc:493] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.292717 11411 raft_consensus.cc:515] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:11.293299 11411 leader_election.cc:290] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:11.293932 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "631ff72613734913818a3cad24f20dbc" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:11.294452 9566 leader_election.cc:343] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.295560 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_00304e11dcc548499ea100119c5c976e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:11.296115 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00304e11dcc548499ea100119c5c976e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:11.305275 9568 leader_election.cc:343] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 631ff72613734913818a3cad24f20dbc
I20251027 21:08:11.305531 9568 leader_election.cc:304] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:11.306304 11411 raft_consensus.cc:2749] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:11.306668 10041 tablet_replica.cc:333] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.307040 10041 raft_consensus.cc:2243] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.307483 10041 raft_consensus.cc:2272] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.308100 10041 ts_tablet_manager.cc:1916] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.332849 11447 tablet.cc:1722] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:11.333446 11392 tablet.cc:1722] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:11.334653 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_109b65ba33734a5f8ffb302c8fc8caaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:11.335147 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_109b65ba33734a5f8ffb302c8fc8caaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.336465 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0cc61333cd424ee2b2f850d933fc6340" } new_table_name: "default.Table_8e90db76473340da90948310ba7e320f" modify_external_catalogs: true
I20251027 21:08:11.339021 11447 tablet.cc:1722] T 9fa39697360c46138ef4cd21e8c7a950 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251027 21:08:11.341074 10041 ts_tablet_manager.cc:1929] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.341190 10041 log.cc:1199] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/28a2e85021394f9d8a9ea524959b52a1
I20251027 21:08:11.341490 10041 ts_tablet_manager.cc:1950] T 28a2e85021394f9d8a9ea524959b52a1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.341521 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c3120318fe0f4feda2d71e836f0e02e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:11.342036 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3120318fe0f4feda2d71e836f0e02e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.348474 10043 ts_tablet_manager.cc:1929] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.348737 10043 log.cc:1199] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/01e9db9eaf8e41c6a2ee13b9ed127421
I20251027 21:08:11.349167 10029 tablet_bootstrap.cc:492] T 7425b6b0184e4d0b871777b25d904ac4 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:11.349335 10029 ts_tablet_manager.cc:1403] T 7425b6b0184e4d0b871777b25d904ac4 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.286s user 0.009s sys 0.008s
I20251027 21:08:11.350353 10029 raft_consensus.cc:359] T 7425b6b0184e4d0b871777b25d904ac4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.350602 10029 raft_consensus.cc:385] T 7425b6b0184e4d0b871777b25d904ac4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.350762 10029 raft_consensus.cc:740] T 7425b6b0184e4d0b871777b25d904ac4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.351297 11392 tablet.cc:1722] T 9fa39697360c46138ef4cd21e8c7a950 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251027 21:08:11.351876 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f40312591b4147acbe2ba36ff49b01f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:11.352368 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f40312591b4147acbe2ba36ff49b01f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.355862 10029 consensus_queue.cc:260] T 7425b6b0184e4d0b871777b25d904ac4 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:11.358893 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b261415f48a43229fbb342ac7f7a677 consensus state has no leader
I20251027 21:08:11.359380 10029 ts_tablet_manager.cc:1434] T 7425b6b0184e4d0b871777b25d904ac4 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251027 21:08:11.359725 10029 tablet_bootstrap.cc:492] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:11.361435 10029 tablet_bootstrap.cc:654] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:11.362231 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6bb3152125264bc6a28eda3f88cee649" } new_table_name: "default.Table_69ff6879149248aa9f620f4398ec1e3e" modify_external_catalogs: true
I20251027 21:08:11.369122 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1074) took 3268 ms. Trace:
I20251027 21:08:11.369223 10041 rpcz_store.cc:276] 1027 21:08:08.100878 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.101205 (+ 327us) service_pool.cc:225] Handling call
1027 21:08:11.285074 (+3183869us) ts_tablet_manager.cc:1151] Deleting tablet 28a2e85021394f9d8a9ea524959b52a1
1027 21:08:11.304706 (+ 19632us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.334202 (+ 29496us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.340988 (+ 6786us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.369101 (+ 28113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3162017}
I20251027 21:08:11.369520 10041 tablet_replica.cc:333] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.369694 10041 raft_consensus.cc:2243] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.369827 10041 raft_consensus.cc:2272] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.370322 10041 ts_tablet_manager.cc:1916] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.376528 10043 ts_tablet_manager.cc:1950] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:11.376886 11447 tablet.cc:1717] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678: Skipping requested alter to schema version 2, tablet already version 2
I20251027 21:08:11.377103 11392 tablet.cc:1717] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14: Skipping requested alter to schema version 2, tablet already version 2
I20251027 21:08:11.257136 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "842c1a1298234d9ea2719bab29c54ae3" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:11.378127 9859 leader_election.cc:343] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.378280 9859 leader_election.cc:304] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:11.378453 9859 raft_consensus.cc:2686] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251027 21:08:11.378814 9298 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 57ef07c764664deaa4b2e9bcae7e7973 (table default.Table_7d6b8d0ba5ad42fa9fdbcf13c7647d41 [id=a541c8d30a9540f5a7d4dd98a5d3739a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 57ef07c764664deaa4b2e9bcae7e7973 consensus state has no leader
I20251027 21:08:11.378511 10041 ts_tablet_manager.cc:1929] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.379346 10041 log.cc:1199] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/842c1a1298234d9ea2719bab29c54ae3
I20251027 21:08:11.379629 10041 ts_tablet_manager.cc:1950] T 842c1a1298234d9ea2719bab29c54ae3 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.380299 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1075) took 3279 ms. Trace:
I20251027 21:08:11.380375 10041 rpcz_store.cc:276] 1027 21:08:08.100975 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.101364 (+ 389us) service_pool.cc:225] Handling call
1027 21:08:11.369479 (+3268115us) ts_tablet_manager.cc:1151] Deleting tablet 842c1a1298234d9ea2719bab29c54ae3
1027 21:08:11.369487 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.373218 (+ 3731us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.378418 (+ 5200us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.380282 (+ 1864us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3267973}
I20251027 21:08:11.380934 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1042) took 4936 ms. Trace:
I20251027 21:08:11.381062 10043 rpcz_store.cc:276] 1027 21:08:06.444741 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.735737 (+290996us) service_pool.cc:225] Handling call
1027 21:08:11.277968 (+4542231us) ts_tablet_manager.cc:1151] Deleting tablet 01e9db9eaf8e41c6a2ee13b9ed127421
1027 21:08:11.281660 (+ 3692us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.310799 (+ 29139us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.348060 (+ 37261us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.380914 (+ 32854us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4516614}
I20251027 21:08:11.381668 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 01e9db9eaf8e41c6a2ee13b9ed127421 (table default.Table_11fbadd9eed14cdba7664715e389f7e5 [id=83ec44dba5c444aba48846811a5f5fc9]) successfully deleted
I20251027 21:08:11.381750 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1046) took 4756 ms. Trace:
I20251027 21:08:11.381893 10043 rpcz_store.cc:276] 1027 21:08:06.625341 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.791761 (+166420us) service_pool.cc:225] Handling call
1027 21:08:11.381556 (+4589795us) ts_tablet_manager.cc:1151] Deleting tablet c548d2413a5940c8b114ced667b4dc41
1027 21:08:11.381566 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.381698 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4589600}
I20251027 21:08:11.382614 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1088) took 2921 ms. Trace:
I20251027 21:08:11.382695 10041 rpcz_store.cc:276] 1027 21:08:08.461554 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.461604 (+ 50us) service_pool.cc:225] Handling call
1027 21:08:11.382509 (+2920905us) ts_tablet_manager.cc:1151] Deleting tablet 750b27c74ba3473dad743d60c21912c5
1027 21:08:11.382519 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.382598 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2920422}
W20251027 21:08:11.384075 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet c548d2413a5940c8b114ced667b4dc41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c548d2413a5940c8b114ced667b4dc41 already in progress: creating tablet
I20251027 21:08:11.385330 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1089) took 2845 ms. Trace:
I20251027 21:08:11.385411 10041 rpcz_store.cc:276] 1027 21:08:08.539812 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.540098 (+ 286us) service_pool.cc:225] Handling call
1027 21:08:11.385230 (+2845132us) ts_tablet_manager.cc:1151] Deleting tablet a21a6a3725934266bbfdfc70b40de687
1027 21:08:11.385240 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.385315 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2844880}
I20251027 21:08:11.386209 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_499a1282ff2b4f10918afa3d2c096d07" } new_table_name: "default.Table_480db6390424454995a7b920981df986" modify_external_catalogs: true
I20251027 21:08:11.386329 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1090) took 2846 ms. Trace:
I20251027 21:08:11.386387 10041 rpcz_store.cc:276] 1027 21:08:08.539943 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.540400 (+ 457us) service_pool.cc:225] Handling call
1027 21:08:11.386235 (+2845835us) ts_tablet_manager.cc:1151] Deleting tablet e53b2c28c75b43bdafa0d20f4ad61fcc
1027 21:08:11.386243 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.386314 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2845630}
I20251027 21:08:11.387230 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1091) took 2847 ms. Trace:
I20251027 21:08:11.387295 10041 rpcz_store.cc:276] 1027 21:08:08.540052 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.540635 (+ 583us) service_pool.cc:225] Handling call
1027 21:08:11.387139 (+2846504us) ts_tablet_manager.cc:1151] Deleting tablet 8827e4746ecc408a8fea98b6b5ddd57b
1027 21:08:11.387147 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.387215 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2846388}
I20251027 21:08:11.388089 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1092) took 2787 ms. Trace:
I20251027 21:08:11.388020 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_548bc7accdaa4dd8bfb6373d3d15667d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:11.388151 10041 rpcz_store.cc:276] 1027 21:08:08.600894 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.600957 (+ 63us) service_pool.cc:225] Handling call
1027 21:08:11.387999 (+2787042us) ts_tablet_manager.cc:1151] Deleting tablet cc97e55cfbcb41ec8c6c7410bbf14fb3
1027 21:08:11.388007 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.388075 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2786797}
I20251027 21:08:11.388831 10041 tablet_replica.cc:333] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.389029 10041 raft_consensus.cc:2243] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.389199 10041 raft_consensus.cc:2272] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.389737 10041 ts_tablet_manager.cc:1916] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.390854 10031 tablet_bootstrap.cc:654] T f775a7e42bdc4910bdb604b26a28a7f4 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:11.394857 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c3d07b891a248efb0860cd000e44432 consensus state has no leader
I20251027 21:08:11.395185 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_38420733ed6d43799874a162313d5828"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:11.405895 10041 ts_tablet_manager.cc:1929] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 21:08:11.406013 10041 log.cc:1199] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/94f2819dd0394530ae93754bdd1e482f
I20251027 21:08:11.406368 10041 ts_tablet_manager.cc:1950] T 94f2819dd0394530ae93754bdd1e482f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.407151 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1093) took 2794 ms. Trace:
I20251027 21:08:11.407238 10041 rpcz_store.cc:276] 1027 21:08:08.612907 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.612957 (+ 50us) service_pool.cc:225] Handling call
1027 21:08:11.388788 (+2775831us) ts_tablet_manager.cc:1151] Deleting tablet 94f2819dd0394530ae93754bdd1e482f
1027 21:08:11.388797 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.402733 (+ 13936us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.405806 (+ 3073us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.407131 (+ 1325us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2775643}
I20251027 21:08:11.407747 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1047) took 4741 ms. Trace:
I20251027 21:08:11.410252 10032 tablet_bootstrap.cc:492] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
W20251027 21:08:11.410840 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b7e9c0bb583f438881c82f6ad308407e consensus state has no leader
I20251027 21:08:11.410117 10029 tablet_bootstrap.cc:492] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:11.410708 10041 tablet_replica.cc:333] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.411705 10041 raft_consensus.cc:2243] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.411883 10041 raft_consensus.cc:2272] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.408977 10043 rpcz_store.cc:276] 1027 21:08:06.665948 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.791968 (+126020us) service_pool.cc:225] Handling call
1027 21:08:11.405458 (+4613490us) ts_tablet_manager.cc:1151] Deleting tablet 4c67c5cb3b244feb8478f9594b834fbf
1027 21:08:11.405471 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.407110 (+ 1639us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4612983}
I20251027 21:08:11.412374 10041 ts_tablet_manager.cc:1916] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:11.412676 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 4c67c5cb3b244feb8478f9594b834fbf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c67c5cb3b244feb8478f9594b834fbf already in progress: creating tablet
I20251027 21:08:11.369618 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 28a2e85021394f9d8a9ea524959b52a1 (table default.Table_b730ec55ffb24974928bd0b0d9416460 [id=d72c1295ca1a4ec6955476a6eb5b0bb4]) successfully deleted
W20251027 21:08:11.413547 9299 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 43937us
I20251027 21:08:11.413802 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 842c1a1298234d9ea2719bab29c54ae3 (table default.Table_773dc0a1742a4f9194e9b26ea3d07123 [id=4372567fda9b4a178b8adc2cde3e1704]) successfully deleted
W20251027 21:08:11.414191 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet a21a6a3725934266bbfdfc70b40de687 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a21a6a3725934266bbfdfc70b40de687 already in progress: creating tablet
I20251027 21:08:11.414168 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_fb3083bf1d0a4db382cd40ee844e8d50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:11.415225 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet e53b2c28c75b43bdafa0d20f4ad61fcc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e53b2c28c75b43bdafa0d20f4ad61fcc already in progress: creating tablet
W20251027 21:08:11.415576 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 8827e4746ecc408a8fea98b6b5ddd57b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8827e4746ecc408a8fea98b6b5ddd57b already in progress: creating tablet
W20251027 21:08:11.415884 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet cc97e55cfbcb41ec8c6c7410bbf14fb3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc97e55cfbcb41ec8c6c7410bbf14fb3 already in progress: creating tablet
I20251027 21:08:11.416179 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 94f2819dd0394530ae93754bdd1e482f (table default.Table_f46fff92cdc244f6a020795467099e74 [id=63147f61013c4d6dbc96fd25769b6136]) successfully deleted
W20251027 21:08:11.417039 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38420733ed6d43799874a162313d5828 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:11.417057 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb3083bf1d0a4db382cd40ee844e8d50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.416810 10029 ts_tablet_manager.cc:1403] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.001s
W20251027 21:08:11.419440 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_548bc7accdaa4dd8bfb6373d3d15667d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.419332 10040 ts_tablet_manager.cc:1929] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.419719 10040 log.cc:1199] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/a26ded6012174348b7295fc7c0e05e6e
I20251027 21:08:11.420639 10029 raft_consensus.cc:359] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.416487 10032 ts_tablet_manager.cc:1403] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.160s user 0.000s sys 0.013s
I20251027 21:08:11.423175 10032 raft_consensus.cc:359] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.424582 10032 raft_consensus.cc:385] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.424894 10032 raft_consensus.cc:740] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.429216 10029 raft_consensus.cc:385] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.429339 10029 raft_consensus.cc:740] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.429582 10029 consensus_queue.cc:260] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.430051 10032 consensus_queue.cc:260] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.430552 10041 ts_tablet_manager.cc:1929] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.430661 10041 log.cc:1199] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/d2203ee62f7e404c95c5f86b5ae85feb
I20251027 21:08:11.430946 10041 ts_tablet_manager.cc:1950] T d2203ee62f7e404c95c5f86b5ae85feb P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.431643 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1098) took 2738 ms. Trace:
I20251027 21:08:11.431649 9305 master_service.cc:954] ReplaceTablet: received request to replace tablet e4b84d8c0ee74dcbaee8aeb49aa30c57 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:11.431728 10041 rpcz_store.cc:276] 1027 21:08:08.693101 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.693139 (+ 38us) service_pool.cc:225] Handling call
1027 21:08:11.410662 (+2717523us) ts_tablet_manager.cc:1151] Deleting tablet d2203ee62f7e404c95c5f86b5ae85feb
1027 21:08:11.410671 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.427128 (+ 16457us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.430461 (+ 3333us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.431624 (+ 1163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2717346}
I20251027 21:08:11.432278 10041 tablet_replica.cc:333] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.432480 10041 raft_consensus.cc:2243] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.432628 10041 raft_consensus.cc:2272] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.433106 10041 ts_tablet_manager.cc:1916] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.433676 10029 ts_tablet_manager.cc:1434] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20251027 21:08:11.422392 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1048) took 4674 ms. Trace:
I20251027 21:08:11.435938 10043 rpcz_store.cc:276] 1027 21:08:06.747687 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.792097 (+ 44410us) service_pool.cc:225] Handling call
1027 21:08:11.422270 (+4630173us) ts_tablet_manager.cc:1151] Deleting tablet 17a19ce92da8448bba471f94e3ee4647
1027 21:08:11.422281 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.422373 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4630037}
W20251027 21:08:11.436425 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 17a19ce92da8448bba471f94e3ee4647 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17a19ce92da8448bba471f94e3ee4647 already in progress: creating tablet
I20251027 21:08:11.436852 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1049) took 4689 ms. Trace:
I20251027 21:08:11.436966 10043 rpcz_store.cc:276] 1027 21:08:06.747756 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.792232 (+ 44476us) service_pool.cc:225] Handling call
1027 21:08:11.436725 (+4644493us) ts_tablet_manager.cc:1151] Deleting tablet 35b99f8c57e847b1848cbdea9e50da11
1027 21:08:11.436734 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.436834 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4644389}
W20251027 21:08:11.437424 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 35b99f8c57e847b1848cbdea9e50da11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35b99f8c57e847b1848cbdea9e50da11 already in progress: creating tablet
I20251027 21:08:11.436879 11446 tablet.cc:1722] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:11.438077 10040 ts_tablet_manager.cc:1950] T a26ded6012174348b7295fc7c0e05e6e P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:11.438313 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1050) took 4690 ms. Trace:
I20251027 21:08:11.438401 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1042) took 4277 ms. Trace:
I20251027 21:08:11.438445 10043 rpcz_store.cc:276] 1027 21:08:06.747810 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.792339 (+ 44529us) service_pool.cc:225] Handling call
1027 21:08:11.438120 (+4645781us) ts_tablet_manager.cc:1151] Deleting tablet 933fb952d66d4cb1ade70c334fbbb3e8
1027 21:08:11.438206 (+ 86us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.438295 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4645691}
I20251027 21:08:11.438481 10040 rpcz_store.cc:276] 1027 21:08:07.160459 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.294578 (+134119us) service_pool.cc:225] Handling call
1027 21:08:11.274468 (+3979890us) ts_tablet_manager.cc:1151] Deleting tablet a26ded6012174348b7295fc7c0e05e6e
1027 21:08:11.274479 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.280168 (+ 5689us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.419232 (+139064us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.438385 (+ 19153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3976049}
I20251027 21:08:11.438803 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet a26ded6012174348b7295fc7c0e05e6e (table default.Table_69f618a7f3c74a1d87cb5e0c17f38f94 [id=d37bad05f12e4d0da8934baeb7627de6]) successfully deleted
I20251027 21:08:11.438899 10040 tablet_replica.cc:333] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:11.439060 10040 raft_consensus.cc:2243] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
W20251027 21:08:11.439286 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 933fb952d66d4cb1ade70c334fbbb3e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 933fb952d66d4cb1ade70c334fbbb3e8 already in progress: creating tablet
I20251027 21:08:11.439358 10040 pending_rounds.cc:70] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
I20251027 21:08:11.439399 10040 pending_rounds.cc:77] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510681315139584 op_type: NO_OP noop_request { }
I20251027 21:08:11.439479 10040 raft_consensus.cc:2889] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:11.439553 10040 raft_consensus.cc:2272] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.439818 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1052) took 4603 ms. Trace:
I20251027 21:08:11.439935 10043 rpcz_store.cc:276] 1027 21:08:06.836193 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.840515 (+ 4322us) service_pool.cc:225] Handling call
1027 21:08:11.439715 (+4599200us) ts_tablet_manager.cc:1151] Deleting tablet 8dc358445526415d8cd4f1ac6779126d
1027 21:08:11.439723 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.439804 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4599037}
I20251027 21:08:11.440068 10040 ts_tablet_manager.cc:1916] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:11.440421 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 8dc358445526415d8cd4f1ac6779126d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dc358445526415d8cd4f1ac6779126d already in progress: creating tablet
I20251027 21:08:11.440358 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8e90db76473340da90948310ba7e320f" } new_table_name: "default.Table_ca1bbd50b8e449c2ad12a3fbb5690ad8" modify_external_catalogs: true
I20251027 21:08:11.441685 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1053) took 4603 ms. Trace:
I20251027 21:08:11.441793 10043 rpcz_store.cc:276] 1027 21:08:06.838582 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.840694 (+ 2112us) service_pool.cc:225] Handling call
1027 21:08:11.441595 (+4600901us) ts_tablet_manager.cc:1151] Deleting tablet db2735604dc14039810df4ecd330c671
1027 21:08:11.441603 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.441670 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4600781}
W20251027 21:08:11.442323 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet db2735604dc14039810df4ecd330c671 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db2735604dc14039810df4ecd330c671 already in progress: creating tablet
I20251027 21:08:11.442761 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1051) took 4606 ms. Trace:
I20251027 21:08:11.442881 10043 rpcz_store.cc:276] 1027 21:08:06.836076 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.840319 (+ 4243us) service_pool.cc:225] Handling call
1027 21:08:11.442653 (+4602334us) ts_tablet_manager.cc:1151] Deleting tablet 4a621d5246f14997a2f2952ab640efc8
1027 21:08:11.442664 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.442744 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4601730}
W20251027 21:08:11.443339 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 4a621d5246f14997a2f2952ab640efc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a621d5246f14997a2f2952ab640efc8 already in progress: creating tablet
I20251027 21:08:11.443708 10043 tablet_replica.cc:333] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:11.443984 10043 raft_consensus.cc:2243] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.444200 10043 raft_consensus.cc:2272] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.444849 10043 ts_tablet_manager.cc:1916] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.445848 10041 ts_tablet_manager.cc:1929] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.446239 10041 log.cc:1199] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/e6ed9d7a700446edafcd0daa02aa7ef6
I20251027 21:08:11.447147 10041 ts_tablet_manager.cc:1950] T e6ed9d7a700446edafcd0daa02aa7ef6 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.449577 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1094) took 2756 ms. Trace:
I20251027 21:08:11.449862 10041 rpcz_store.cc:276] 1027 21:08:08.692645 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.693142 (+ 497us) service_pool.cc:225] Handling call
1027 21:08:11.432230 (+2739088us) ts_tablet_manager.cc:1151] Deleting tablet e6ed9d7a700446edafcd0daa02aa7ef6
1027 21:08:11.432238 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.437058 (+ 4820us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.445553 (+ 8495us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.449465 (+ 3912us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":279,"tablet-delete.queue_time_us":2738913}
I20251027 21:08:11.430585 10032 ts_tablet_manager.cc:1434] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20251027 21:08:11.456286 10041 tablet_replica.cc:333] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.456610 10041 raft_consensus.cc:2243] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.456843 10041 raft_consensus.cc:2272] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.457762 10041 ts_tablet_manager.cc:1916] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.457988 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_69ef4f8f04d042a18d248c08958ee9f7" } modify_external_catalogs: true
I20251027 21:08:11.458153 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_69ef4f8f04d042a18d248c08958ee9f7" } modify_external_catalogs: true
I20251027 21:08:11.459843 11446 tablet.cc:1717] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218: Skipping requested alter to schema version 2, tablet already version 2
I20251027 21:08:11.459640 10032 tablet_bootstrap.cc:492] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:08:11.464996 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b261415f48a43229fbb342ac7f7a677 consensus state has no leader
I20251027 21:08:11.465044 10031 tablet_bootstrap.cc:492] T f775a7e42bdc4910bdb604b26a28a7f4 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:11.465152 10031 ts_tablet_manager.cc:1403] T f775a7e42bdc4910bdb604b26a28a7f4 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.247s user 0.004s sys 0.004s
I20251027 21:08:11.466012 10031 raft_consensus.cc:359] T f775a7e42bdc4910bdb604b26a28a7f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:11.466180 10031 raft_consensus.cc:385] T f775a7e42bdc4910bdb604b26a28a7f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.466224 10031 raft_consensus.cc:740] T f775a7e42bdc4910bdb604b26a28a7f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.466392 10031 consensus_queue.cc:260] T f775a7e42bdc4910bdb604b26a28a7f4 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:11.466859 10031 ts_tablet_manager.cc:1434] T f775a7e42bdc4910bdb604b26a28a7f4 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:11.467073 10031 tablet_bootstrap.cc:492] T a033551a591146039b017b4193b221ca P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:11.432085 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet d2203ee62f7e404c95c5f86b5ae85feb (table default.Table_0432e7918b494dccb2f6b921657c0d81 [id=f773d65a07a4473ab8b87cd82ac50bcf]) successfully deleted
I20251027 21:08:11.468127 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet e6ed9d7a700446edafcd0daa02aa7ef6 (table default.Table_c814f5ad5b6f46f2aa8fc2ecf7ce2d73 [id=af0612025c07415fa46cf2fad2609787]) successfully deleted
I20251027 21:08:11.469305 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_cdf514bc48a4478da802d1bbfd0f1c71" } modify_external_catalogs: true
I20251027 21:08:11.469444 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_cdf514bc48a4478da802d1bbfd0f1c71" } modify_external_catalogs: true
I20251027 21:08:11.472306 10032 tablet_bootstrap.cc:654] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:11.453369 10029 tablet_bootstrap.cc:492] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
W20251027 21:08:11.465503 9568 consensus_peers.cc:597] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2d26784bbb24ae3893103802c41affb. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:11.476854 10029 tablet_bootstrap.cc:654] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:11.477388 11142 raft_consensus.cc:493] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fbd567f929a4f6d8e4d78feef7ed678)
I20251027 21:08:11.477717 11142 raft_consensus.cc:515] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:11.478319 11142 leader_election.cc:290] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:11.479633 9568 leader_election.cc:343] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 96a680e8f16343b1908829a67e960b60
I20251027 21:08:11.479931 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96a680e8f16343b1908829a67e960b60" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:11.480405 9566 leader_election.cc:343] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.480516 9566 leader_election.cc:304] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:11.481002 11411 raft_consensus.cc:2749] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251027 21:08:11.485401 10031 tablet_bootstrap.cc:654] T a033551a591146039b017b4193b221ca P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:11.489012 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 0d0099e7b3bd403590ca5c9c73198853
I20251027 21:08:11.490221 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d0099e7b3bd403590ca5c9c73198853 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:11.490267 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d0099e7b3bd403590ca5c9c73198853 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:11.476670 10041 ts_tablet_manager.cc:1929] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.490455 10041 log.cc:1199] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/c9c86253ce914916bee6584675a82067
I20251027 21:08:11.490767 10041 ts_tablet_manager.cc:1950] T c9c86253ce914916bee6584675a82067 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.491523 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1095) took 2798 ms. Trace:
I20251027 21:08:11.491606 10041 rpcz_store.cc:276] 1027 21:08:08.692774 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.695803 (+ 3029us) service_pool.cc:225] Handling call
1027 21:08:11.451136 (+2755333us) ts_tablet_manager.cc:1151] Deleting tablet c9c86253ce914916bee6584675a82067
1027 21:08:11.456208 (+ 5072us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.473717 (+ 17509us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.476585 (+ 2868us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.491503 (+ 14918us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2755135}
I20251027 21:08:11.492113 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet c9c86253ce914916bee6584675a82067 (table default.Table_4469fd75838845e9b60fd07ddc845f5c [id=5076832e735b4894a0ce4f84c5606b83]) successfully deleted
W20251027 21:08:11.492466 9568 consensus_peers.cc:597] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94f2819dd0394530ae93754bdd1e482f. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:11.493723 10032 tablet_bootstrap.cc:492] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:11.493916 10032 ts_tablet_manager.cc:1403] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.002s
I20251027 21:08:11.494652 10032 raft_consensus.cc:359] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.494874 10032 raft_consensus.cc:385] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.494964 10032 raft_consensus.cc:740] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.495225 10032 consensus_queue.cc:260] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.495765 10032 ts_tablet_manager.cc:1434] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:11.496063 10032 tablet_bootstrap.cc:492] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:11.496937 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b29f906d3a774523839fd36ba1f5d2e4" } new_table_name: "default.Table_9696ac35373c4b46b61c486cf76e482e" modify_external_catalogs: true
I20251027 21:08:11.498384 10032 tablet_bootstrap.cc:654] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:11.499121 9308 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 93a31187eca14b8797737936a80e7292 (table default.Table_92fa91c5b7c24ec2bd881c758ba33889 [id=69a8a630d7ab4b2d8c7b580b41ebe3ba]). Expected version 1 got 0
I20251027 21:08:11.500445 9310 catalog_manager.cc:5649] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:11.500986 9306 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_3e3e0c17a3bf452fbb217c6e3ee8bdcb [id=2eb093cd58e643a58b14e564d5914fbc] alter complete (version 1)
I20251027 21:08:11.502544 10041 tablet_replica.cc:333] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.502733 10041 raft_consensus.cc:2243] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.502868 10041 raft_consensus.cc:2272] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.503255 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 191009830c30467b9c5ab39e13b7c267
I20251027 21:08:11.503334 10041 ts_tablet_manager.cc:1916] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.506669 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet 191009830c30467b9c5ab39e13b7c267 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:11.506670 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 958dafe1e6844f19902524e8fa27d69a (DEFAULT_TABLE table=default.Table_406cb517a4d34f0c921987c19227668b [id=69c1eb2307f24390bdf052bc2d0d1c85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.507115 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 958dafe1e6844f19902524e8fa27d69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.510321 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet 191009830c30467b9c5ab39e13b7c267 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:11.510912 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 278d8f448ef144148cabfc3a57bd24fd
I20251027 21:08:11.512467 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet 278d8f448ef144148cabfc3a57bd24fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:11.512683 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet 278d8f448ef144148cabfc3a57bd24fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:11.515770 10032 tablet_bootstrap.cc:492] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:11.515955 10032 ts_tablet_manager.cc:1403] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20251027 21:08:11.516752 10032 raft_consensus.cc:359] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.516983 10032 raft_consensus.cc:385] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.517079 10032 raft_consensus.cc:740] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.489887 9737 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d0099e7b3bd403590ca5c9c73198853 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:11.493428 11437 raft_consensus.cc:493] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.517335 10032 consensus_queue.cc:260] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.517565 11437 raft_consensus.cc:515] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:08:11.517721 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4540d5fbb5bc4611bc0104bf840bd12d consensus state has no leader
I20251027 21:08:11.517858 10032 ts_tablet_manager.cc:1434] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:11.518103 10032 tablet_bootstrap.cc:492] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:11.501968 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 76e4fb2bea7f4a4696fe9f6bb892bccc with delete_type TABLET_DATA_DELETED (replaced by tablet 5cb8ca3a50c148e6b602e949650353f8) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:11.518710 11437 leader_election.cc:290] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:11.519104 9710 leader_election.cc:343] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 01e9db9eaf8e41c6a2ee13b9ed127421
W20251027 21:08:11.519646 9711 leader_election.cc:343] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 01e9db9eaf8e41c6a2ee13b9ed127421
I20251027 21:08:11.519723 10032 tablet_bootstrap.cc:654] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:11.519773 9711 leader_election.cc:304] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:11.519984 11437 raft_consensus.cc:2749] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:11.504189 10040 ts_tablet_manager.cc:1929] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:11.509289 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 191009830c30467b9c5ab39e13b7c267 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:11.509595 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 958dafe1e6844f19902524e8fa27d69a (DEFAULT_TABLE table=default.Table_406cb517a4d34f0c921987c19227668b [id=69c1eb2307f24390bdf052bc2d0d1c85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.513391 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 278d8f448ef144148cabfc3a57bd24fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:11.522435 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 958dafe1e6844f19902524e8fa27d69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.525095 10041 ts_tablet_manager.cc:1929] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.525190 10041 log.cc:1199] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/cdbde6272af147829ecadd67d634f049
I20251027 21:08:11.525475 10041 ts_tablet_manager.cc:1950] T cdbde6272af147829ecadd67d634f049 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.526173 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1096) took 2833 ms. Trace:
I20251027 21:08:11.526247 10041 rpcz_store.cc:276] 1027 21:08:08.692883 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.696035 (+ 3152us) service_pool.cc:225] Handling call
1027 21:08:11.502490 (+2806455us) ts_tablet_manager.cc:1151] Deleting tablet cdbde6272af147829ecadd67d634f049
1027 21:08:11.502502 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.513020 (+ 10518us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.524997 (+ 11977us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.526136 (+ 1139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2806077}
I20251027 21:08:11.526546 10041 tablet_replica.cc:333] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.526624 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 90f96c21da9944f1ade05d35787d47e2
I20251027 21:08:11.526733 10041 raft_consensus.cc:2243] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.526878 10041 raft_consensus.cc:2272] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.527328 10041 ts_tablet_manager.cc:1916] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.528323 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 90f96c21da9944f1ade05d35787d47e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:11.528934 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet 90f96c21da9944f1ade05d35787d47e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:11.528991 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet cdbde6272af147829ecadd67d634f049 (table default.Table_340ce2d28b454548af6c487cb35008ac [id=abc18d8f40b647c6afa5cbabd1cdaba3]) successfully deleted
I20251027 21:08:11.529294 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_368703da645247fb87af3db268932e92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:11.529799 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_368703da645247fb87af3db268932e92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.533010 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet 532ea47b15b544dfb05ddb2bdacf3953 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c05b69509414fe8bfc809cd307d5f2e) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:11.534394 11400 raft_consensus.cc:493] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.534520 11400 raft_consensus.cc:515] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.535035 11400 leader_election.cc:290] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.535533 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "423047dfbf084bfcb881efeb0352f88d" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:11.535944 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "423047dfbf084bfcb881efeb0352f88d" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:11.536685 9859 leader_election.cc:343] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.536813 10040 log.cc:1199] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/c3b9a461567a48fcb4af514ecdf618b4
W20251027 21:08:11.537235 9860 leader_election.cc:343] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.537328 9860 leader_election.cc:304] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:11.537533 10040 ts_tablet_manager.cc:1950] T c3b9a461567a48fcb4af514ecdf618b4 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:11.537751 11400 raft_consensus.cc:2749] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:11.539674 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1047) took 4067 ms. Trace:
I20251027 21:08:11.539875 10040 rpcz_store.cc:276] 1027 21:08:07.471740 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.471789 (+ 49us) service_pool.cc:225] Handling call
1027 21:08:11.438854 (+3967065us) ts_tablet_manager.cc:1151] Deleting tablet c3b9a461567a48fcb4af514ecdf618b4
1027 21:08:11.438863 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.467595 (+ 28732us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.504087 (+ 36492us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.539648 (+ 35561us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3966876}
I20251027 21:08:11.540477 10040 tablet_replica.cc:333] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:11.540778 11400 raft_consensus.cc:493] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.540915 10040 raft_consensus.cc:2243] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:11.540912 11400 raft_consensus.cc:515] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.541328 10040 pending_rounds.cc:70] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
I20251027 21:08:11.541375 11400 leader_election.cc:290] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.541391 10040 pending_rounds.cc:77] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510682491592704 op_type: NO_OP noop_request { }
I20251027 21:08:11.541852 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "532ea47b15b544dfb05ddb2bdacf3953" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:11.542212 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "532ea47b15b544dfb05ddb2bdacf3953" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:11.542438 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_783ec376f9f34eefadffe56b522df2eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:11.542528 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_243113a951ad4de7a77f46d9c89f5ce4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:11.541513 10040 raft_consensus.cc:2889] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20251027 21:08:11.542966 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_783ec376f9f34eefadffe56b522df2eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.543015 10040 raft_consensus.cc:2272] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:11.543109 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_243113a951ad4de7a77f46d9c89f5ce4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:11.544317 9859 leader_election.cc:343] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.544469 11425 raft_consensus.cc:493] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.544576 11425 raft_consensus.cc:515] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.545055 11425 leader_election.cc:290] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.545975 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7df6f0acec34d399aa25b78c3ed7c7a" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:11.546519 9859 leader_election.cc:343] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.547290 10041 ts_tablet_manager.cc:1929] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.547425 10041 log.cc:1199] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/0e4d35f229bc46e1859ded7c8196ab9d
I20251027 21:08:11.547732 10041 ts_tablet_manager.cc:1950] T 0e4d35f229bc46e1859ded7c8196ab9d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.547731 9308 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 4bc5fbdbbea0406a961e1a4df51b8762 (table default.Table_59b396d426544bb7800dacba4e8212f0 [id=4154657a736742b8a4f7dfbd254209c4]). Expected version 1 got 0
I20251027 21:08:11.548492 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1097) took 2855 ms. Trace:
W20251027 21:08:11.548441 9278 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 68462.6us
I20251027 21:08:11.548595 10041 rpcz_store.cc:276] 1027 21:08:08.693006 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.696137 (+ 3131us) service_pool.cc:225] Handling call
1027 21:08:11.526505 (+2830368us) ts_tablet_manager.cc:1151] Deleting tablet 0e4d35f229bc46e1859ded7c8196ab9d
1027 21:08:11.526513 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.535909 (+ 9396us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.547190 (+ 11281us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.548470 (+ 1280us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2829753}
I20251027 21:08:11.548893 10041 tablet_replica.cc:333] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.549057 10041 raft_consensus.cc:2243] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.549182 10041 raft_consensus.cc:2272] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:11.549360 9860 leader_election.cc:343] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.549439 9860 leader_election.cc:304] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:11.549660 10041 ts_tablet_manager.cc:1916] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.549772 11425 raft_consensus.cc:2749] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:11.550809 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7df6f0acec34d399aa25b78c3ed7c7a" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:11.550922 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_09274fe9916e428a82701f8bd406e392"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:11.551486 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09274fe9916e428a82701f8bd406e392 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:11.555922 9860 leader_election.cc:343] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.555971 9304 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet e6b58b1f4f63461aa96e416cae879be8 (table default.Table_f3c2de263e8849caaef87b570fc69459 [id=ef50ef2238804c98ad19c5b1c0b881c4]). Expected version 1 got 0
I20251027 21:08:11.556053 9860 leader_election.cc:304] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:11.556506 11400 raft_consensus.cc:2749] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:11.557220 9304 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet d672118ecd6f4094855adf106e5f7115 (table default.Table_1b066b58b2c649a2887ed91461f15d7c [id=887ea33c2fbe4573a1d7169dfd226567]). Expected version 2 got 0
I20251027 21:08:11.557852 9304 catalog_manager.cc:5649] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:11.558519 9304 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 67cb71ce43244d8ab9e71ead6bb315c9 (table default.Table_6327cacf4ed54d56b522291ad9594784 [id=3eb54d245ec64ee48022c541c2e49b17]). Expected version 2 got 0
I20251027 21:08:11.565977 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet c3b9a461567a48fcb4af514ecdf618b4 (table default.Table_fc2da109d1934a3fbe14ae51b20e771d [id=56f8db49ad8c4670a73a29bf4ed613e9]) successfully deleted
I20251027 21:08:11.567454 9738 tablet_service.cc:1505] Processing CreateTablet for tablet aff7b063160f40de85e053c74bff2f3c (DEFAULT_TABLE table=default.Table_b0918d68920b4aa5b1fbcd706612d409 [id=56d078d10d5b4b66a2eeff2a3f04ca73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.567874 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aff7b063160f40de85e053c74bff2f3c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:11.569835 9298 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.9.14.65:44531, user_credentials={real_user=slave}} blocked reactor thread for 52573.5us
I20251027 21:08:11.571184 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 958dafe1e6844f19902524e8fa27d69a (DEFAULT_TABLE table=default.Table_406cb517a4d34f0c921987c19227668b [id=69c1eb2307f24390bdf052bc2d0d1c85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.571578 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 958dafe1e6844f19902524e8fa27d69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.571568 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet b162734c536a40c08755201a85255f4c
I20251027 21:08:11.572216 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_554983eff7d04fefb9f5f9a7bf6ccc55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:11.572718 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_554983eff7d04fefb9f5f9a7bf6ccc55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.573778 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet b6c4fc648fa04d9aa4a10b5b6b07cd21
I20251027 21:08:11.574098 9307 catalog_manager.cc:6719] ReplaceTablet: tablet b6c4fc648fa04d9aa4a10b5b6b07cd21 deleted and replaced by tablet d5c1c9eafeb841d1834d87bccedd62c3
I20251027 21:08:11.574465 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ecf1a7cbc64b46579e08c39312812361" } new_table_name: "default.Table_9bf7b754a163446db873b9b81a700e4d" modify_external_catalogs: true
I20251027 21:08:11.576874 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e4b84d8c0ee74dcbaee8aeb49aa30c57
I20251027 21:08:11.576956 10041 ts_tablet_manager.cc:1929] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.577044 10041 log.cc:1199] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/e1e58f6228254e3eb25dbf627a505ca1
I20251027 21:08:11.577280 9305 catalog_manager.cc:6719] ReplaceTablet: tablet e4b84d8c0ee74dcbaee8aeb49aa30c57 deleted and replaced by tablet 054d78c606d64944b32618b93615befd
I20251027 21:08:11.577348 10041 ts_tablet_manager.cc:1950] T e1e58f6228254e3eb25dbf627a505ca1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.577956 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet b162734c536a40c08755201a85255f4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:11.578222 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet b6c4fc648fa04d9aa4a10b5b6b07cd21 with delete_type TABLET_DATA_DELETED (replaced by tablet d5c1c9eafeb841d1834d87bccedd62c3) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:11.580724 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet b162734c536a40c08755201a85255f4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:11.580935 9610 tablet_service.cc:1552] Processing DeleteTablet for tablet b6c4fc648fa04d9aa4a10b5b6b07cd21 with delete_type TABLET_DATA_DELETED (replaced by tablet d5c1c9eafeb841d1834d87bccedd62c3) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:11.581110 9611 tablet_service.cc:1552] Processing DeleteTablet for tablet e4b84d8c0ee74dcbaee8aeb49aa30c57 with delete_type TABLET_DATA_DELETED (replaced by tablet 054d78c606d64944b32618b93615befd) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:11.581760 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet b162734c536a40c08755201a85255f4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:11.581944 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet b6c4fc648fa04d9aa4a10b5b6b07cd21 with delete_type TABLET_DATA_DELETED (replaced by tablet d5c1c9eafeb841d1834d87bccedd62c3) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:11.581944 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet e4b84d8c0ee74dcbaee8aeb49aa30c57 with delete_type TABLET_DATA_DELETED (replaced by tablet 054d78c606d64944b32618b93615befd) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:11.582062 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet 90f96c21da9944f1ade05d35787d47e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:11.582274 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet e4b84d8c0ee74dcbaee8aeb49aa30c57 with delete_type TABLET_DATA_DELETED (replaced by tablet 054d78c606d64944b32618b93615befd) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:11.582633 10029 tablet_bootstrap.cc:492] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:11.582800 10029 ts_tablet_manager.cc:1403] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.130s user 0.002s sys 0.002s
I20251027 21:08:11.583539 10029 raft_consensus.cc:359] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.583781 10029 raft_consensus.cc:385] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.583863 10029 raft_consensus.cc:740] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.584102 10029 consensus_queue.cc:260] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.584467 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1109) took 2529 ms. Trace:
I20251027 21:08:11.584578 10041 rpcz_store.cc:276] 1027 21:08:09.054807 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.054872 (+ 65us) service_pool.cc:225] Handling call
1027 21:08:11.548853 (+2493981us) ts_tablet_manager.cc:1151] Deleting tablet e1e58f6228254e3eb25dbf627a505ca1
1027 21:08:11.548863 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.564766 (+ 15903us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.576870 (+ 12104us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.584449 (+ 7579us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2493769}
I20251027 21:08:11.584978 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet e1e58f6228254e3eb25dbf627a505ca1 (table default.Table_3e9f6e0f115040639f405946e55a2637 [id=dd2a4ed46c1c4a6c8687547e809f2677]) successfully deleted
I20251027 21:08:11.586438 10041 tablet_replica.cc:333] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.586915 10041 raft_consensus.cc:2243] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.587133 9311 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_d8da07f0725347338012d992737c5d04 [id=bb676562b97f48088767941239cfbf33] alter complete (version 2)
I20251027 21:08:11.587410 10041 raft_consensus.cc:2272] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.587945 10041 ts_tablet_manager.cc:1916] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.588452 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet c3b9a461567a48fcb4af514ecdf618b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:11.588639 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet f775a7e42bdc4910bdb604b26a28a7f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 11627a84ff9b478c87c323617a09570b) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:11.590350 10031 tablet_bootstrap.cc:492] T a033551a591146039b017b4193b221ca P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:11.590523 10031 ts_tablet_manager.cc:1403] T a033551a591146039b017b4193b221ca P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.124s user 0.004s sys 0.001s
I20251027 21:08:11.591337 10031 raft_consensus.cc:359] T a033551a591146039b017b4193b221ca P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:11.591539 10031 raft_consensus.cc:385] T a033551a591146039b017b4193b221ca P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.591620 10031 raft_consensus.cc:740] T a033551a591146039b017b4193b221ca P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.591882 10031 consensus_queue.cc:260] T a033551a591146039b017b4193b221ca P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:11.592396 10031 ts_tablet_manager.cc:1434] T a033551a591146039b017b4193b221ca P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:11.593240 10029 ts_tablet_manager.cc:1434] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251027 21:08:11.593531 10029 tablet_bootstrap.cc:492] T 9ed4f9c61ad54e1292281b30bcd19533 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:11.593602 9605 tablet_service.cc:1505] Processing CreateTablet for tablet aff7b063160f40de85e053c74bff2f3c (DEFAULT_TABLE table=default.Table_b0918d68920b4aa5b1fbcd706612d409 [id=56d078d10d5b4b66a2eeff2a3f04ca73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.594009 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aff7b063160f40de85e053c74bff2f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.595165 10029 tablet_bootstrap.cc:654] T 9ed4f9c61ad54e1292281b30bcd19533 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:11.598248 10040 ts_tablet_manager.cc:1916] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.603147 9891 tablet_service.cc:1505] Processing CreateTablet for tablet aff7b063160f40de85e053c74bff2f3c (DEFAULT_TABLE table=default.Table_b0918d68920b4aa5b1fbcd706612d409 [id=56d078d10d5b4b66a2eeff2a3f04ca73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.606060 10040 ts_tablet_manager.cc:1929] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:11.606065 10041 ts_tablet_manager.cc:1929] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.606194 10041 log.cc:1199] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/42371496a7494e77a7bf1169de5b083a
I20251027 21:08:11.606516 10041 ts_tablet_manager.cc:1950] T 42371496a7494e77a7bf1169de5b083a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.609359 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aff7b063160f40de85e053c74bff2f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.607009 10043 ts_tablet_manager.cc:1929] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.615291 10043 log.cc:1199] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/f775a7e42bdc4910bdb604b26a28a7f4
I20251027 21:08:11.615216 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c30777b91b944d75a8b8758a620faf1c" } new_table_name: "default.Table_7389f97338764d72b7d951933afbc4dd" modify_external_catalogs: true
I20251027 21:08:11.615624 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1111) took 2560 ms. Trace:
I20251027 21:08:11.616236 10041 rpcz_store.cc:276] 1027 21:08:09.055499 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.055549 (+ 50us) service_pool.cc:225] Handling call
1027 21:08:11.586356 (+2530807us) ts_tablet_manager.cc:1151] Deleting tablet 42371496a7494e77a7bf1169de5b083a
1027 21:08:11.586367 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.590944 (+ 4577us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.605956 (+ 15012us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.615564 (+ 9608us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2530648}
I20251027 21:08:11.616533 10043 ts_tablet_manager.cc:1950] T f775a7e42bdc4910bdb604b26a28a7f4 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:11.616786 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6ef774fe4a464edc8b6ac91a3011fd41" } new_table_name: "default.Table_d481a540123e4620874b629ce6db65a7" modify_external_catalogs: true
I20251027 21:08:11.616914 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_70046cf1a7e44ca287756df96f94e51e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:11.617049 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 42371496a7494e77a7bf1169de5b083a (table default.Table_6f3e1c9642e3433eab3f8e5a409470c4 [id=da9123bd1cc24db8b3be1d53b992292f]) successfully deleted
W20251027 21:08:11.617425 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70046cf1a7e44ca287756df96f94e51e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.617858 10041 tablet_replica.cc:333] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.618291 10041 raft_consensus.cc:2243] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.618768 10041 raft_consensus.cc:2272] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.618829 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1054) took 4765 ms. Trace:
I20251027 21:08:11.618847 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_14d4a985bc664c44b50d535a4757aa00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:11.618997 10043 rpcz_store.cc:276] 1027 21:08:06.853342 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:06.854045 (+ 703us) service_pool.cc:225] Handling call
1027 21:08:11.443656 (+4589611us) ts_tablet_manager.cc:1151] Deleting tablet f775a7e42bdc4910bdb604b26a28a7f4
1027 21:08:11.443666 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.564245 (+120579us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.606910 (+ 42665us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.618810 (+ 11900us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4589404}
W20251027 21:08:11.619290 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14d4a985bc664c44b50d535a4757aa00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.619434 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet f775a7e42bdc4910bdb604b26a28a7f4 (table default.Table_7b970942803b42f29feb36d6d845167a [id=20a729dd21c440a882083323ddb2561e]) successfully deleted
I20251027 21:08:11.619536 10041 ts_tablet_manager.cc:1916] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.619709 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1063) took 4076 ms. Trace:
I20251027 21:08:11.619778 10043 rpcz_store.cc:276] 1027 21:08:07.542873 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.542925 (+ 52us) service_pool.cc:225] Handling call
1027 21:08:11.619592 (+4076667us) ts_tablet_manager.cc:1151] Deleting tablet 050e9442041d495496bfa7ce25ac6a88
1027 21:08:11.619604 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.619694 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4076498}
I20251027 21:08:11.620085 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1064) took 4076 ms. Trace:
W20251027 21:08:11.620165 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 050e9442041d495496bfa7ce25ac6a88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 050e9442041d495496bfa7ce25ac6a88 already in progress: creating tablet
I20251027 21:08:11.620147 10043 rpcz_store.cc:276] 1027 21:08:07.543507 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.543552 (+ 45us) service_pool.cc:225] Handling call
1027 21:08:11.619996 (+4076444us) ts_tablet_manager.cc:1151] Deleting tablet 473cc9f814c943a8967d92e58ed7dfd2
1027 21:08:11.620004 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.620071 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4076299}
I20251027 21:08:11.620436 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1065) took 4076 ms. Trace:
W20251027 21:08:11.620491 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 473cc9f814c943a8967d92e58ed7dfd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 473cc9f814c943a8967d92e58ed7dfd2 already in progress: creating tablet
I20251027 21:08:11.620654 10040 log.cc:1199] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/2a4aa1fe05114c1287ca580c1f411d3f
I20251027 21:08:11.621172 10040 ts_tablet_manager.cc:1950] T 2a4aa1fe05114c1287ca580c1f411d3f P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:11.622591 10043 rpcz_store.cc:276] 1027 21:08:07.544320 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.544366 (+ 46us) service_pool.cc:225] Handling call
1027 21:08:11.620338 (+4075972us) ts_tablet_manager.cc:1151] Deleting tablet a24dbceaa95e4f43a5ea416a20e7041e
1027 21:08:11.620347 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.620421 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4075831}
W20251027 21:08:11.623159 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet a24dbceaa95e4f43a5ea416a20e7041e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a24dbceaa95e4f43a5ea416a20e7041e already in progress: creating tablet
I20251027 21:08:11.623558 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1066) took 4039 ms. Trace:
I20251027 21:08:11.624147 11437 raft_consensus.cc:493] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.624480 10043 rpcz_store.cc:276] 1027 21:08:07.584615 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.584682 (+ 67us) service_pool.cc:225] Handling call
1027 21:08:11.623469 (+4038787us) ts_tablet_manager.cc:1151] Deleting tablet c2992034f072482382a4a593fb7f7aaa
1027 21:08:11.623478 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.623544 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4033231}
I20251027 21:08:11.624509 11437 raft_consensus.cc:515] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:11.625128 11437 leader_election.cc:290] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:11.625537 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76e4fb2bea7f4a4696fe9f6bb892bccc" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:11.625754 9634 raft_consensus.cc:2468] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
W20251027 21:08:11.626070 9711 leader_election.cc:343] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 76e4fb2bea7f4a4696fe9f6bb892bccc
I20251027 21:08:11.626248 9710 leader_election.cc:304] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
W20251027 21:08:11.625067 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet c2992034f072482382a4a593fb7f7aaa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2992034f072482382a4a593fb7f7aaa already in progress: creating tablet
I20251027 21:08:11.626492 11437 raft_consensus.cc:2804] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:11.626556 11437 raft_consensus.cc:493] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:11.626621 11437 raft_consensus.cc:3060] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:11.628351 11142 raft_consensus.cc:493] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.628473 11142 raft_consensus.cc:515] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.628700 11437 raft_consensus.cc:515] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:11.629031 11142 leader_election.cc:290] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.629314 11437 leader_election.cc:290] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:11.629724 10041 ts_tablet_manager.cc:1929] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251027 21:08:11.629977 9711 leader_election.cc:343] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 76e4fb2bea7f4a4696fe9f6bb892bccc
I20251027 21:08:11.630668 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76e4fb2bea7f4a4696fe9f6bb892bccc" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:11.630826 9634 raft_consensus.cc:3060] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:11.631573 9568 leader_election.cc:343] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 30ce703650c94c06b93bce976cb5d1fd
I20251027 21:08:11.633052 9634 raft_consensus.cc:2468] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:11.633657 9710 leader_election.cc:304] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:11.633878 11437 raft_consensus.cc:2804] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:11.633967 11437 raft_consensus.cc:697] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:11.634229 11437 consensus_queue.cc:237] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:11.633739 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 7e32e33235b841aeb2a1eabf25bd9ff3 (DEFAULT_TABLE table=default.Table_18c39792431548bf80997615fa3dd505 [id=b216a71e20ed44b688526117a511b357]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.636149 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e22a34d2f5e848fcbe891c4d0325fad4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:11.636612 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e22a34d2f5e848fcbe891c4d0325fad4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.637454 10031 tablet_bootstrap.cc:492] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:11.638116 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30ce703650c94c06b93bce976cb5d1fd" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:11.638223 10041 log.cc:1199] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/d8691aa22fe84cf29893ed4e79d156cc
I20251027 21:08:11.638553 10041 ts_tablet_manager.cc:1950] T d8691aa22fe84cf29893ed4e79d156cc P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
W20251027 21:08:11.638828 9566 leader_election.cc:343] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.638891 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e32e33235b841aeb2a1eabf25bd9ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.638931 9566 leader_election.cc:304] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:11.639232 9308 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_fe1d774fd34f41fa916f26ced6fb7612 [id=434ed8c436c149ef9c7eb86ad252291d] alter complete (version 5)
I20251027 21:08:11.639310 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1110) took 2583 ms. Trace:
I20251027 21:08:11.639433 11411 raft_consensus.cc:2749] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:11.639400 10041 rpcz_store.cc:276] 1027 21:08:09.055389 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.055755 (+ 366us) service_pool.cc:225] Handling call
1027 21:08:11.617797 (+2562042us) ts_tablet_manager.cc:1151] Deleting tablet d8691aa22fe84cf29893ed4e79d156cc
1027 21:08:11.617806 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.625963 (+ 8157us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.629610 (+ 3647us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.639273 (+ 9663us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2561882}
I20251027 21:08:11.639869 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet d8691aa22fe84cf29893ed4e79d156cc (table default.Table_da0e93f761894c259f883d648d7f25d8 [id=a3955ead580245ebaad587dbe2d98a27]) successfully deleted
I20251027 21:08:11.640580 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet e04f4c5cf724419ab0be0e84d30a9a57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:11.640784 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet e9b8f76c33444bcb9b488af65e4d3e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:11.641772 11142 raft_consensus.cc:493] T 7425b6b0184e4d0b871777b25d904ac4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.641882 11142 raft_consensus.cc:515] T 7425b6b0184e4d0b871777b25d904ac4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.642544 11142 leader_election.cc:290] T 7425b6b0184e4d0b871777b25d904ac4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.642575 10041 tablet_replica.cc:333] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.642831 10041 raft_consensus.cc:2243] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.643070 10041 raft_consensus.cc:2272] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.643054 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7425b6b0184e4d0b871777b25d904ac4" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:11.643500 9566 leader_election.cc:343] T 7425b6b0184e4d0b871777b25d904ac4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.643796 10041 ts_tablet_manager.cc:1916] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.643829 10031 tablet_bootstrap.cc:654] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:11.644601 9568 leader_election.cc:343] T 7425b6b0184e4d0b871777b25d904ac4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 7425b6b0184e4d0b871777b25d904ac4
I20251027 21:08:11.644716 9568 leader_election.cc:304] T 7425b6b0184e4d0b871777b25d904ac4 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:11.645129 11411 raft_consensus.cc:2749] T 7425b6b0184e4d0b871777b25d904ac4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:11.646872 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 7e32e33235b841aeb2a1eabf25bd9ff3 (DEFAULT_TABLE table=default.Table_18c39792431548bf80997615fa3dd505 [id=b216a71e20ed44b688526117a511b357]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.646917 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_8ddc08df525c4d329846f82ba45beeca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:11.647254 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e32e33235b841aeb2a1eabf25bd9ff3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:11.647411 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ddc08df525c4d329846f82ba45beeca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.647930 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 36eff1ae33cd4621bc7715fa06533c9b (DEFAULT_TABLE table=default.Table_b3acc4069419400c9cf22f7450e14d03 [id=da0da486a6c9498a9ef01d95778476bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.648360 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36eff1ae33cd4621bc7715fa06533c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.648895 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 36eff1ae33cd4621bc7715fa06533c9b (DEFAULT_TABLE table=default.Table_b3acc4069419400c9cf22f7450e14d03 [id=da0da486a6c9498a9ef01d95778476bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.649210 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36eff1ae33cd4621bc7715fa06533c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.653659 10032 tablet_bootstrap.cc:492] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:11.653810 11400 raft_consensus.cc:493] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.653861 10032 ts_tablet_manager.cc:1403] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.136s user 0.013s sys 0.024s
I20251027 21:08:11.653950 11400 raft_consensus.cc:515] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.654577 11400 leader_election.cc:290] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.654657 10032 raft_consensus.cc:359] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.654808 10032 raft_consensus.cc:385] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.654888 10032 raft_consensus.cc:740] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.655082 10032 consensus_queue.cc:260] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.655151 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22dbfa50996b47269983c107cc399000" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:11.655426 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22dbfa50996b47269983c107cc399000" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:11.655737 10032 ts_tablet_manager.cc:1434] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:11.656085 10032 tablet_bootstrap.cc:492] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:08:11.656291 9860 leader_election.cc:343] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:11.656534 9859 leader_election.cc:343] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.656636 9859 leader_election.cc:304] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:11.657099 11425 raft_consensus.cc:2749] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:11.657873 10032 tablet_bootstrap.cc:654] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:11.662698 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1048) took 4190 ms. Trace:
I20251027 21:08:11.662899 10040 rpcz_store.cc:276] 1027 21:08:07.471880 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.475312 (+ 3432us) service_pool.cc:225] Handling call
1027 21:08:11.540416 (+4065104us) ts_tablet_manager.cc:1151] Deleting tablet 2a4aa1fe05114c1287ca580c1f411d3f
1027 21:08:11.540423 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.603101 (+ 62678us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.605979 (+ 2878us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.662675 (+ 56696us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4064724}
I20251027 21:08:11.663538 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 2a4aa1fe05114c1287ca580c1f411d3f (table default.Table_f08509866aae44609f26ce5e30963f6c [id=63bc2d46b97e4054b557025436c371d9]) successfully deleted
I20251027 21:08:11.663720 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1049) took 4131 ms. Trace:
I20251027 21:08:11.663838 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 36eff1ae33cd4621bc7715fa06533c9b (DEFAULT_TABLE table=default.Table_b3acc4069419400c9cf22f7450e14d03 [id=da0da486a6c9498a9ef01d95778476bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.663867 10040 rpcz_store.cc:276] 1027 21:08:07.532209 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.532524 (+ 315us) service_pool.cc:225] Handling call
1027 21:08:11.663598 (+4131074us) ts_tablet_manager.cc:1151] Deleting tablet 050e9442041d495496bfa7ce25ac6a88
1027 21:08:11.663609 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.663702 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4130859}
I20251027 21:08:11.664234 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36eff1ae33cd4621bc7715fa06533c9b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:11.664387 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 050e9442041d495496bfa7ce25ac6a88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 050e9442041d495496bfa7ce25ac6a88 already in progress: creating tablet
I20251027 21:08:11.664506 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1051) took 4132 ms. Trace:
I20251027 21:08:11.664628 10040 rpcz_store.cc:276] 1027 21:08:07.532473 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.541894 (+ 9421us) service_pool.cc:225] Handling call
1027 21:08:11.664387 (+4122493us) ts_tablet_manager.cc:1151] Deleting tablet a24dbceaa95e4f43a5ea416a20e7041e
1027 21:08:11.664398 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.664489 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4122018}
W20251027 21:08:11.665056 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet a24dbceaa95e4f43a5ea416a20e7041e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a24dbceaa95e4f43a5ea416a20e7041e already in progress: creating tablet
I20251027 21:08:11.665164 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1050) took 4132 ms. Trace:
I20251027 21:08:11.665238 10040 rpcz_store.cc:276] 1027 21:08:07.532345 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.550253 (+ 17908us) service_pool.cc:225] Handling call
1027 21:08:11.665065 (+4114812us) ts_tablet_manager.cc:1151] Deleting tablet 473cc9f814c943a8967d92e58ed7dfd2
1027 21:08:11.665075 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.665149 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4114595}
W20251027 21:08:11.665675 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 473cc9f814c943a8967d92e58ed7dfd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 473cc9f814c943a8967d92e58ed7dfd2 already in progress: creating tablet
I20251027 21:08:11.665763 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1052) took 4082 ms. Trace:
I20251027 21:08:11.665838 10040 rpcz_store.cc:276] 1027 21:08:07.583470 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.585520 (+ 2050us) service_pool.cc:225] Handling call
1027 21:08:11.665664 (+4080144us) ts_tablet_manager.cc:1151] Deleting tablet c2992034f072482382a4a593fb7f7aaa
1027 21:08:11.665675 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.665749 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4079907}
I20251027 21:08:11.666508 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1053) took 4065 ms. Trace:
W20251027 21:08:11.666715 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c2992034f072482382a4a593fb7f7aaa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2992034f072482382a4a593fb7f7aaa already in progress: creating tablet
I20251027 21:08:11.666728 9605 tablet_service.cc:1505] Processing CreateTablet for tablet 7e32e33235b841aeb2a1eabf25bd9ff3 (DEFAULT_TABLE table=default.Table_18c39792431548bf80997615fa3dd505 [id=b216a71e20ed44b688526117a511b357]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.666817 10040 rpcz_store.cc:276] 1027 21:08:07.601104 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.648881 (+ 47777us) service_pool.cc:225] Handling call
1027 21:08:11.666401 (+4017520us) ts_tablet_manager.cc:1151] Deleting tablet c0e2772d66404d74891e0893c49a66e0
1027 21:08:11.666419 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.666494 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4017260}
I20251027 21:08:11.667114 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e32e33235b841aeb2a1eabf25bd9ff3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:11.667217 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c0e2772d66404d74891e0893c49a66e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0e2772d66404d74891e0893c49a66e0 already in progress: creating tablet
I20251027 21:08:11.667352 10040 tablet_replica.cc:333] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:11.667591 10040 raft_consensus.cc:2243] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.667779 10040 raft_consensus.cc:2272] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.668274 10040 ts_tablet_manager.cc:1916] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.673255 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 55c1f8e1bb9740148be630b94a13e52b (DEFAULT_TABLE table=default.Table_98e35c35ab94438a8db3dbd106f9f265 [id=5a2b49d58d954528a1a2bba6b965b65e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.673660 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55c1f8e1bb9740148be630b94a13e52b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.681309 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet c27c699cc2744f40811f7b9395ca76be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:11.682330 10040 ts_tablet_manager.cc:1929] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.682508 10040 log.cc:1199] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/da26c83976df40328cf9b11734fcfa44
I20251027 21:08:11.682881 10040 ts_tablet_manager.cc:1950] T da26c83976df40328cf9b11734fcfa44 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:11.683306 10032 tablet_bootstrap.cc:492] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:11.683522 10032 ts_tablet_manager.cc:1403] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20251027 21:08:11.684245 10032 raft_consensus.cc:359] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.684481 10032 raft_consensus.cc:385] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.684584 10032 raft_consensus.cc:740] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.683840 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5e855e21102b4ee5b98896cf48eacde6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:11.684738 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet a033551a591146039b017b4193b221ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:11.684980 10032 consensus_queue.cc:260] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.684999 11142 consensus_queue.cc:1048] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251027 21:08:11.685164 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e855e21102b4ee5b98896cf48eacde6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.685657 10032 ts_tablet_manager.cc:1434] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:11.685943 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1054) took 4028 ms. Trace:
I20251027 21:08:11.686128 10032 tablet_bootstrap.cc:492] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:11.686067 10040 rpcz_store.cc:276] 1027 21:08:07.657220 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.660500 (+ 3280us) service_pool.cc:225] Handling call
1027 21:08:11.667312 (+4006812us) ts_tablet_manager.cc:1151] Deleting tablet da26c83976df40328cf9b11734fcfa44
1027 21:08:11.667319 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.675662 (+ 8343us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.682243 (+ 6581us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.685925 (+ 3682us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4006358}
I20251027 21:08:11.686509 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet da26c83976df40328cf9b11734fcfa44 (table default.Table_ac66ea1435fe49518c073dfc9b2368c8 [id=985d97a0b33f4ef79a9b0df362a75c32]) successfully deleted
I20251027 21:08:11.686650 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1067) took 3223 ms. Trace:
I20251027 21:08:11.686784 10040 rpcz_store.cc:276] 1027 21:08:08.463282 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.463333 (+ 51us) service_pool.cc:225] Handling call
1027 21:08:11.686548 (+3223215us) ts_tablet_manager.cc:1151] Deleting tablet 750b27c74ba3473dad743d60c21912c5
1027 21:08:11.686557 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.686637 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3222999}
I20251027 21:08:11.687258 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1068) took 3005 ms. Trace:
I20251027 21:08:11.687369 10040 rpcz_store.cc:276] 1027 21:08:08.682058 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.682337 (+ 279us) service_pool.cc:225] Handling call
1027 21:08:11.687166 (+3004829us) ts_tablet_manager.cc:1151] Deleting tablet a21a6a3725934266bbfdfc70b40de687
1027 21:08:11.687176 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.687244 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3004671}
W20251027 21:08:11.687733 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet a21a6a3725934266bbfdfc70b40de687 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a21a6a3725934266bbfdfc70b40de687 already in progress: creating tablet
I20251027 21:08:11.687834 10032 tablet_bootstrap.cc:654] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:11.687813 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1070) took 3005 ms. Trace:
I20251027 21:08:11.687872 10040 rpcz_store.cc:276] 1027 21:08:08.682262 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.682550 (+ 288us) service_pool.cc:225] Handling call
1027 21:08:11.687729 (+3005179us) ts_tablet_manager.cc:1151] Deleting tablet 8827e4746ecc408a8fea98b6b5ddd57b
1027 21:08:11.687737 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.687800 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3003094}
W20251027 21:08:11.688231 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 8827e4746ecc408a8fea98b6b5ddd57b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8827e4746ecc408a8fea98b6b5ddd57b already in progress: creating tablet
I20251027 21:08:11.688302 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1071) took 3003 ms. Trace:
I20251027 21:08:11.688365 10040 rpcz_store.cc:276] 1027 21:08:08.684823 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.686017 (+ 1194us) service_pool.cc:225] Handling call
1027 21:08:11.688222 (+3002205us) ts_tablet_manager.cc:1151] Deleting tablet cc97e55cfbcb41ec8c6c7410bbf14fb3
1027 21:08:11.688229 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.688290 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3002045}
I20251027 21:08:11.688763 10040 tablet_replica.cc:333] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:11.689021 10040 raft_consensus.cc:2243] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.689344 10040 raft_consensus.cc:2272] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.690464 10040 ts_tablet_manager.cc:1916] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:11.691488 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet cc97e55cfbcb41ec8c6c7410bbf14fb3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc97e55cfbcb41ec8c6c7410bbf14fb3 already in progress: creating tablet
I20251027 21:08:11.691951 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 55c1f8e1bb9740148be630b94a13e52b (DEFAULT_TABLE table=default.Table_98e35c35ab94438a8db3dbd106f9f265 [id=5a2b49d58d954528a1a2bba6b965b65e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.692533 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55c1f8e1bb9740148be630b94a13e52b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.695408 10041 ts_tablet_manager.cc:1929] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.697821 10040 ts_tablet_manager.cc:1929] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 21:08:11.697973 10040 log.cc:1199] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/94f2819dd0394530ae93754bdd1e482f
I20251027 21:08:11.698479 10040 ts_tablet_manager.cc:1950] T 94f2819dd0394530ae93754bdd1e482f P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:11.699426 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1072) took 3014 ms. Trace:
I20251027 21:08:11.699590 10040 rpcz_store.cc:276] 1027 21:08:08.684916 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.686209 (+ 1293us) service_pool.cc:225] Handling call
1027 21:08:11.688725 (+3002516us) ts_tablet_manager.cc:1151] Deleting tablet 94f2819dd0394530ae93754bdd1e482f
1027 21:08:11.688733 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.694399 (+ 5666us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.697752 (+ 3353us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.699383 (+ 1631us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3002404}
I20251027 21:08:11.700136 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 94f2819dd0394530ae93754bdd1e482f (table default.Table_f46fff92cdc244f6a020795467099e74 [id=63147f61013c4d6dbc96fd25769b6136]) successfully deleted
I20251027 21:08:11.700263 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1069) took 3018 ms. Trace:
I20251027 21:08:11.700367 10040 rpcz_store.cc:276] 1027 21:08:08.682179 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.682527 (+ 348us) service_pool.cc:225] Handling call
1027 21:08:11.700149 (+3017622us) ts_tablet_manager.cc:1151] Deleting tablet e53b2c28c75b43bdafa0d20f4ad61fcc
1027 21:08:11.700160 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.700248 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2997729}
I20251027 21:08:11.700925 10040 tablet_replica.cc:333] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
W20251027 21:08:11.700887 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet e53b2c28c75b43bdafa0d20f4ad61fcc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e53b2c28c75b43bdafa0d20f4ad61fcc already in progress: creating tablet
I20251027 21:08:11.701200 10040 raft_consensus.cc:2243] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:11.701632 10040 pending_rounds.cc:70] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
I20251027 21:08:11.701732 10040 pending_rounds.cc:77] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510687990767616 op_type: NO_OP noop_request { }
I20251027 21:08:11.701922 10040 raft_consensus.cc:2889] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:11.702260 10041 log.cc:1199] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/465be1b24d16457e834023a2b2194049
I20251027 21:08:11.702350 10040 raft_consensus.cc:2272] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.702796 10041 ts_tablet_manager.cc:1950] T 465be1b24d16457e834023a2b2194049 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.703137 10040 ts_tablet_manager.cc:1916] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.703750 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1112) took 2647 ms. Trace:
I20251027 21:08:11.704146 10041 rpcz_store.cc:276] 1027 21:08:09.055887 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.055951 (+ 64us) service_pool.cc:225] Handling call
1027 21:08:11.642514 (+2586563us) ts_tablet_manager.cc:1151] Deleting tablet 465be1b24d16457e834023a2b2194049
1027 21:08:11.642523 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.675919 (+ 33396us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.695302 (+ 19383us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.703725 (+ 8423us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2586433}
I20251027 21:08:11.704854 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 465be1b24d16457e834023a2b2194049 (table default.Table_185df62a04014137809bfc1bc3e5ecad [id=80175945bd1e436c92630ce62cc1c4ce]) successfully deleted
I20251027 21:08:11.705281 10041 tablet_replica.cc:333] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.705603 10041 raft_consensus.cc:2243] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.705837 10041 raft_consensus.cc:2272] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.706943 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 55c1f8e1bb9740148be630b94a13e52b (DEFAULT_TABLE table=default.Table_98e35c35ab94438a8db3dbd106f9f265 [id=5a2b49d58d954528a1a2bba6b965b65e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.707361 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55c1f8e1bb9740148be630b94a13e52b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.708101 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1067) took 4108 ms. Trace:
I20251027 21:08:11.708742 10043 rpcz_store.cc:276] 1027 21:08:07.599499 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.599555 (+ 56us) service_pool.cc:225] Handling call
1027 21:08:11.707959 (+4108404us) ts_tablet_manager.cc:1151] Deleting tablet c0e2772d66404d74891e0893c49a66e0
1027 21:08:11.707972 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.708076 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4108212}
W20251027 21:08:11.709579 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet c0e2772d66404d74891e0893c49a66e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0e2772d66404d74891e0893c49a66e0 already in progress: creating tablet
I20251027 21:08:11.712132 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 801cfdd144e34ec3bd43a49890858edd (DEFAULT_TABLE table=default.Table_0c303276539a43feb144ecfce1f3d172 [id=88737da7779d475ab0a2b0372963fa8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.712766 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 801cfdd144e34ec3bd43a49890858edd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:11.714864 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aaccda3227f44a28a18d77f7bc7e6633 consensus state has no leader
W20251027 21:08:11.715639 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9ace4188d3494a05a5678381f8d8fd17 consensus state has no leader
I20251027 21:08:11.715828 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet 138a5d18566240858b5c92c3c03b8087 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:11.718851 10043 tablet_replica.cc:333] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:11.718863 10032 tablet_bootstrap.cc:492] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:11.718972 10032 ts_tablet_manager.cc:1403] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20251027 21:08:11.719580 10043 raft_consensus.cc:2243] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.719658 10032 raft_consensus.cc:359] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.719811 10032 raft_consensus.cc:385] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.719861 10032 raft_consensus.cc:740] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.719790 10043 raft_consensus.cc:2272] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.720043 10032 consensus_queue.cc:260] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.720444 10043 ts_tablet_manager.cc:1916] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.720619 10032 ts_tablet_manager.cc:1434] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:11.720906 10032 tablet_bootstrap.cc:492] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:11.721539 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 801cfdd144e34ec3bd43a49890858edd (DEFAULT_TABLE table=default.Table_0c303276539a43feb144ecfce1f3d172 [id=88737da7779d475ab0a2b0372963fa8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.721989 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 801cfdd144e34ec3bd43a49890858edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.722203 10032 tablet_bootstrap.cc:654] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:11.722365 10041 ts_tablet_manager.cc:1916] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.728776 10041 ts_tablet_manager.cc:1929] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.729079 9736 tablet_service.cc:1505] Processing CreateTablet for tablet a95007feaf3d4901ae95b8ee00c6e4ce (DEFAULT_TABLE table=default.Table_f3248ba254bd46baad5416e8b608e8c1 [id=881aef35f90c4b58a2fc26a0bf3462e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.729423 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95007feaf3d4901ae95b8ee00c6e4ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.742506 10040 ts_tablet_manager.cc:1929] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:11.742623 10040 log.cc:1199] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/bbb4116d1edd492893adfbf5fb3ddeef
I20251027 21:08:11.742992 10040 ts_tablet_manager.cc:1950] T bbb4116d1edd492893adfbf5fb3ddeef P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:11.743829 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1083) took 2677 ms. Trace:
I20251027 21:08:11.743945 10040 rpcz_store.cc:276] 1027 21:08:09.066300 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.066390 (+ 90us) service_pool.cc:225] Handling call
1027 21:08:11.700884 (+2634494us) ts_tablet_manager.cc:1151] Deleting tablet bbb4116d1edd492893adfbf5fb3ddeef
1027 21:08:11.700893 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.706687 (+ 5794us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.742091 (+ 35404us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.743807 (+ 1716us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2634297}
I20251027 21:08:11.744412 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet bbb4116d1edd492893adfbf5fb3ddeef (table default.Table_dfa3f51576fa4c689b2682b0e277df60 [id=088a35e69cc840c0b9386df033a885a8]) successfully deleted
I20251027 21:08:11.747967 10031 tablet_bootstrap.cc:492] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:11.748188 10031 ts_tablet_manager.cc:1403] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.111s user 0.002s sys 0.003s
I20251027 21:08:11.749058 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1085) took 2647 ms. Trace:
I20251027 21:08:11.733263 10032 tablet_bootstrap.cc:492] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:11.749157 10040 rpcz_store.cc:276] 1027 21:08:09.101283 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.101349 (+ 66us) service_pool.cc:225] Handling call
1027 21:08:11.748936 (+2647587us) ts_tablet_manager.cc:1151] Deleting tablet 95446c4333424f10a71829ce32d857e4
1027 21:08:11.748945 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.749038 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2647302}
I20251027 21:08:11.737807 11400 raft_consensus.cc:493] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.749521 10032 ts_tablet_manager.cc:1403] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.002s
I20251027 21:08:11.749508 11400 raft_consensus.cc:515] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:11.749617 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 95446c4333424f10a71829ce32d857e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95446c4333424f10a71829ce32d857e4 already in progress: creating tablet
I20251027 21:08:11.749063 10031 raft_consensus.cc:359] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:11.749987 10031 raft_consensus.cc:385] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.750056 10031 raft_consensus.cc:740] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.750195 11400 leader_election.cc:290] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.750308 10032 raft_consensus.cc:359] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.750458 10032 raft_consensus.cc:385] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.750514 10032 raft_consensus.cc:740] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.750675 10032 consensus_queue.cc:260] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.751259 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93d2f7df14a74731aea84927f0b7ed45" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:11.751498 10032 ts_tablet_manager.cc:1434] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251027 21:08:11.751801 10032 tablet_bootstrap.cc:492] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:08:11.751762 9859 leader_election.cc:343] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.751727 10031 consensus_queue.cc:260] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:11.752238 10031 ts_tablet_manager.cc:1434] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251027 21:08:11.753087 10032 tablet_bootstrap.cc:654] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:11.753727 10041 log.cc:1199] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/6210d3d10fc44b9e9e2b0780883186ce
I20251027 21:08:11.754043 10041 ts_tablet_manager.cc:1950] T 6210d3d10fc44b9e9e2b0780883186ce P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.754747 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1113) took 2698 ms. Trace:
I20251027 21:08:11.754882 10041 rpcz_store.cc:276] 1027 21:08:09.056680 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.057116 (+ 436us) service_pool.cc:225] Handling call
1027 21:08:11.705221 (+2648105us) ts_tablet_manager.cc:1151] Deleting tablet 6210d3d10fc44b9e9e2b0780883186ce
1027 21:08:11.705235 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.725936 (+ 20701us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.728689 (+ 2753us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.754728 (+ 26039us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2647907}
I20251027 21:08:11.755285 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1114) took 2697 ms. Trace:
I20251027 21:08:11.755419 10041 rpcz_store.cc:276] 1027 21:08:09.057348 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.058290 (+ 942us) service_pool.cc:225] Handling call
1027 21:08:11.755202 (+2696912us) ts_tablet_manager.cc:1151] Deleting tablet 9c2d713d72fd4b5ca34cb3e828c75761
1027 21:08:11.755210 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.755269 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2696747}
I20251027 21:08:11.755955 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1115) took 2685 ms. Trace:
I20251027 21:08:11.755980 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 6210d3d10fc44b9e9e2b0780883186ce (table default.Table_6c3cfd4d761643dca83d16b2441b0c64 [id=20a2daa7746f40e69803fc010dc4bcbe]) successfully deleted
I20251027 21:08:11.756078 10041 rpcz_store.cc:276] 1027 21:08:09.070457 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.070507 (+ 50us) service_pool.cc:225] Handling call
1027 21:08:11.755859 (+2685352us) ts_tablet_manager.cc:1151] Deleting tablet 229ea4caa6af493cb5b3d593f22aa113
1027 21:08:11.755867 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.755942 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2685170}
I20251027 21:08:11.753352 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1084) took 2652 ms. Trace:
W20251027 21:08:11.756232 9299 catalog_manager.cc:4952] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 9c2d713d72fd4b5ca34cb3e828c75761 because the tablet was not found. No further retry: Not found: Tablet not found: 9c2d713d72fd4b5ca34cb3e828c75761
I20251027 21:08:11.756283 10040 rpcz_store.cc:276] 1027 21:08:09.101133 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.101684 (+ 551us) service_pool.cc:225] Handling call
1027 21:08:11.752913 (+2651229us) ts_tablet_manager.cc:1151] Deleting tablet 229ea4caa6af493cb5b3d593f22aa113
1027 21:08:11.753257 (+ 344us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.753336 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2650995}
W20251027 21:08:11.756434 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 229ea4caa6af493cb5b3d593f22aa113 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 229ea4caa6af493cb5b3d593f22aa113 already in progress: creating tablet
I20251027 21:08:11.756477 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1118) took 2673 ms. Trace:
I20251027 21:08:11.756543 10041 rpcz_store.cc:276] 1027 21:08:09.082913 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.083287 (+ 374us) service_pool.cc:225] Handling call
1027 21:08:11.756363 (+2673076us) ts_tablet_manager.cc:1151] Deleting tablet cb5ac0fff70b491e8c8842f7a331005e
1027 21:08:11.756371 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.756460 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2672879}
I20251027 21:08:11.756740 10031 tablet_bootstrap.cc:492] T e6b58b1f4f63461aa96e416cae879be8 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:08:11.756908 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet cb5ac0fff70b491e8c8842f7a331005e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb5ac0fff70b491e8c8842f7a331005e already in progress: creating tablet
I20251027 21:08:11.757254 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1117) took 2674 ms. Trace:
W20251027 21:08:11.757345 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 229ea4caa6af493cb5b3d593f22aa113 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 229ea4caa6af493cb5b3d593f22aa113 already in progress: creating tablet
I20251027 21:08:11.757444 10041 rpcz_store.cc:276] 1027 21:08:09.082765 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.083511 (+ 746us) service_pool.cc:225] Handling call
1027 21:08:11.757161 (+2673650us) ts_tablet_manager.cc:1151] Deleting tablet f1a931c36fec45029715b01754d0b161
1027 21:08:11.757169 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.757238 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2673537}
W20251027 21:08:11.757918 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet f1a931c36fec45029715b01754d0b161 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1a931c36fec45029715b01754d0b161 already in progress: creating tablet
I20251027 21:08:11.757985 9902 tablet_service.cc:1505] Processing CreateTablet for tablet a95007feaf3d4901ae95b8ee00c6e4ce (DEFAULT_TABLE table=default.Table_f3248ba254bd46baad5416e8b608e8c1 [id=881aef35f90c4b58a2fc26a0bf3462e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.758344 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95007feaf3d4901ae95b8ee00c6e4ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.758424 10031 tablet_bootstrap.cc:654] T e6b58b1f4f63461aa96e416cae879be8 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:11.761250 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 371051fb3491426e8cf7797317baa3db consensus state has no leader
I20251027 21:08:11.762681 10032 tablet_bootstrap.cc:492] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:11.762840 10032 ts_tablet_manager.cc:1403] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20251027 21:08:11.762843 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 18dfddf6d1a646058c257196e27ebb70 (DEFAULT_TABLE table=default.Table_afddc510687a430dad9e6988b4cc2328 [id=b2bb82576abf4a5cae80f7b47d6d2854]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.763211 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18dfddf6d1a646058c257196e27ebb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.763650 10032 raft_consensus.cc:359] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:11.764730 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7d20b20b1f274f4f94a9497e2d030235" } modify_external_catalogs: true
I20251027 21:08:11.764874 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7d20b20b1f274f4f94a9497e2d030235" } modify_external_catalogs: true
I20251027 21:08:11.765884 10032 raft_consensus.cc:385] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.765952 10032 raft_consensus.cc:740] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.766047 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_439c4d078d484b7da5f47d76aca5a4a2" } new_table_name: "default.Table_350cdf0535494ceb8c2ec052651a5a67" modify_external_catalogs: true
I20251027 21:08:11.766161 10032 consensus_queue.cc:260] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:11.766767 10032 ts_tablet_manager.cc:1434] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251027 21:08:11.767040 10032 tablet_bootstrap.cc:492] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:11.767369 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f396e1e1ebc44cf9a774f6523942ff0b" } modify_external_catalogs: true
I20251027 21:08:11.767510 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f396e1e1ebc44cf9a774f6523942ff0b" } modify_external_catalogs: true
I20251027 21:08:11.768041 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_17c224e94fc94d99b0d23c088cef1820" } modify_external_catalogs: true
I20251027 21:08:11.768169 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_17c224e94fc94d99b0d23c088cef1820" } modify_external_catalogs: true
I20251027 21:08:11.768678 10032 tablet_bootstrap.cc:654] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:11.769515 10041 tablet_replica.cc:333] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.769690 10041 raft_consensus.cc:2243] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.769816 10041 raft_consensus.cc:2272] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.770323 10041 ts_tablet_manager.cc:1916] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.758651 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1087) took 2626 ms. Trace:
I20251027 21:08:11.770802 10040 rpcz_store.cc:276] 1027 21:08:09.132449 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.132613 (+ 164us) service_pool.cc:225] Handling call
1027 21:08:11.756523 (+2623910us) ts_tablet_manager.cc:1151] Deleting tablet cb5ac0fff70b491e8c8842f7a331005e
1027 21:08:11.758516 (+ 1993us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.758634 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2623659}
I20251027 21:08:11.771951 9598 tablet_service.cc:1552] Processing DeleteTablet for tablet 7425b6b0184e4d0b871777b25d904ac4 with delete_type TABLET_DATA_DELETED (replaced by tablet 070167be7d8d4ddf8da561df14448360) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:11.772344 9604 tablet_service.cc:1505] Processing CreateTablet for tablet a95007feaf3d4901ae95b8ee00c6e4ce (DEFAULT_TABLE table=default.Table_f3248ba254bd46baad5416e8b608e8c1 [id=881aef35f90c4b58a2fc26a0bf3462e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.772423 9601 tablet_service.cc:1552] Processing DeleteTablet for tablet f4187b2f66f24055b1df170f1062cc88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:11.772742 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95007feaf3d4901ae95b8ee00c6e4ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.772359 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 801cfdd144e34ec3bd43a49890858edd (DEFAULT_TABLE table=default.Table_0c303276539a43feb144ecfce1f3d172 [id=88737da7779d475ab0a2b0372963fa8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.773168 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 801cfdd144e34ec3bd43a49890858edd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:11.779623 9566 consensus_peers.cc:597] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:11.779728 9898 tablet_service.cc:1505] Processing CreateTablet for tablet fe1edd16342f4b7dadf74e566af6447d (DEFAULT_TABLE table=default.Table_116d0f9e18e840409b5a038c264e39cc [id=6df3c696665d4d14a59608b98f5e7688]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.779816 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 18dfddf6d1a646058c257196e27ebb70 (DEFAULT_TABLE table=default.Table_afddc510687a430dad9e6988b4cc2328 [id=b2bb82576abf4a5cae80f7b47d6d2854]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.780077 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe1edd16342f4b7dadf74e566af6447d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.780313 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 107c608f9d904d86b74e2ad682e1bdc7 (DEFAULT_TABLE table=default.Table_152a2bed589f47858b8a99ab7df9984c [id=c1fcc453aee04cbca24ccb8bdfa3ba00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.780625 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 107c608f9d904d86b74e2ad682e1bdc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.781822 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 18dfddf6d1a646058c257196e27ebb70 (DEFAULT_TABLE table=default.Table_afddc510687a430dad9e6988b4cc2328 [id=b2bb82576abf4a5cae80f7b47d6d2854]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.782167 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18dfddf6d1a646058c257196e27ebb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.782362 9598 tablet_service.cc:1505] Processing CreateTablet for tablet 107c608f9d904d86b74e2ad682e1bdc7 (DEFAULT_TABLE table=default.Table_152a2bed589f47858b8a99ab7df9984c [id=c1fcc453aee04cbca24ccb8bdfa3ba00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.784951 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18dfddf6d1a646058c257196e27ebb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.786978 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet 133e474f9133487f89c44dbfa6266ca7 with delete_type TABLET_DATA_DELETED (replaced by tablet 6cd39721fd28422896d517d349009c66) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:11.790544 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 138a5d18566240858b5c92c3c03b8087
I20251027 21:08:11.790994 9307 catalog_manager.cc:6719] ReplaceTablet: tablet 138a5d18566240858b5c92c3c03b8087 deleted and replaced by tablet c734e20b87ce4c62a2c1b8851783c8b1
I20251027 21:08:11.791041 11400 raft_consensus.cc:493] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.791168 11400 raft_consensus.cc:515] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.791671 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet 138a5d18566240858b5c92c3c03b8087 with delete_type TABLET_DATA_DELETED (replaced by tablet c734e20b87ce4c62a2c1b8851783c8b1) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:11.791781 11400 leader_election.cc:290] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.792559 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet 138a5d18566240858b5c92c3c03b8087 with delete_type TABLET_DATA_DELETED (replaced by tablet c734e20b87ce4c62a2c1b8851783c8b1) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:11.793834 10032 tablet_bootstrap.cc:492] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:11.793955 10032 ts_tablet_manager.cc:1403] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.001s
I20251027 21:08:11.794701 10032 raft_consensus.cc:359] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.797936 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_203191e192be4e3a9a7477b4313ef4f4" } new_table_name: "default.Table_ade21e4db36445fc8f91817f46606b51" modify_external_catalogs: true
I20251027 21:08:11.798393 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 6a63227afb4d44ada8950fff99512af6 (DEFAULT_TABLE table=default.Table_146c25bb53a14c978f7015defb5a6ae0 [id=d620ab8b6d1c4b9f9dda07ae62dab61d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.798806 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_10a096bf3cc34dc89007e7abeb182f61" } modify_external_catalogs: true
I20251027 21:08:11.798941 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_10a096bf3cc34dc89007e7abeb182f61" } modify_external_catalogs: true
I20251027 21:08:11.799243 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a94891d2bdc54cdb84928a313c318421" } modify_external_catalogs: true
I20251027 21:08:11.799377 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a94891d2bdc54cdb84928a313c318421" } modify_external_catalogs: true
I20251027 21:08:11.801090 9308 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 7c15b74f53bc45e1af33b043f2cf225f (table default.Table_c97bb7def2964a3aacb4b0627420bf5a [id=32fe13ae5b414c62aba38f34c648d6ce]). Expected version 2 got 0
I20251027 21:08:11.802172 10032 raft_consensus.cc:385] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.802331 10032 raft_consensus.cc:740] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.802588 10032 consensus_queue.cc:260] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.802886 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet a7d8d060916941a5b7e944e8b08a2882 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:11.795928 10041 ts_tablet_manager.cc:1929] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.803016 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 6a63227afb4d44ada8950fff99512af6 (DEFAULT_TABLE table=default.Table_146c25bb53a14c978f7015defb5a6ae0 [id=d620ab8b6d1c4b9f9dda07ae62dab61d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.803148 10041 log.cc:1199] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/95446c4333424f10a71829ce32d857e4
I20251027 21:08:11.803445 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a63227afb4d44ada8950fff99512af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.803639 10041 ts_tablet_manager.cc:1950] T 95446c4333424f10a71829ce32d857e4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.806242 9888 tablet_service.cc:1552] Processing DeleteTablet for tablet 61ba8d7df6fd45abb6f5231ee8c4ec38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:11.812984 10032 ts_tablet_manager.cc:1434] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20251027 21:08:11.817909 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_8fc21ba8792f4de0aa3bb6f4b76db889"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:11.818441 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fc21ba8792f4de0aa3bb6f4b76db889 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.820165 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_546dee185ffb414b9876fbdda68cfd0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:11.820613 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_546dee185ffb414b9876fbdda68cfd0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.823109 10032 tablet_bootstrap.cc:492] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:11.824717 10032 tablet_bootstrap.cc:654] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:11.825220 9888 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ae222307a374a91916f560a0fbe8d8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:11.779021 9742 tablet_service.cc:1505] Processing CreateTablet for tablet fe1edd16342f4b7dadf74e566af6447d (DEFAULT_TABLE table=default.Table_116d0f9e18e840409b5a038c264e39cc [id=6df3c696665d4d14a59608b98f5e7688]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.820232 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1116) took 2737 ms. Trace:
W20251027 21:08:11.825636 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet cb5ac0fff70b491e8c8842f7a331005e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb5ac0fff70b491e8c8842f7a331005e already in progress: creating tablet
I20251027 21:08:11.825516 10041 rpcz_store.cc:276] 1027 21:08:09.082634 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.083671 (+ 1037us) service_pool.cc:225] Handling call
1027 21:08:11.769470 (+2685799us) ts_tablet_manager.cc:1151] Deleting tablet 95446c4333424f10a71829ce32d857e4
1027 21:08:11.769480 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.788724 (+ 19244us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.795849 (+ 7125us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.820213 (+ 24364us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2673919}
I20251027 21:08:11.825784 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe1edd16342f4b7dadf74e566af6447d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.825932 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 95446c4333424f10a71829ce32d857e4 (table default.Table_c40e78267cfd4b84b63987b1c910818f [id=3246662942e949b9ac37a58dd81f0ae9]) successfully deleted
I20251027 21:08:11.826001 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1088) took 2693 ms. Trace:
I20251027 21:08:11.826069 10040 rpcz_store.cc:276] 1027 21:08:09.132564 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.132940 (+ 376us) service_pool.cc:225] Handling call
1027 21:08:11.825887 (+2692947us) ts_tablet_manager.cc:1151] Deleting tablet 520f7142f8af4599ac4880b0692e3151
1027 21:08:11.825895 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.825985 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2575149}
I20251027 21:08:11.826234 10041 tablet_replica.cc:333] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
W20251027 21:08:11.826505 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 520f7142f8af4599ac4880b0692e3151 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 520f7142f8af4599ac4880b0692e3151 already in progress: creating tablet
I20251027 21:08:11.826859 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1086) took 2694 ms. Trace:
I20251027 21:08:11.826922 10040 rpcz_store.cc:276] 1027 21:08:09.132322 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.148092 (+ 15770us) service_pool.cc:225] Handling call
1027 21:08:11.826737 (+2678645us) ts_tablet_manager.cc:1151] Deleting tablet f1a931c36fec45029715b01754d0b161
1027 21:08:11.826746 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.826844 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2502407}
W20251027 21:08:11.827231 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet f1a931c36fec45029715b01754d0b161 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1a931c36fec45029715b01754d0b161 already in progress: creating tablet
I20251027 21:08:11.827529 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1089) took 2575 ms. Trace:
I20251027 21:08:11.827594 10040 rpcz_store.cc:276] 1027 21:08:09.252054 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.252099 (+ 45us) service_pool.cc:225] Handling call
1027 21:08:11.827436 (+2575337us) ts_tablet_manager.cc:1151] Deleting tablet b8c368698cfb40799571567e8bfdd8dc
1027 21:08:11.827444 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.827514 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2486478}
W20251027 21:08:11.827919 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet b8c368698cfb40799571567e8bfdd8dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8c368698cfb40799571567e8bfdd8dc already in progress: creating tablet
I20251027 21:08:11.828204 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1104) took 2347 ms. Trace:
I20251027 21:08:11.828264 10040 rpcz_store.cc:276] 1027 21:08:09.480686 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.481524 (+ 838us) service_pool.cc:225] Handling call
1027 21:08:11.828119 (+2346595us) ts_tablet_manager.cc:1151] Deleting tablet ece28e82383540bea3457b1cbc65558b
1027 21:08:11.828127 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.828190 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2346151}
W20251027 21:08:11.828557 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet ece28e82383540bea3457b1cbc65558b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ece28e82383540bea3457b1cbc65558b already in progress: creating tablet
I20251027 21:08:11.828871 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1106) took 2347 ms. Trace:
I20251027 21:08:11.828934 10040 rpcz_store.cc:276] 1027 21:08:09.481766 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.570452 (+ 88686us) service_pool.cc:225] Handling call
1027 21:08:11.828773 (+2258321us) ts_tablet_manager.cc:1151] Deleting tablet 0e64d21abef342de86555e876f020787
1027 21:08:11.828781 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.828856 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2258138}
W20251027 21:08:11.829295 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 0e64d21abef342de86555e876f020787 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e64d21abef342de86555e876f020787 already in progress: creating tablet
I20251027 21:08:11.829599 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1107) took 2347 ms. Trace:
I20251027 21:08:11.829671 10040 rpcz_store.cc:276] 1027 21:08:09.482049 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.570645 (+ 88596us) service_pool.cc:225] Handling call
1027 21:08:11.829505 (+2258860us) ts_tablet_manager.cc:1151] Deleting tablet e24312671b4b4106a1790c61a5548a76
1027 21:08:11.829513 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.829584 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2258727}
W20251027 21:08:11.830008 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet e24312671b4b4106a1790c61a5548a76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e24312671b4b4106a1790c61a5548a76 already in progress: creating tablet
I20251027 21:08:11.779351 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 107c608f9d904d86b74e2ad682e1bdc7 (DEFAULT_TABLE table=default.Table_152a2bed589f47858b8a99ab7df9984c [id=c1fcc453aee04cbca24ccb8bdfa3ba00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.830440 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 107c608f9d904d86b74e2ad682e1bdc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.832409 9736 tablet_service.cc:1505] Processing CreateTablet for tablet fb44b7df21804d3cbe5097b9313a7f84 (DEFAULT_TABLE table=default.Table_2d8e45c40e9243faac5ee9568774bd62 [id=35b881db9ba843d0bdb1d1914dfa03e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.832721 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb44b7df21804d3cbe5097b9313a7f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.792558 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 138a5d18566240858b5c92c3c03b8087 with delete_type TABLET_DATA_DELETED (replaced by tablet c734e20b87ce4c62a2c1b8851783c8b1) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:11.834071 10041 raft_consensus.cc:2243] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.834283 10041 raft_consensus.cc:2272] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.793305 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd9e6656d8df43afbb7f70706bc14585" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:11.834794 10041 ts_tablet_manager.cc:1916] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.795158 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet 96a680e8f16343b1908829a67e960b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:11.796738 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 6a63227afb4d44ada8950fff99512af6 (DEFAULT_TABLE table=default.Table_146c25bb53a14c978f7015defb5a6ae0 [id=d620ab8b6d1c4b9f9dda07ae62dab61d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.835103 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a63227afb4d44ada8950fff99512af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.836946 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b29fc23df2164435ae09aebea289defe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:11.837447 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b29fc23df2164435ae09aebea289defe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:11.838415 9859 leader_election.cc:343] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.843053 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1105) took 2362 ms. Trace:
I20251027 21:08:11.843138 10040 rpcz_store.cc:276] 1027 21:08:09.480823 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.481989 (+ 1166us) service_pool.cc:225] Handling call
1027 21:08:11.842942 (+2360953us) ts_tablet_manager.cc:1151] Deleting tablet 784df834432244e8812d1802f9e1a1bc
1027 21:08:11.842949 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.843033 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2253448}
I20251027 21:08:11.843479 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1108) took 2250 ms. Trace:
I20251027 21:08:11.843556 10040 rpcz_store.cc:276] 1027 21:08:09.592905 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.593392 (+ 487us) service_pool.cc:225] Handling call
1027 21:08:11.843382 (+2249990us) ts_tablet_manager.cc:1151] Deleting tablet 952583fd7fde4b4baccdb98c01bdf9e1
1027 21:08:11.843390 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.843464 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2249824}
W20251027 21:08:11.843750 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 784df834432244e8812d1802f9e1a1bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 784df834432244e8812d1802f9e1a1bc already in progress: creating tablet
I20251027 21:08:11.843813 10040 tablet_replica.cc:333] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:11.843968 10040 raft_consensus.cc:2243] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.844120 10040 raft_consensus.cc:2272] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:11.844219 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 952583fd7fde4b4baccdb98c01bdf9e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 952583fd7fde4b4baccdb98c01bdf9e1 already in progress: creating tablet
I20251027 21:08:11.844607 10040 ts_tablet_manager.cc:1916] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.851508 10041 ts_tablet_manager.cc:1929] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.853714 9892 tablet_service.cc:1505] Processing CreateTablet for tablet fb44b7df21804d3cbe5097b9313a7f84 (DEFAULT_TABLE table=default.Table_2d8e45c40e9243faac5ee9568774bd62 [id=35b881db9ba843d0bdb1d1914dfa03e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.854175 10041 log.cc:1199] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/520f7142f8af4599ac4880b0692e3151
I20251027 21:08:11.854346 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb44b7df21804d3cbe5097b9313a7f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.854350 9742 tablet_service.cc:1505] Processing CreateTablet for tablet ed8a50cfa7d84c1182b7ce5daf38d7cb (DEFAULT_TABLE table=default.Table_1f15260477b0409794ec2c672c472be3 [id=af4c904ad30b4821ba59915937080ef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.854689 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed8a50cfa7d84c1182b7ce5daf38d7cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.854715 10041 ts_tablet_manager.cc:1950] T 520f7142f8af4599ac4880b0692e3151 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:11.855746 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1119) took 2762 ms. Trace:
I20251027 21:08:11.856642 10041 rpcz_store.cc:276] 1027 21:08:09.092979 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.093057 (+ 78us) service_pool.cc:225] Handling call
1027 21:08:11.826191 (+2733134us) ts_tablet_manager.cc:1151] Deleting tablet 520f7142f8af4599ac4880b0692e3151
1027 21:08:11.826201 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.848359 (+ 22158us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.851425 (+ 3066us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.855725 (+ 4300us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2732910}
I20251027 21:08:11.857268 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 520f7142f8af4599ac4880b0692e3151 (table default.Table_e6955caf15904f3bae744add8221897f [id=bd9615f2023e45688ae88cd7d323e61b]) successfully deleted
W20251027 21:08:11.857436 9711 consensus_peers.cc:597] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cef2bfee222f45e6b6f20c6d63ff44d4. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:11.857712 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1120) took 2738 ms. Trace:
I20251027 21:08:11.858076 10041 rpcz_store.cc:276] 1027 21:08:09.119566 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.119644 (+ 78us) service_pool.cc:225] Handling call
1027 21:08:11.857611 (+2737967us) ts_tablet_manager.cc:1151] Deleting tablet b8c368698cfb40799571567e8bfdd8dc
1027 21:08:11.857619 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.857696 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2737764}
W20251027 21:08:11.858822 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet b8c368698cfb40799571567e8bfdd8dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8c368698cfb40799571567e8bfdd8dc already in progress: creating tablet
I20251027 21:08:11.859156 10041 tablet_replica.cc:333] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:11.859540 10041 raft_consensus.cc:2243] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.859913 10041 raft_consensus.cc:2272] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.860576 10041 ts_tablet_manager.cc:1916] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.863452 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet f19b5e8e460a4c4ba7399b981c603b84
W20251027 21:08:11.865134 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e2645a6c003a4523ac32787f6c6d3d1d consensus state has no leader
I20251027 21:08:11.866247 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet f19b5e8e460a4c4ba7399b981c603b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:11.866997 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 7057c55a820d4e379fe1d9abfa353821
I20251027 21:08:11.868333 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet 7057c55a820d4e379fe1d9abfa353821 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:11.781724 9607 tablet_service.cc:1505] Processing CreateTablet for tablet fe1edd16342f4b7dadf74e566af6447d (DEFAULT_TABLE table=default.Table_116d0f9e18e840409b5a038c264e39cc [id=6df3c696665d4d14a59608b98f5e7688]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:11.870970 9566 consensus_peers.cc:597] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:11.872066 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet d27f9c498eca4e5f8d2bfe9c805d06b3
I20251027 21:08:11.873103 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 4bfa70ded2f54e3fb2ca744c723395e2
I20251027 21:08:11.874517 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bfa70ded2f54e3fb2ca744c723395e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:11.875126 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 107c608f9d904d86b74e2ad682e1bdc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.875464 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a63227afb4d44ada8950fff99512af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.874513 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet d27f9c498eca4e5f8d2bfe9c805d06b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:11.876039 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe1edd16342f4b7dadf74e566af6447d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.876405 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 5cf3dd1b5f4b4b4ba112df78bf0df8f3
I20251027 21:08:11.877636 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet f19b5e8e460a4c4ba7399b981c603b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:11.877728 9602 tablet_service.cc:1505] Processing CreateTablet for tablet fb44b7df21804d3cbe5097b9313a7f84 (DEFAULT_TABLE table=default.Table_2d8e45c40e9243faac5ee9568774bd62 [id=35b881db9ba843d0bdb1d1914dfa03e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.878094 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb44b7df21804d3cbe5097b9313a7f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.878175 9609 tablet_service.cc:1552] Processing DeleteTablet for tablet 7057c55a820d4e379fe1d9abfa353821 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:11.878360 9600 tablet_service.cc:1552] Processing DeleteTablet for tablet d27f9c498eca4e5f8d2bfe9c805d06b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:11.878535 9608 tablet_service.cc:1552] Processing DeleteTablet for tablet 5cf3dd1b5f4b4b4ba112df78bf0df8f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:11.878646 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bfa70ded2f54e3fb2ca744c723395e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:11.879247 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet 5cf3dd1b5f4b4b4ba112df78bf0df8f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:11.879599 11412 raft_consensus.cc:493] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.879735 11412 raft_consensus.cc:515] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:11.879995 9603 tablet_service.cc:1505] Processing CreateTablet for tablet ed8a50cfa7d84c1182b7ce5daf38d7cb (DEFAULT_TABLE table=default.Table_1f15260477b0409794ec2c672c472be3 [id=af4c904ad30b4821ba59915937080ef2]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:11.880204 9710 connection.cc:537] client connection to 127.9.14.125:46313 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251027 21:08:11.880352 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed8a50cfa7d84c1182b7ce5daf38d7cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.880786 11412 leader_election.cc:290] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.882506 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a033551a591146039b017b4193b221ca" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:11.882696 9773 raft_consensus.cc:2468] T a033551a591146039b017b4193b221ca P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
W20251027 21:08:11.882778 9298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.14.125:46313 (error 108) [suppressed 4 similar messages]
W20251027 21:08:11.882954 9567 connection.cc:537] client connection to 127.9.14.125:46313 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251027 21:08:11.883291 9566 leader_election.cc:304] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:11.883562 11412 raft_consensus.cc:2804] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:11.883625 11412 raft_consensus.cc:493] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:11.883683 11412 raft_consensus.cc:3060] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:11.885761 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_66880e44b20b4f4cb43d1b827409bbd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:11.886449 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66880e44b20b4f4cb43d1b827409bbd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.888651 11373 raft_consensus.cc:493] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.888769 11373 raft_consensus.cc:515] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.892422 10031 tablet_bootstrap.cc:492] T e6b58b1f4f63461aa96e416cae879be8 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:11.892755 10031 ts_tablet_manager.cc:1403] T e6b58b1f4f63461aa96e416cae879be8 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.136s user 0.003s sys 0.001s
I20251027 21:08:11.893743 10031 raft_consensus.cc:359] T e6b58b1f4f63461aa96e416cae879be8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:11.894137 10029 tablet_bootstrap.cc:492] T 9ed4f9c61ad54e1292281b30bcd19533 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:11.894395 10029 ts_tablet_manager.cc:1403] T 9ed4f9c61ad54e1292281b30bcd19533 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.301s user 0.003s sys 0.010s
I20251027 21:08:11.888499 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd9e6656d8df43afbb7f70706bc14585" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:11.898056 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93d2f7df14a74731aea84927f0b7ed45" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:11.899442 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 1bd2e223198a4af9baca11a8ff21ab63 (DEFAULT_TABLE table=default.Table_a23d9519c070420782525d2a7397288c [id=bf11d0dacc1f423a9cd84e0fb27d02cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.900449 10031 raft_consensus.cc:385] T e6b58b1f4f63461aa96e416cae879be8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:11.900520 10031 raft_consensus.cc:740] T e6b58b1f4f63461aa96e416cae879be8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:11.900712 10031 consensus_queue.cc:260] T e6b58b1f4f63461aa96e416cae879be8 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:11.901188 10031 ts_tablet_manager.cc:1434] T e6b58b1f4f63461aa96e416cae879be8 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.008s user 0.000s sys 0.000s
W20251027 21:08:11.901837 11489 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 21:08:11.881269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.14.125:46313 (local address 127.9.14.66:58211)
1027 21:08:11.881664 (+ 395us) negotiation.cc:107] Waiting for socket to connect
1027 21:08:11.881685 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1027 21:08:11.881800 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 21:08:11.901681 (+ 19881us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.14.125:46313: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":639,"threads_started":1}
I20251027 21:08:11.900792 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bd2e223198a4af9baca11a8ff21ab63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.904948 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 538c1a8045da4864ab2f4c8b0c672f29 (DEFAULT_TABLE table=default.Table_10376e17301944fe9c799d6a87ff1f34 [id=47313707951a4a4ea44cc4ab4e62736b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.906023 10031 tablet_bootstrap.cc:492] T a4d26f73b8534f08bcaa386d4de75c01 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:11.907457 9305 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet e6b58b1f4f63461aa96e416cae879be8 (table default.Table_f3c2de263e8849caaef87b570fc69459 [id=ef50ef2238804c98ad19c5b1c0b881c4]). Expected version 1 got 0
I20251027 21:08:11.910362 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 538c1a8045da4864ab2f4c8b0c672f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:11.911682 11412 raft_consensus.cc:515] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:11.914301 9305 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet e6b58b1f4f63461aa96e416cae879be8 (table default.Table_f3c2de263e8849caaef87b570fc69459 [id=ef50ef2238804c98ad19c5b1c0b881c4]). Expected version 1 got 0
I20251027 21:08:11.916121 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a033551a591146039b017b4193b221ca" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:11.916294 9773 raft_consensus.cc:3060] T a033551a591146039b017b4193b221ca P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:11.917373 11373 leader_election.cc:290] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.918377 11373 raft_consensus.cc:493] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.918531 11373 raft_consensus.cc:515] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:11.918881 9773 raft_consensus.cc:2468] T a033551a591146039b017b4193b221ca P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:11.919124 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "986bf148b3e9482fa2fe53bbac54750d" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:11.919216 11373 leader_election.cc:290] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.919441 10031 tablet_bootstrap.cc:654] T a4d26f73b8534f08bcaa386d4de75c01 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:11.919798 11412 leader_election.cc:290] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.920243 9566 leader_election.cc:304] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:11.920528 11412 raft_consensus.cc:2804] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
W20251027 21:08:11.920564 9566 leader_election.cc:343] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.921221 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c2d713d72fd4b5ca34cb3e828c75761" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:11.921519 11412 raft_consensus.cc:697] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:11.921773 11412 consensus_queue.cc:237] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:11.923189 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b00669c899a945d283e7e68bcbc4384b" } new_table_name: "default.Table_d6a23e2259ce4ce9a41fc3567f893c08" modify_external_catalogs: true
I20251027 21:08:11.923346 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_196251158be7437dada70c9ff3938aa5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:11.923007 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d3089b6fc3e64e108d93bdab2c49aaac" } new_table_name: "default.Table_a32072ae75034f6091ef80369921c5a1" modify_external_catalogs: true
W20251027 21:08:11.924032 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_196251158be7437dada70c9ff3938aa5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:11.924837 9566 leader_election.cc:343] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.930222 10040 ts_tablet_manager.cc:1929] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.931501 10040 log.cc:1199] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/04c579bb008448e9baa841135b6dbc61
I20251027 21:08:11.934237 11400 raft_consensus.cc:493] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.934387 11400 raft_consensus.cc:515] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.935111 11400 leader_election.cc:290] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.935681 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e04f4c5cf724419ab0be0e84d30a9a57" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:11.936730 10040 ts_tablet_manager.cc:1950] T 04c579bb008448e9baa841135b6dbc61 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:11.938185 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1109) took 2318 ms. Trace:
I20251027 21:08:11.939885 10040 rpcz_store.cc:276] 1027 21:08:09.620641 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.626306 (+ 5665us) service_pool.cc:225] Handling call
1027 21:08:11.843775 (+2217469us) ts_tablet_manager.cc:1151] Deleting tablet 04c579bb008448e9baa841135b6dbc61
1027 21:08:11.843782 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.907620 (+ 63838us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.930089 (+ 22469us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.938168 (+ 8079us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2217285}
I20251027 21:08:11.940878 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 04c579bb008448e9baa841135b6dbc61 (table default.Table_fe308a2645c14e6c8efc7ce330b5add7 [id=73cba162a3af4c0fa5dfde81e1a81257]) successfully deleted
I20251027 21:08:11.941392 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1112) took 2242 ms. Trace:
I20251027 21:08:11.942039 10040 rpcz_store.cc:276] 1027 21:08:09.698506 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.705305 (+ 6799us) service_pool.cc:225] Handling call
1027 21:08:11.941285 (+2235980us) ts_tablet_manager.cc:1151] Deleting tablet 2a9c6bf4f75144f487305520b3714b84
1027 21:08:11.941294 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.941378 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2235146}
W20251027 21:08:11.943017 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 2a9c6bf4f75144f487305520b3714b84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a9c6bf4f75144f487305520b3714b84 already in progress: creating tablet
I20251027 21:08:11.943437 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1116) took 2198 ms. Trace:
I20251027 21:08:11.944010 10040 rpcz_store.cc:276] 1027 21:08:09.745301 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.745343 (+ 42us) service_pool.cc:225] Handling call
1027 21:08:11.943333 (+2197990us) ts_tablet_manager.cc:1151] Deleting tablet bc387163e2d64532b87c32c5842a285a
1027 21:08:11.943342 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.943420 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2197835}
W20251027 21:08:11.945502 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet bc387163e2d64532b87c32c5842a285a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc387163e2d64532b87c32c5842a285a already in progress: creating tablet
I20251027 21:08:11.945952 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1117) took 2200 ms. Trace:
I20251027 21:08:11.946545 10040 rpcz_store.cc:276] 1027 21:08:09.745647 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.745692 (+ 45us) service_pool.cc:225] Handling call
1027 21:08:11.945856 (+2200164us) ts_tablet_manager.cc:1151] Deleting tablet 53948dc94c334329bdf81f1eba94863c
1027 21:08:11.945865 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.945938 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2200025}
W20251027 21:08:11.947304 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 53948dc94c334329bdf81f1eba94863c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53948dc94c334329bdf81f1eba94863c already in progress: creating tablet
I20251027 21:08:11.947842 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1111) took 2249 ms. Trace:
I20251027 21:08:11.950462 10040 rpcz_store.cc:276] 1027 21:08:09.698387 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.787887 (+ 89500us) service_pool.cc:225] Handling call
1027 21:08:11.947743 (+2159856us) ts_tablet_manager.cc:1151] Deleting tablet 280ee23d80d64587b18811ff06340eee
1027 21:08:11.947751 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.947826 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2159433}
W20251027 21:08:11.951254 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 280ee23d80d64587b18811ff06340eee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 280ee23d80d64587b18811ff06340eee already in progress: creating tablet
I20251027 21:08:11.951709 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1123) took 2147 ms. Trace:
I20251027 21:08:11.952308 10040 rpcz_store.cc:276] 1027 21:08:09.804080 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.805079 (+ 999us) service_pool.cc:225] Handling call
1027 21:08:11.951594 (+2146515us) ts_tablet_manager.cc:1151] Deleting tablet d042a15408484fbc9ee9666476b103d5
1027 21:08:11.951603 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.951681 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2146252}
W20251027 21:08:11.952996 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet d042a15408484fbc9ee9666476b103d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d042a15408484fbc9ee9666476b103d5 already in progress: creating tablet
W20251027 21:08:11.953192 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32c32d85389f4255ae2041e4481a3765 consensus state has no leader
I20251027 21:08:11.953428 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1139) took 1849 ms. Trace:
I20251027 21:08:11.953502 10040 rpcz_store.cc:276] 1027 21:08:10.103633 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.103684 (+ 51us) service_pool.cc:225] Handling call
1027 21:08:11.953336 (+1849652us) ts_tablet_manager.cc:1151] Deleting tablet 423047dfbf084bfcb881efeb0352f88d
1027 21:08:11.953344 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.953413 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1742677}
W20251027 21:08:11.953830 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 423047dfbf084bfcb881efeb0352f88d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 423047dfbf084bfcb881efeb0352f88d already in progress: creating tablet
W20251027 21:08:11.954259 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eb8a89a572314d72a7d1ce77e6754abb consensus state has no leader
I20251027 21:08:11.955011 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1138) took 1851 ms. Trace:
I20251027 21:08:11.955096 10040 rpcz_store.cc:276] 1027 21:08:10.103551 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.103996 (+ 445us) service_pool.cc:225] Handling call
1027 21:08:11.954910 (+1850914us) ts_tablet_manager.cc:1151] Deleting tablet befdd3a485a1426b8a3adf340d4eae4f
1027 21:08:11.954921 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.954996 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1740664}
W20251027 21:08:11.955440 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet befdd3a485a1426b8a3adf340d4eae4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet befdd3a485a1426b8a3adf340d4eae4f already in progress: creating tablet
I20251027 21:08:11.956730 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1141) took 1852 ms. Trace:
W20251027 21:08:11.957371 9298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.125:46313, user_credentials={real_user=slave}} blocked reactor thread for 74596us
I20251027 21:08:11.958091 11400 raft_consensus.cc:493] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.958281 11400 raft_consensus.cc:515] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:11.958745 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 1bd2e223198a4af9baca11a8ff21ab63 (DEFAULT_TABLE table=default.Table_a23d9519c070420782525d2a7397288c [id=bf11d0dacc1f423a9cd84e0fb27d02cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:11.958897 11400 leader_election.cc:290] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.959105 10040 rpcz_store.cc:276] 1027 21:08:10.104030 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.104075 (+ 45us) service_pool.cc:225] Handling call
1027 21:08:11.956623 (+1852548us) ts_tablet_manager.cc:1151] Deleting tablet ca14d51a66de4a0d8476117e0984d1e9
1027 21:08:11.956635 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.956716 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1742311}
I20251027 21:08:11.959237 9613 tablet_service.cc:1505] Processing CreateTablet for tablet 538c1a8045da4864ab2f4c8b0c672f29 (DEFAULT_TABLE table=default.Table_10376e17301944fe9c799d6a87ff1f34 [id=47313707951a4a4ea44cc4ab4e62736b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:11.957914 9278 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 97211.5us
W20251027 21:08:11.959523 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet ca14d51a66de4a0d8476117e0984d1e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca14d51a66de4a0d8476117e0984d1e9 already in progress: creating tablet
I20251027 21:08:11.963404 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1140) took 1859 ms. Trace:
I20251027 21:08:11.963953 10040 rpcz_store.cc:276] 1027 21:08:10.103940 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.104119 (+ 179us) service_pool.cc:225] Handling call
1027 21:08:11.963307 (+1859188us) ts_tablet_manager.cc:1151] Deleting tablet 8d61afc9ce6b4b80b5e64259b9c04f85
1027 21:08:11.963315 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.963390 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1748960}
W20251027 21:08:11.964648 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 8d61afc9ce6b4b80b5e64259b9c04f85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d61afc9ce6b4b80b5e64259b9c04f85 already in progress: creating tablet
I20251027 21:08:11.965063 10040 tablet_replica.cc:333] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:11.965638 10040 raft_consensus.cc:2243] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:11.966279 10040 raft_consensus.cc:2272] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:11.966490 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_f24ffb28a47c46b7998af4df9cad118e" } new_table_name: "default.Table_e733707bb6444081a8a79a18d0052bb3" modify_external_catalogs: true
I20251027 21:08:11.967190 10040 ts_tablet_manager.cc:1916] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:11.966130 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f9fbc28451dc4d219623c15566ea0827"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:11.968411 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9fbc28451dc4d219623c15566ea0827 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.975467 11400 raft_consensus.cc:493] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.975596 11400 raft_consensus.cc:515] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.976110 11400 leader_election.cc:290] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.975260 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0d90eb17ab224b21b7a594324d408c7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:11.978149 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d90eb17ab224b21b7a594324d408c7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:11.980124 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6fe4618b580745428d8809010d4e64fe consensus state has no leader
W20251027 21:08:11.936331 9859 leader_election.cc:343] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:11.982275 9859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 45956.8us
I20251027 21:08:11.983193 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d8d060916941a5b7e944e8b08a2882" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:11.983376 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaaf70c950044387a63820079db3c829" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:11.984261 9859 leader_election.cc:343] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:11.984479 9859 leader_election.cc:343] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.984759 11425 raft_consensus.cc:493] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:11.984894 11425 raft_consensus.cc:515] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:11.985592 11425 leader_election.cc:290] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:11.984319 9911 raft_consensus.cc:1275] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Refusing update from remote peer 51902b6b61ff4d5584ff21896e847e14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 21:08:11.989511 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93a31187eca14b8797737936a80e7292" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:11.989905 10040 ts_tablet_manager.cc:1929] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:11.990015 10040 log.cc:1199] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/59b6818d29624da5a49dd1a7d20994c6
W20251027 21:08:11.990104 9859 leader_election.cc:343] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:11.990320 10040 ts_tablet_manager.cc:1950] T 59b6818d29624da5a49dd1a7d20994c6 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:11.990168 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e8e08989f9c4416fa4fb7ba0895ee310"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:11.990914 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8e08989f9c4416fa4fb7ba0895ee310 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.992290 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a440aeae7b724bfc9703e0524b2eb72f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:11.992457 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_916903f431254b8c89108557bc5f37bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:11.992864 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a440aeae7b724bfc9703e0524b2eb72f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:11.993016 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_916903f431254b8c89108557bc5f37bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:11.998903 9309 master_service.cc:954] ReplaceTablet: received request to replace tablet 283b72f626644ac2b9b61186387b3839 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:12.002413 10031 tablet_bootstrap.cc:492] T a4d26f73b8534f08bcaa386d4de75c01 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:12.002696 10031 ts_tablet_manager.cc:1403] T a4d26f73b8534f08bcaa386d4de75c01 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.097s user 0.009s sys 0.008s
I20251027 21:08:12.003535 10031 raft_consensus.cc:359] T a4d26f73b8534f08bcaa386d4de75c01 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.005947 10031 raft_consensus.cc:385] T a4d26f73b8534f08bcaa386d4de75c01 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.006687 10031 raft_consensus.cc:740] T a4d26f73b8534f08bcaa386d4de75c01 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.007229 10031 consensus_queue.cc:260] T a4d26f73b8534f08bcaa386d4de75c01 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.006510 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1142) took 1902 ms. Trace:
I20251027 21:08:12.009917 10040 rpcz_store.cc:276] 1027 21:08:10.104287 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.104331 (+ 44us) service_pool.cc:225] Handling call
1027 21:08:11.965014 (+1860683us) ts_tablet_manager.cc:1151] Deleting tablet 59b6818d29624da5a49dd1a7d20994c6
1027 21:08:11.965023 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:11.976862 (+ 11839us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:11.989809 (+ 12947us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.006484 (+ 16675us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1750628}
I20251027 21:08:12.011098 10040 tablet_replica.cc:333] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:12.010525 10031 ts_tablet_manager.cc:1434] T a4d26f73b8534f08bcaa386d4de75c01 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20251027 21:08:12.012336 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 59b6818d29624da5a49dd1a7d20994c6 (table default.Table_9904948d307e479ea184d6dc441b5e61 [id=500c0fe58264475087ef89652bf83263]) successfully deleted
I20251027 21:08:12.012676 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet a4d26f73b8534f08bcaa386d4de75c01 with delete_type TABLET_DATA_DELETED (replaced by tablet db2735604dc14039810df4ecd330c671) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:12.012836 10040 raft_consensus.cc:2243] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:12.012991 10040 raft_consensus.cc:2272] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.013564 10040 ts_tablet_manager.cc:1916] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.013629 10029 raft_consensus.cc:359] T 9ed4f9c61ad54e1292281b30bcd19533 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.013805 10029 raft_consensus.cc:385] T 9ed4f9c61ad54e1292281b30bcd19533 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.013851 10029 raft_consensus.cc:740] T 9ed4f9c61ad54e1292281b30bcd19533 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.014043 10029 consensus_queue.cc:260] T 9ed4f9c61ad54e1292281b30bcd19533 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.014613 10029 ts_tablet_manager.cc:1434] T 9ed4f9c61ad54e1292281b30bcd19533 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.120s user 0.002s sys 0.000s
I20251027 21:08:12.012692 10031 tablet_bootstrap.cc:492] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:08:12.018748 11492 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 21:08:11.884959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.14.125:46313 (local address 127.9.14.65:33149)
1027 21:08:12.018446 (+133487us) negotiation.cc:107] Waiting for socket to connect
1027 21:08:12.018534 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.14.125:46313: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":133367,"threads_started":1}
I20251027 21:08:12.018869 10031 tablet_bootstrap.cc:654] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.027179 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet f19b5e8e460a4c4ba7399b981c603b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:08:12.029150 9298 consensus_peers.cc:597] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 -> Peer 18b15927ce5744ad8ba81dda52725c11 (127.9.14.125:46313): Couldn't send request to peer 18b15927ce5744ad8ba81dda52725c11. Status: Network error: Client connection negotiation failed: client connection to 127.9.14.125:46313: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:12.029366 9860 connection.cc:537] client connection to 127.9.14.125:46313 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251027 21:08:12.030297 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e04f4c5cf724419ab0be0e84d30a9a57" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:12.031018 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93a31187eca14b8797737936a80e7292" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:12.031181 9608 tablet_service.cc:1552] Processing DeleteTablet for tablet 30ce703650c94c06b93bce976cb5d1fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:12.031877 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7d8d060916941a5b7e944e8b08a2882" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:12.032285 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaaf70c950044387a63820079db3c829" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:12.033535 9860 leader_election.cc:343] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.033658 9860 leader_election.cc:304] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:12.034027 9860 leader_election.cc:343] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.034112 9860 leader_election.cc:304] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.034571 11425 raft_consensus.cc:2749] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.034710 11425 raft_consensus.cc:2749] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.037283 10040 ts_tablet_manager.cc:1929] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:12.037415 10040 log.cc:1199] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/980eb6755aee4bcfb2ea44d84009021f
I20251027 21:08:12.037788 10040 ts_tablet_manager.cc:1950] T 980eb6755aee4bcfb2ea44d84009021f P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:12.038580 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1155) took 1700 ms. Trace:
I20251027 21:08:12.038676 10040 rpcz_store.cc:276] 1027 21:08:10.338477 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.338546 (+ 69us) service_pool.cc:225] Handling call
1027 21:08:12.011039 (+1672493us) ts_tablet_manager.cc:1151] Deleting tablet 980eb6755aee4bcfb2ea44d84009021f
1027 21:08:12.011052 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.034225 (+ 23173us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.037202 (+ 2977us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.038559 (+ 1357us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1672164}
I20251027 21:08:12.039062 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 980eb6755aee4bcfb2ea44d84009021f (table default.Table_bb56319b6fe0405991a4d171477621e7 [id=42c8fedcabe0435dbaa11427141228ed]) successfully deleted
I20251027 21:08:12.043174 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bd2e223198a4af9baca11a8ff21ab63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.046073 9605 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ed4f9c61ad54e1292281b30bcd19533 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:12.046403 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 538c1a8045da4864ab2f4c8b0c672f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.051502 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1157) took 1463 ms. Trace:
I20251027 21:08:12.051590 10040 rpcz_store.cc:276] 1027 21:08:10.588053 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.588106 (+ 53us) service_pool.cc:225] Handling call
1027 21:08:12.051361 (+1463255us) ts_tablet_manager.cc:1151] Deleting tablet 44804ae1c85749cb9286777e7b5c5707
1027 21:08:12.051374 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.051478 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1463082}
I20251027 21:08:12.052644 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1159) took 1464 ms. Trace:
I20251027 21:08:12.052731 10040 rpcz_store.cc:276] 1027 21:08:10.588261 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.588310 (+ 49us) service_pool.cc:225] Handling call
1027 21:08:12.052533 (+1464223us) ts_tablet_manager.cc:1151] Deleting tablet c97cba61e08d42a7bdb3f51db0a63ec8
1027 21:08:12.052542 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.052627 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1464107}
I20251027 21:08:12.053594 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1158) took 1465 ms. Trace:
I20251027 21:08:12.053671 10040 rpcz_store.cc:276] 1027 21:08:10.588174 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.589868 (+ 1694us) service_pool.cc:225] Handling call
1027 21:08:12.053503 (+1463635us) ts_tablet_manager.cc:1151] Deleting tablet 93e8b973b39244be9de1107b14d6d93f
1027 21:08:12.053512 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.053581 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1463492}
I20251027 21:08:12.054711 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1160) took 1464 ms. Trace:
I20251027 21:08:12.054780 10040 rpcz_store.cc:276] 1027 21:08:10.590314 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.592820 (+ 2506us) service_pool.cc:225] Handling call
1027 21:08:12.054614 (+1461794us) ts_tablet_manager.cc:1151] Deleting tablet 824ff6913b0142d5889c75aa8e3cba3e
1027 21:08:12.054622 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.054697 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1461650}
I20251027 21:08:12.055496 10040 tablet_replica.cc:333] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:12.055727 10040 raft_consensus.cc:2243] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:12.055999 10040 pending_rounds.cc:70] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
I20251027 21:08:12.056063 10040 pending_rounds.cc:77] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510696356139008 op_type: NO_OP noop_request { }
I20251027 21:08:12.056154 10040 raft_consensus.cc:2889] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:12.056242 10040 raft_consensus.cc:2272] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:12.052564 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 44804ae1c85749cb9286777e7b5c5707 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44804ae1c85749cb9286777e7b5c5707 already in progress: creating tablet
W20251027 21:08:12.057476 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c97cba61e08d42a7bdb3f51db0a63ec8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c97cba61e08d42a7bdb3f51db0a63ec8 already in progress: creating tablet
W20251027 21:08:12.057662 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 93e8b973b39244be9de1107b14d6d93f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93e8b973b39244be9de1107b14d6d93f already in progress: creating tablet
I20251027 21:08:12.058620 10040 ts_tablet_manager.cc:1916] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:12.057878 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 824ff6913b0142d5889c75aa8e3cba3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 824ff6913b0142d5889c75aa8e3cba3e already in progress: creating tablet
I20251027 21:08:12.061196 11400 raft_consensus.cc:493] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.061350 11400 raft_consensus.cc:515] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.067660 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_875795479fc54534888d2b89e3480f29" } modify_external_catalogs: true
I20251027 21:08:12.067806 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_875795479fc54534888d2b89e3480f29" } modify_external_catalogs: true
I20251027 21:08:12.069124 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0f287ac2ad254338bd5151726cc40821" } new_table_name: "default.Table_c5fa3b7965474f9d9db43a7c6de2ded3" modify_external_catalogs: true
I20251027 21:08:12.070772 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0d7161154d094df4a95cdf85beffb72f" } modify_external_catalogs: true
I20251027 21:08:12.070904 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0d7161154d094df4a95cdf85beffb72f" } modify_external_catalogs: true
I20251027 21:08:12.074606 11373 raft_consensus.cc:493] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.074800 11373 raft_consensus.cc:515] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.075572 11373 leader_election.cc:290] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.076377 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4d26f73b8534f08bcaa386d4de75c01" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:12.076623 9771 raft_consensus.cc:2468] T a4d26f73b8534f08bcaa386d4de75c01 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:12.077167 9566 leader_election.cc:304] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:12.078843 11373 raft_consensus.cc:2804] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:12.078949 11373 raft_consensus.cc:493] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:12.078985 11411 raft_consensus.cc:493] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.079035 11373 raft_consensus.cc:3060] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:12.079119 11411 raft_consensus.cc:515] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.079742 11411 leader_election.cc:290] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.080288 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_14d4a985bc664c44b50d535a4757aa00" } modify_external_catalogs: true
I20251027 21:08:12.080482 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_14d4a985bc664c44b50d535a4757aa00" } modify_external_catalogs: true
I20251027 21:08:12.081348 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c34c8f1433b40d491ae05e7c6e60ba1" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:12.081614 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_904989a77e13498f8fa57464da33ecfe" } new_table_name: "default.Table_ffbfc9257b1f40b4827912923b9ac7fb" modify_external_catalogs: true
I20251027 21:08:12.081975 10040 ts_tablet_manager.cc:1929] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:12.082441 10040 log.cc:1199] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/cef2bfee222f45e6b6f20c6d63ff44d4
I20251027 21:08:12.082957 10040 ts_tablet_manager.cc:1950] T cef2bfee222f45e6b6f20c6d63ff44d4 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:12.083040 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1155ecb9419544b7b940e6699cb4464c" } modify_external_catalogs: true
I20251027 21:08:12.083194 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1155ecb9419544b7b940e6699cb4464c" } modify_external_catalogs: true
W20251027 21:08:12.083854 9566 leader_election.cc:343] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.084096 11373 raft_consensus.cc:515] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.085060 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1156) took 1505 ms. Trace:
I20251027 21:08:12.085515 11373 leader_election.cc:290] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.087050 11419 raft_consensus.cc:493] T f4187b2f66f24055b1df170f1062cc88 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.087175 11419 raft_consensus.cc:515] T f4187b2f66f24055b1df170f1062cc88 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.087694 11419 leader_election.cc:290] T f4187b2f66f24055b1df170f1062cc88 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:12.088821 9566 consensus_peers.cc:597] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94f2819dd0394530ae93754bdd1e482f. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:12.089288 10029 tablet_bootstrap.cc:492] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:12.092140 10029 tablet_bootstrap.cc:654] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.085361 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4d26f73b8534f08bcaa386d4de75c01" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:12.089313 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4187b2f66f24055b1df170f1062cc88" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:12.092988 10040 rpcz_store.cc:276] 1027 21:08:10.580018 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.580088 (+ 70us) service_pool.cc:225] Handling call
1027 21:08:12.055453 (+1475365us) ts_tablet_manager.cc:1151] Deleting tablet cef2bfee222f45e6b6f20c6d63ff44d4
1027 21:08:12.055461 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.066308 (+ 10847us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.078897 (+ 12589us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.085041 (+ 6144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1403207}
I20251027 21:08:12.091847 9769 raft_consensus.cc:3060] T e6b58b1f4f63461aa96e416cae879be8 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:12.093696 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1161) took 1406 ms. Trace:
I20251027 21:08:12.093777 10040 rpcz_store.cc:276] 1027 21:08:10.687127 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.687172 (+ 45us) service_pool.cc:225] Handling call
1027 21:08:12.093593 (+1406421us) ts_tablet_manager.cc:1151] Deleting tablet f954daa93d5a42bfabfbf8fd4b72172b
1027 21:08:12.093602 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.093681 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1406260}
I20251027 21:08:12.094674 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1164) took 1347 ms. Trace:
I20251027 21:08:12.094745 10040 rpcz_store.cc:276] 1027 21:08:10.747091 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.747147 (+ 56us) service_pool.cc:225] Handling call
1027 21:08:12.094574 (+1347427us) ts_tablet_manager.cc:1151] Deleting tablet 0166ba1e634747cd866fabc4e9acf1b5
1027 21:08:12.094580 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.094659 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1347231}
I20251027 21:08:12.095481 10031 tablet_bootstrap.cc:492] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:12.095688 10031 ts_tablet_manager.cc:1403] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.083s user 0.001s sys 0.017s
I20251027 21:08:12.096447 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1163) took 1349 ms. Trace:
I20251027 21:08:12.098320 10040 rpcz_store.cc:276] 1027 21:08:10.747009 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.747347 (+ 338us) service_pool.cc:225] Handling call
1027 21:08:12.096323 (+1348976us) ts_tablet_manager.cc:1151] Deleting tablet 22dbfa50996b47269983c107cc399000
1027 21:08:12.096334 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.096430 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1348828}
W20251027 21:08:12.098677 9566 leader_election.cc:343] T f4187b2f66f24055b1df170f1062cc88 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.093099 9771 raft_consensus.cc:3060] T a4d26f73b8534f08bcaa386d4de75c01 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:12.098843 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1171) took 1267 ms. Trace:
I20251027 21:08:12.098919 10040 rpcz_store.cc:276] 1027 21:08:10.831676 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.831735 (+ 59us) service_pool.cc:225] Handling call
1027 21:08:12.098748 (+1267013us) ts_tablet_manager.cc:1151] Deleting tablet 953b137f14a44955ae41ab503d04b43c
1027 21:08:12.098758 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.098827 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1266829}
I20251027 21:08:12.096832 10031 raft_consensus.cc:359] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.099182 10031 raft_consensus.cc:385] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.099246 10031 raft_consensus.cc:740] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.099443 10031 consensus_queue.cc:260] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.099867 10031 ts_tablet_manager.cc:1434] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251027 21:08:12.100095 10031 tablet_bootstrap.cc:492] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:12.101542 9771 raft_consensus.cc:2468] T a4d26f73b8534f08bcaa386d4de75c01 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:12.101850 10031 tablet_bootstrap.cc:654] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.101953 10043 ts_tablet_manager.cc:1929] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:12.102188 10043 log.cc:1199] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/76e4fb2bea7f4a4696fe9f6bb892bccc
I20251027 21:08:12.102555 10040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40732 (request call id 1170) took 1271 ms. Trace:
I20251027 21:08:12.102640 10040 rpcz_store.cc:276] 1027 21:08:10.831547 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.831982 (+ 435us) service_pool.cc:225] Handling call
1027 21:08:12.102444 (+1270462us) ts_tablet_manager.cc:1151] Deleting tablet 5cb8ca3a50c148e6b602e949650353f8
1027 21:08:12.102453 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.102540 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1270333}
I20251027 21:08:12.102988 9566 leader_election.cc:304] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:12.103395 11411 raft_consensus.cc:2804] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:12.103417 10043 ts_tablet_manager.cc:1950] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:12.103514 11411 raft_consensus.cc:697] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:12.103778 11411 consensus_queue.cc:237] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.104588 10029 tablet_bootstrap.cc:492] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:12.104691 10029 ts_tablet_manager.cc:1403] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20251027 21:08:12.105103 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet cef2bfee222f45e6b6f20c6d63ff44d4 (table default.Table_c3f0afa8e76249b69b9fbe4b3f67d020 [id=43518e3f60164df7a812afb41f6c4936]) successfully deleted
W20251027 21:08:12.105361 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet f954daa93d5a42bfabfbf8fd4b72172b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f954daa93d5a42bfabfbf8fd4b72172b already in progress: creating tablet
I20251027 21:08:12.105366 10029 raft_consensus.cc:359] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.105516 10029 raft_consensus.cc:385] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251027 21:08:12.105531 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 0166ba1e634747cd866fabc4e9acf1b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0166ba1e634747cd866fabc4e9acf1b5 already in progress: creating tablet
I20251027 21:08:12.105568 10029 raft_consensus.cc:740] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
W20251027 21:08:12.105712 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 22dbfa50996b47269983c107cc399000 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22dbfa50996b47269983c107cc399000 already in progress: creating tablet
I20251027 21:08:12.105775 10029 consensus_queue.cc:260] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:12.105880 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 953b137f14a44955ae41ab503d04b43c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 953b137f14a44955ae41ab503d04b43c already in progress: creating tablet
W20251027 21:08:12.106057 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 5cb8ca3a50c148e6b602e949650353f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cb8ca3a50c148e6b602e949650353f8 already in progress: creating tablet
I20251027 21:08:12.106129 10040 tablet_replica.cc:333] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:12.106232 10029 ts_tablet_manager.cc:1434] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251027 21:08:12.114154 10040 raft_consensus.cc:2243] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:12.114341 10040 raft_consensus.cc:2272] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.114900 10040 ts_tablet_manager.cc:1916] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.127028 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b1128d40cd914a9c9170d4d77fefa7cf" } modify_external_catalogs: true
I20251027 21:08:12.127223 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b1128d40cd914a9c9170d4d77fefa7cf" } modify_external_catalogs: true
I20251027 21:08:12.132180 10029 tablet_bootstrap.cc:492] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:12.132453 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1069) took 4412 ms. Trace:
I20251027 21:08:12.132527 10043 rpcz_store.cc:276] 1027 21:08:07.720232 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:07.720300 (+ 68us) service_pool.cc:225] Handling call
1027 21:08:11.709451 (+3989151us) ts_tablet_manager.cc:1151] Deleting tablet 76e4fb2bea7f4a4696fe9f6bb892bccc
1027 21:08:11.709463 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.015635 (+306172us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.096135 (+ 80500us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.132434 (+ 36299us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3988973}
I20251027 21:08:12.132961 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1087) took 3557 ms. Trace:
I20251027 21:08:12.133042 10043 rpcz_store.cc:276] 1027 21:08:08.575906 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.575949 (+ 43us) service_pool.cc:225] Handling call
1027 21:08:12.132822 (+3556873us) ts_tablet_manager.cc:1151] Deleting tablet 750b27c74ba3473dad743d60c21912c5
1027 21:08:12.132834 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.132940 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3556705}
I20251027 21:08:12.133338 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1088) took 3556 ms. Trace:
I20251027 21:08:12.133395 10043 rpcz_store.cc:276] 1027 21:08:08.576472 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.576521 (+ 49us) service_pool.cc:225] Handling call
1027 21:08:12.133254 (+3556733us) ts_tablet_manager.cc:1151] Deleting tablet a21a6a3725934266bbfdfc70b40de687
1027 21:08:12.133262 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.133325 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3556556}
I20251027 21:08:12.133666 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1089) took 3556 ms. Trace:
I20251027 21:08:12.133812 10029 tablet_bootstrap.cc:654] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.133735 10043 rpcz_store.cc:276] 1027 21:08:08.576850 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.576896 (+ 46us) service_pool.cc:225] Handling call
1027 21:08:12.133565 (+3556669us) ts_tablet_manager.cc:1151] Deleting tablet e53b2c28c75b43bdafa0d20f4ad61fcc
1027 21:08:12.133572 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.133639 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3556522}
I20251027 21:08:12.134025 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1090) took 3556 ms. Trace:
I20251027 21:08:12.134095 10043 rpcz_store.cc:276] 1027 21:08:08.577167 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.577212 (+ 45us) service_pool.cc:225] Handling call
1027 21:08:12.133920 (+3556708us) ts_tablet_manager.cc:1151] Deleting tablet 8827e4746ecc408a8fea98b6b5ddd57b
1027 21:08:12.133927 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.134009 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3556564}
I20251027 21:08:12.134413 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1091) took 3519 ms. Trace:
I20251027 21:08:12.134472 10043 rpcz_store.cc:276] 1027 21:08:08.615082 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.615202 (+ 120us) service_pool.cc:225] Handling call
1027 21:08:12.134319 (+3519117us) ts_tablet_manager.cc:1151] Deleting tablet cc97e55cfbcb41ec8c6c7410bbf14fb3
1027 21:08:12.134326 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.134399 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3518954}
I20251027 21:08:12.134752 10043 tablet_replica.cc:333] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
W20251027 21:08:12.136816 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet a21a6a3725934266bbfdfc70b40de687 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a21a6a3725934266bbfdfc70b40de687 already in progress: creating tablet
W20251027 21:08:12.137045 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet e53b2c28c75b43bdafa0d20f4ad61fcc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e53b2c28c75b43bdafa0d20f4ad61fcc already in progress: creating tablet
W20251027 21:08:12.137231 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 8827e4746ecc408a8fea98b6b5ddd57b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8827e4746ecc408a8fea98b6b5ddd57b already in progress: creating tablet
W20251027 21:08:12.137476 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet cc97e55cfbcb41ec8c6c7410bbf14fb3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc97e55cfbcb41ec8c6c7410bbf14fb3 already in progress: creating tablet
I20251027 21:08:12.142527 10031 tablet_bootstrap.cc:492] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:12.142751 10031 ts_tablet_manager.cc:1403] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20251027 21:08:12.146801 10029 tablet_bootstrap.cc:492] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:12.147001 10029 ts_tablet_manager.cc:1403] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20251027 21:08:12.147754 10029 raft_consensus.cc:359] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.147984 10029 raft_consensus.cc:385] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.148089 10029 raft_consensus.cc:740] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.148343 10029 consensus_queue.cc:260] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.148867 10029 ts_tablet_manager.cc:1434] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:12.149191 10029 tablet_bootstrap.cc:492] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:12.149401 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2689f2aa6d3e4d30b155fa18f7f6b903"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:12.150092 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2689f2aa6d3e4d30b155fa18f7f6b903 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:12.150527 10031 raft_consensus.cc:359] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.150769 10031 raft_consensus.cc:385] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.150882 10029 tablet_bootstrap.cc:654] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.151266 10031 raft_consensus.cc:740] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.151540 10031 consensus_queue.cc:260] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.152113 10031 ts_tablet_manager.cc:1434] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20251027 21:08:12.152424 10031 tablet_bootstrap.cc:492] T f2710f265ae9485dafec389168328067 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:12.154853 11255 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b8b31d47da48aaa386a8e64a5749c2" member_type: VOTER last_known_addr { host: "127.9.14.126" port: 34057 } } peers { permanent_uuid: "18b15927ce5744ad8ba81dda52725c11" member_type: VOTER last_known_addr { host: "127.9.14.125" port: 46313 } } peers { permanent_uuid: "6e86ca5e83dd4df5b72f045e7b787e81" member_type: VOTER last_known_addr { host: "127.9.14.124" port: 45071 } } }
I20251027 21:08:12.155012 11255 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 [sys.catalog]: This master's current role is: LEADER
I20251027 21:08:12.165504 10040 ts_tablet_manager.cc:1929] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:12.165645 10040 log.cc:1199] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/01e9db9eaf8e41c6a2ee13b9ed127421
I20251027 21:08:12.165931 10040 ts_tablet_manager.cc:1950] T 01e9db9eaf8e41c6a2ee13b9ed127421 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:12.170295 10031 tablet_bootstrap.cc:654] T f2710f265ae9485dafec389168328067 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.172778 9911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "986bf148b3e9482fa2fe53bbac54750d" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:12.173012 9911 raft_consensus.cc:2468] T 986bf148b3e9482fa2fe53bbac54750d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:12.179598 10031 tablet_bootstrap.cc:492] T f2710f265ae9485dafec389168328067 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:12.181840 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet 7057c55a820d4e379fe1d9abfa353821 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:12.182300 10031 ts_tablet_manager.cc:1403] T f2710f265ae9485dafec389168328067 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20251027 21:08:12.183669 10031 raft_consensus.cc:359] T f2710f265ae9485dafec389168328067 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.185573 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bfa70ded2f54e3fb2ca744c723395e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:12.185987 11400 leader_election.cc:290] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.187414 10031 raft_consensus.cc:385] T f2710f265ae9485dafec389168328067 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.187602 10031 raft_consensus.cc:740] T f2710f265ae9485dafec389168328067 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.188393 9888 tablet_service.cc:1552] Processing DeleteTablet for tablet d27f9c498eca4e5f8d2bfe9c805d06b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:12.188876 9888 tablet_service.cc:1552] Processing DeleteTablet for tablet 5cf3dd1b5f4b4b4ba112df78bf0df8f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:12.189818 9891 tablet_service.cc:1505] Processing CreateTablet for tablet ed8a50cfa7d84c1182b7ce5daf38d7cb (DEFAULT_TABLE table=default.Table_1f15260477b0409794ec2c672c472be3 [id=af4c904ad30b4821ba59915937080ef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.190258 10043 raft_consensus.cc:2243] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:12.190938 10043 pending_rounds.cc:70] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14: Trying to abort 1 pending ops.
I20251027 21:08:12.191007 10043 pending_rounds.cc:77] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510700459458560 op_type: NO_OP noop_request { }
I20251027 21:08:12.191116 10043 raft_consensus.cc:2889] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:12.191188 10043 raft_consensus.cc:2272] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.191330 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bc5fbdbbea0406a961e1a4df51b8762" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:12.187891 10031 consensus_queue.cc:260] T f2710f265ae9485dafec389168328067 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.191776 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 01e9db9eaf8e41c6a2ee13b9ed127421 (table default.Table_11fbadd9eed14cdba7664715e389f7e5 [id=83ec44dba5c444aba48846811a5f5fc9]) successfully deleted
W20251027 21:08:12.191792 9859 leader_election.cc:343] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:12.192134 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet bd6152f7ac8a4123872098955b8837ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd6152f7ac8a4123872098955b8837ac already in progress: creating tablet
I20251027 21:08:12.193084 10040 tablet_replica.cc:333] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:12.193243 10040 raft_consensus.cc:2243] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
W20251027 21:08:12.198479 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8885465e81444b468b96955f0e44e9d2 consensus state has no leader
I20251027 21:08:12.198745 9310 master_service.cc:954] ReplaceTablet: received request to replace tablet eeb667b3b10746dbad351c4c123ffd97 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:12.200227 11419 raft_consensus.cc:493] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.200354 11419 raft_consensus.cc:515] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.200675 9615 tablet_service.cc:1505] Processing CreateTablet for tablet 2a9a8b179a7b4f8bba74343814e0461e (DEFAULT_TABLE table=default.Table_8fc21ba8792f4de0aa3bb6f4b76db889 [id=f1ea6b50c9824147942fc173dee793dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.200959 11419 leader_election.cc:290] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.201062 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a9a8b179a7b4f8bba74343814e0461e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.201529 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aabaa34787cf4d179822269b7f323314" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:12.201707 9772 raft_consensus.cc:2468] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:12.202224 9566 leader_election.cc:304] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
W20251027 21:08:12.202674 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5a4118310fd45609f7f0b7d12d253ca consensus state has no leader
I20251027 21:08:12.203487 10029 tablet_bootstrap.cc:492] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:12.203655 10029 ts_tablet_manager.cc:1403] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.011s
I20251027 21:08:12.203994 9613 tablet_service.cc:1505] Processing CreateTablet for tablet 46ed8d7e48c24914b8166940fc8d1842 (DEFAULT_TABLE table=default.Table_70046cf1a7e44ca287756df96f94e51e [id=8d019f69a74c4ca3aff21211946556cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.204531 10029 raft_consensus.cc:359] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.204689 10029 raft_consensus.cc:385] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.204733 10029 raft_consensus.cc:740] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.204753 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46ed8d7e48c24914b8166940fc8d1842. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.204907 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 594dba3c9d0b4c0ea3089073c2ea8ad8 (DEFAULT_TABLE table=default.Table_554983eff7d04fefb9f5f9a7bf6ccc55 [id=ec241f5b424648e4b5205146cf1a1c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.204898 10029 consensus_queue.cc:260] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.205144 11412 raft_consensus.cc:2804] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:12.205202 11412 raft_consensus.cc:493] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:12.205258 11412 raft_consensus.cc:3060] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:12.207894 10029 ts_tablet_manager.cc:1434] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:12.208580 11412 raft_consensus.cc:515] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.209136 11412 leader_election.cc:290] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.211002 9602 tablet_service.cc:1505] Processing CreateTablet for tablet afff93eb06ab4bcca2990c8bc534b234 (DEFAULT_TABLE table=default.Table_368703da645247fb87af3db268932e92 [id=0f7b2bb531b042c59c311cdeb55f4edd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.211179 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aabaa34787cf4d179822269b7f323314" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:12.211352 9772 raft_consensus.cc:3060] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:12.212299 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 1f49db288462455d9afd8e49db4841a8 (DEFAULT_TABLE table=default.Table_fb3083bf1d0a4db382cd40ee844e8d50 [id=cf8a23a0ab4745a5a6b396cc8dfb6909]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.213407 9603 tablet_service.cc:1505] Processing CreateTablet for tablet bbc08f77a3ac46f8ab38d9ee47972af9 (DEFAULT_TABLE table=default.Table_00304e11dcc548499ea100119c5c976e [id=80b60763d16140d3b9ec411309509745]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.214033 9772 raft_consensus.cc:2468] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:12.215091 9604 tablet_service.cc:1505] Processing CreateTablet for tablet 5d6d0a77fbec49b780f7f71664411cb6 (DEFAULT_TABLE table=default.Table_67fb7650bd3745438426fc06514bcc9c [id=458746f594a841389948bb1af86b4fff]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:12.216007 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 0d0099e7b3bd403590ca5c9c73198853 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d0099e7b3bd403590ca5c9c73198853 already in progress: creating tablet
I20251027 21:08:12.217155 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 594dba3c9d0b4c0ea3089073c2ea8ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.217396 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afff93eb06ab4bcca2990c8bc534b234. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.218906 9599 tablet_service.cc:1505] Processing CreateTablet for tablet e630ff4532584128a21ff1aebfe4d05b (DEFAULT_TABLE table=default.Table_9695e56a968e424c84c3dd11d12d5fb5 [id=45a2c14cb0794f47b484065f9fe6efe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.219206 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e630ff4532584128a21ff1aebfe4d05b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.219383 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 538c1a8045da4864ab2f4c8b0c672f29 (DEFAULT_TABLE table=default.Table_10376e17301944fe9c799d6a87ff1f34 [id=47313707951a4a4ea44cc4ab4e62736b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.219727 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 538c1a8045da4864ab2f4c8b0c672f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.220355 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed8a50cfa7d84c1182b7ce5daf38d7cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.222352 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f49db288462455d9afd8e49db4841a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.224277 9905 tablet_service.cc:1505] Processing CreateTablet for tablet afff93eb06ab4bcca2990c8bc534b234 (DEFAULT_TABLE table=default.Table_368703da645247fb87af3db268932e92 [id=0f7b2bb531b042c59c311cdeb55f4edd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.224614 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afff93eb06ab4bcca2990c8bc534b234. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.225026 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 46ed8d7e48c24914b8166940fc8d1842 (DEFAULT_TABLE table=default.Table_70046cf1a7e44ca287756df96f94e51e [id=8d019f69a74c4ca3aff21211946556cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.225351 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46ed8d7e48c24914b8166940fc8d1842. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.225744 9894 tablet_service.cc:1505] Processing CreateTablet for tablet 594dba3c9d0b4c0ea3089073c2ea8ad8 (DEFAULT_TABLE table=default.Table_554983eff7d04fefb9f5f9a7bf6ccc55 [id=ec241f5b424648e4b5205146cf1a1c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.226048 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 594dba3c9d0b4c0ea3089073c2ea8ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.226967 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbc08f77a3ac46f8ab38d9ee47972af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.227162 11400 raft_consensus.cc:493] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.227285 11400 raft_consensus.cc:515] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.230716 9598 tablet_service.cc:1505] Processing CreateTablet for tablet 5bc30731304648d8bc702cf7793b8d22 (DEFAULT_TABLE table=default.Table_c26460a4ac0742d997bcc2dc9d88ce5e [id=f086c981c5694fac91b28c0a0236bbfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.230780 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 1f49db288462455d9afd8e49db4841a8 (DEFAULT_TABLE table=default.Table_fb3083bf1d0a4db382cd40ee844e8d50 [id=cf8a23a0ab4745a5a6b396cc8dfb6909]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.231051 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f49db288462455d9afd8e49db4841a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.231051 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bc30731304648d8bc702cf7793b8d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.231750 11400 leader_election.cc:290] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.231992 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d6d0a77fbec49b780f7f71664411cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.232513 9887 tablet_service.cc:1505] Processing CreateTablet for tablet e630ff4532584128a21ff1aebfe4d05b (DEFAULT_TABLE table=default.Table_9695e56a968e424c84c3dd11d12d5fb5 [id=45a2c14cb0794f47b484065f9fe6efe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.232786 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e630ff4532584128a21ff1aebfe4d05b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:12.233999 9568 leader_election.cc:343] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: a033551a591146039b017b4193b221ca
I20251027 21:08:12.233968 11419 consensus_queue.cc:1048] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:12.230711 9566 leader_election.cc:304] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
W20251027 21:08:12.234464 9568 leader_election.cc:343] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: a033551a591146039b017b4193b221ca
I20251027 21:08:12.234654 11419 raft_consensus.cc:2804] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:12.234730 9568 leader_election.cc:304] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.234782 11419 raft_consensus.cc:697] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:12.234984 11373 raft_consensus.cc:2804] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:12.235044 11373 raft_consensus.cc:493] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:12.235293 11373 raft_consensus.cc:3060] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:12.236251 11400 raft_consensus.cc:493] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.236356 11400 raft_consensus.cc:515] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.236864 11400 leader_election.cc:290] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.237527 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 2a9a8b179a7b4f8bba74343814e0461e (DEFAULT_TABLE table=default.Table_8fc21ba8792f4de0aa3bb6f4b76db889 [id=f1ea6b50c9824147942fc173dee793dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.237794 11373 raft_consensus.cc:515] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.237931 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a9a8b179a7b4f8bba74343814e0461e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.239111 9903 tablet_service.cc:1505] Processing CreateTablet for tablet bbc08f77a3ac46f8ab38d9ee47972af9 (DEFAULT_TABLE table=default.Table_00304e11dcc548499ea100119c5c976e [id=80b60763d16140d3b9ec411309509745]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.239724 11373 leader_election.cc:290] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.240286 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "986bf148b3e9482fa2fe53bbac54750d" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:12.240864 9566 leader_election.cc:343] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.241066 10032 tablet_bootstrap.cc:492] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:12.241189 10032 ts_tablet_manager.cc:1403] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.418s user 0.090s sys 0.164s
W20251027 21:08:12.244043 9568 leader_election.cc:343] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 9c2d713d72fd4b5ca34cb3e828c75761
I20251027 21:08:12.244148 9568 leader_election.cc:304] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.244649 11412 raft_consensus.cc:2749] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251027 21:08:12.245260 9568 leader_election.cc:343] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: a4d26f73b8534f08bcaa386d4de75c01
W20251027 21:08:12.245507 9568 leader_election.cc:343] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 5c34c8f1433b40d491ae05e7c6e60ba1
I20251027 21:08:12.245610 9568 leader_election.cc:304] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.245971 10029 tablet_bootstrap.cc:492] T 842c1a1298234d9ea2719bab29c54ae3 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:12.246253 11412 raft_consensus.cc:2749] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.235044 11419 consensus_queue.cc:237] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.257412 10040 pending_rounds.cc:70] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
I20251027 21:08:12.257493 10040 pending_rounds.cc:77] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510698536325120 op_type: NO_OP noop_request { }
I20251027 21:08:12.257587 10040 raft_consensus.cc:2889] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:12.257705 10040 raft_consensus.cc:2272] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.258493 11437 raft_consensus.cc:493] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.258661 11437 raft_consensus.cc:515] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.258909 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 46ed8d7e48c24914b8166940fc8d1842 (DEFAULT_TABLE table=default.Table_70046cf1a7e44ca287756df96f94e51e [id=8d019f69a74c4ca3aff21211946556cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.259168 11437 leader_election.cc:290] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:12.259282 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46ed8d7e48c24914b8166940fc8d1842. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.259378 9748 tablet_service.cc:1505] Processing CreateTablet for tablet e630ff4532584128a21ff1aebfe4d05b (DEFAULT_TABLE table=default.Table_9695e56a968e424c84c3dd11d12d5fb5 [id=45a2c14cb0794f47b484065f9fe6efe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.259447 9737 tablet_service.cc:1505] Processing CreateTablet for tablet bbc08f77a3ac46f8ab38d9ee47972af9 (DEFAULT_TABLE table=default.Table_00304e11dcc548499ea100119c5c976e [id=80b60763d16140d3b9ec411309509745]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.259541 9738 tablet_service.cc:1505] Processing CreateTablet for tablet afff93eb06ab4bcca2990c8bc534b234 (DEFAULT_TABLE table=default.Table_368703da645247fb87af3db268932e92 [id=0f7b2bb531b042c59c311cdeb55f4edd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.259691 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e630ff4532584128a21ff1aebfe4d05b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.259889 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 5d6d0a77fbec49b780f7f71664411cb6 (DEFAULT_TABLE table=default.Table_67fb7650bd3745438426fc06514bcc9c [id=458746f594a841389948bb1af86b4fff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.260042 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 5bc30731304648d8bc702cf7793b8d22 (DEFAULT_TABLE table=default.Table_c26460a4ac0742d997bcc2dc9d88ce5e [id=f086c981c5694fac91b28c0a0236bbfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.260141 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d6d0a77fbec49b780f7f71664411cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.260308 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bc30731304648d8bc702cf7793b8d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.261327 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbc08f77a3ac46f8ab38d9ee47972af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.262380 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 594dba3c9d0b4c0ea3089073c2ea8ad8 (DEFAULT_TABLE table=default.Table_554983eff7d04fefb9f5f9a7bf6ccc55 [id=ec241f5b424648e4b5205146cf1a1c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.262663 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 594dba3c9d0b4c0ea3089073c2ea8ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.265594 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afff93eb06ab4bcca2990c8bc534b234. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.266362 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 1f49db288462455d9afd8e49db4841a8 (DEFAULT_TABLE table=default.Table_fb3083bf1d0a4db382cd40ee844e8d50 [id=cf8a23a0ab4745a5a6b396cc8dfb6909]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.266654 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f49db288462455d9afd8e49db4841a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.271200 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aabaa34787cf4d179822269b7f323314" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:12.272192 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 2a9a8b179a7b4f8bba74343814e0461e (DEFAULT_TABLE table=default.Table_8fc21ba8792f4de0aa3bb6f4b76db889 [id=f1ea6b50c9824147942fc173dee793dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.272497 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a9a8b179a7b4f8bba74343814e0461e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.274597 10040 ts_tablet_manager.cc:1916] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.276314 10029 tablet_bootstrap.cc:654] T 842c1a1298234d9ea2719bab29c54ae3 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.276271 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "823620e5486e4cf982fdf9105e8cc4f1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:12.276847 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b8f76c33444bcb9b488af65e4d3e6b" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:12.280925 11400 raft_consensus.cc:493] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.281061 11400 raft_consensus.cc:515] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.281441 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 5d6d0a77fbec49b780f7f71664411cb6 (DEFAULT_TABLE table=default.Table_67fb7650bd3745438426fc06514bcc9c [id=458746f594a841389948bb1af86b4fff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.281566 10040 ts_tablet_manager.cc:1929] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:12.281703 10040 log.cc:1199] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/76e4fb2bea7f4a4696fe9f6bb892bccc
I20251027 21:08:12.282083 10040 ts_tablet_manager.cc:1950] T 76e4fb2bea7f4a4696fe9f6bb892bccc P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:12.282567 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d6d0a77fbec49b780f7f71664411cb6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:12.283754 9859 leader_election.cc:343] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:12.283982 9859 leader_election.cc:343] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.289348 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbc08f77a3ac46f8ab38d9ee47972af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.291448 9919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "986bf148b3e9482fa2fe53bbac54750d" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:12.291666 9919 raft_consensus.cc:3060] T 986bf148b3e9482fa2fe53bbac54750d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:12.294816 10043 ts_tablet_manager.cc:1916] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.304418 10043 ts_tablet_manager.cc:1929] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:12.304629 10043 log.cc:1199] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/a4d26f73b8534f08bcaa386d4de75c01
I20251027 21:08:12.305337 10043 ts_tablet_manager.cc:1950] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:12.305882 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1095) took 3690 ms. Trace:
I20251027 21:08:12.306033 10043 rpcz_store.cc:276] 1027 21:08:08.615407 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.615449 (+ 42us) service_pool.cc:225] Handling call
1027 21:08:12.134702 (+3519253us) ts_tablet_manager.cc:1151] Deleting tablet a4d26f73b8534f08bcaa386d4de75c01
1027 21:08:12.134713 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.190765 (+ 56052us) spinlock_profiling.cc:243] Waited 55.3 ms on lock 0x6160001b4c54. stack: 00007f17445599f6 00007f17445597b0 00007f17435353ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f1750424100 00007f1751771ae3 00007f1754332287 00007f17543652d4 00007f1754345d70 00007f1754345afd 0000000000a8867c 00007f174459fd7f 00007f17445a304b 00007f17445a2ec8
1027 21:08:12.301095 (+110330us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.304320 (+ 3225us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.305860 (+ 1540us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":127243648,"tablet-delete.queue_time_us":3519095}
I20251027 21:08:12.306614 10043 tablet_replica.cc:333] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:12.306891 10043 raft_consensus.cc:2243] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:12.307128 10043 raft_consensus.cc:2272] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.307891 10043 ts_tablet_manager.cc:1916] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.302547 11400 leader_election.cc:290] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:12.282728 9711 leader_election.cc:343] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: aabaa34787cf4d179822269b7f323314
I20251027 21:08:12.321166 9711 leader_election.cc:304] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:12.322084 11514 raft_consensus.cc:2749] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251027 21:08:12.308317 10031 ts_tablet_manager.cc:1434] T f2710f265ae9485dafec389168328067 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.126s user 0.013s sys 0.000s
I20251027 21:08:12.322621 10031 tablet_bootstrap.cc:492] T 29693cf06bc24930870dc60457126519 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:12.322404 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 76e4fb2bea7f4a4696fe9f6bb892bccc (table default.Table_b7607fa1c70d431d97f9e3f8a7c3d667 [id=080bf50609c045bd9b05f4e9144feac5]) successfully deleted
I20251027 21:08:12.324213 10031 tablet_bootstrap.cc:654] T 29693cf06bc24930870dc60457126519 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.324553 11400 raft_consensus.cc:493] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.324685 11400 raft_consensus.cc:515] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.325227 11400 leader_election.cc:290] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:12.325851 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 191009830c30467b9c5ab39e13b7c267 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 191009830c30467b9c5ab39e13b7c267 already in progress: creating tablet
W20251027 21:08:12.326606 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 278d8f448ef144148cabfc3a57bd24fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 278d8f448ef144148cabfc3a57bd24fd already in progress: creating tablet
W20251027 21:08:12.327273 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 90f96c21da9944f1ade05d35787d47e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90f96c21da9944f1ade05d35787d47e2 already in progress: creating tablet
W20251027 21:08:12.327903 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet b162734c536a40c08755201a85255f4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b162734c536a40c08755201a85255f4c already in progress: creating tablet
I20251027 21:08:12.328469 10031 tablet_bootstrap.cc:492] T 29693cf06bc24930870dc60457126519 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
W20251027 21:08:12.328490 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet b6c4fc648fa04d9aa4a10b5b6b07cd21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6c4fc648fa04d9aa4a10b5b6b07cd21 already in progress: creating tablet
I20251027 21:08:12.328680 10031 ts_tablet_manager.cc:1403] T 29693cf06bc24930870dc60457126519 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
W20251027 21:08:12.329229 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet e4b84d8c0ee74dcbaee8aeb49aa30c57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4b84d8c0ee74dcbaee8aeb49aa30c57 already in progress: creating tablet
W20251027 21:08:12.329933 9297 catalog_manager.cc:4952] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c3b9a461567a48fcb4af514ecdf618b4 because the tablet was not found. No further retry: Not found: Tablet not found: c3b9a461567a48fcb4af514ecdf618b4
I20251027 21:08:12.330269 10040 tablet_replica.cc:333] T f775a7e42bdc4910bdb604b26a28a7f4 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:12.330451 10040 raft_consensus.cc:2243] T f775a7e42bdc4910bdb604b26a28a7f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:12.330590 10040 raft_consensus.cc:2272] T f775a7e42bdc4910bdb604b26a28a7f4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.332142 10031 raft_consensus.cc:359] T 29693cf06bc24930870dc60457126519 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.333009 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 5bc30731304648d8bc702cf7793b8d22 (DEFAULT_TABLE table=default.Table_c26460a4ac0742d997bcc2dc9d88ce5e [id=f086c981c5694fac91b28c0a0236bbfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.333204 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 1bd2e223198a4af9baca11a8ff21ab63 (DEFAULT_TABLE table=default.Table_a23d9519c070420782525d2a7397288c [id=bf11d0dacc1f423a9cd84e0fb27d02cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.333365 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bc30731304648d8bc702cf7793b8d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.333493 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bd2e223198a4af9baca11a8ff21ab63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.309564 10029 tablet_bootstrap.cc:492] T 842c1a1298234d9ea2719bab29c54ae3 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:12.336728 10029 ts_tablet_manager.cc:1403] T 842c1a1298234d9ea2719bab29c54ae3 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.091s user 0.005s sys 0.005s
I20251027 21:08:12.337566 10029 raft_consensus.cc:359] T 842c1a1298234d9ea2719bab29c54ae3 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.337826 10029 raft_consensus.cc:385] T 842c1a1298234d9ea2719bab29c54ae3 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.337944 10029 raft_consensus.cc:740] T 842c1a1298234d9ea2719bab29c54ae3 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.338243 10029 consensus_queue.cc:260] T 842c1a1298234d9ea2719bab29c54ae3 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.338793 10029 ts_tablet_manager.cc:1434] T 842c1a1298234d9ea2719bab29c54ae3 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:12.340996 10029 tablet_bootstrap.cc:492] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:12.346223 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 1e4120d22d184fa588fd48642d93dd27 (DEFAULT_TABLE table=default.Table_7c14877bc32245418a930cf51449c007 [id=820d0e68ebe3424cbb3507306d805429]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.346676 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e4120d22d184fa588fd48642d93dd27. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.346741 10029 tablet_bootstrap.cc:654] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.350842 11519 tablet.cc:1722] T b8856c5dc8f144358445165324bfb477 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251027 21:08:12.351024 10041 ts_tablet_manager.cc:1929] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:12.351145 10041 log.cc:1199] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/00c90ff84d3f4a7f92fda235416eed1d
I20251027 21:08:12.351468 10041 ts_tablet_manager.cc:1950] T 00c90ff84d3f4a7f92fda235416eed1d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:12.351743 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1135) took 2894 ms. Trace:
I20251027 21:08:12.351828 10041 rpcz_store.cc:276] 1027 21:08:09.457680 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.457770 (+ 90us) service_pool.cc:225] Handling call
1027 21:08:11.859114 (+2401344us) ts_tablet_manager.cc:1151] Deleting tablet 00c90ff84d3f4a7f92fda235416eed1d
1027 21:08:11.859123 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.221968 (+362845us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.344883 (+122915us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.351731 (+ 6848us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2398362}
I20251027 21:08:12.352150 10041 tablet_replica.cc:333] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:12.352504 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 03986aea266e45d49a969ed5cad865d6 (DEFAULT_TABLE table=default.Table_db3806bdc5c2498294b20102a1767b93 [id=2fe4f21376ce4594a6c51d39e373cbfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.352909 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03986aea266e45d49a969ed5cad865d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.355930 10041 raft_consensus.cc:2243] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:12.356094 10041 raft_consensus.cc:2272] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.356595 10041 ts_tablet_manager.cc:1916] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.357108 10031 raft_consensus.cc:385] T 29693cf06bc24930870dc60457126519 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.357230 10031 raft_consensus.cc:740] T 29693cf06bc24930870dc60457126519 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.357688 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 5e3b3d9d43e643f6828cb6078dd8650e (DEFAULT_TABLE table=default.Table_b7e7e18292df4076a52ea8b99c94f901 [id=e2ad8c077bc84fdeb54e41fa643d519e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.357554 10031 consensus_queue.cc:260] T 29693cf06bc24930870dc60457126519 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.358273 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 00c90ff84d3f4a7f92fda235416eed1d (table default.Table_b36a756b7ab54252aaa0c08e563b2ea3 [id=14a74e3dd6ef4f3ba0ef5f190c0f393c]) successfully deleted
I20251027 21:08:12.336241 10040 ts_tablet_manager.cc:1916] T f775a7e42bdc4910bdb604b26a28a7f4 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.343071 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 1e4120d22d184fa588fd48642d93dd27 (DEFAULT_TABLE table=default.Table_7c14877bc32245418a930cf51449c007 [id=820d0e68ebe3424cbb3507306d805429]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.358392 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e3b3d9d43e643f6828cb6078dd8650e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.358624 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e4120d22d184fa588fd48642d93dd27. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.359207 10029 tablet_bootstrap.cc:492] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:12.362335 10029 ts_tablet_manager.cc:1403] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.003s
I20251027 21:08:12.363099 10029 raft_consensus.cc:359] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.363320 10029 raft_consensus.cc:385] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.363420 10029 raft_consensus.cc:740] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.363641 10029 consensus_queue.cc:260] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.364158 10029 ts_tablet_manager.cc:1434] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:12.366533 10031 ts_tablet_manager.cc:1434] T 29693cf06bc24930870dc60457126519 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.038s user 0.000s sys 0.002s
I20251027 21:08:12.371028 10031 tablet_bootstrap.cc:492] T 631ff72613734913818a3cad24f20dbc P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:08:12.361047 9568 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 115542us
W20251027 21:08:12.374629 9568 leader_election.cc:343] T f4187b2f66f24055b1df170f1062cc88 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: f4187b2f66f24055b1df170f1062cc88
I20251027 21:08:12.374853 9568 leader_election.cc:304] T f4187b2f66f24055b1df170f1062cc88 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.375391 11512 raft_consensus.cc:2749] T f4187b2f66f24055b1df170f1062cc88 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:12.375876 9568 leader_election.cc:343] T a4d26f73b8534f08bcaa386d4de75c01 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: a4d26f73b8534f08bcaa386d4de75c01
W20251027 21:08:12.377740 9568 leader_election.cc:343] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: aabaa34787cf4d179822269b7f323314
W20251027 21:08:12.378054 9568 leader_election.cc:343] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: aabaa34787cf4d179822269b7f323314
I20251027 21:08:12.378507 10031 tablet_bootstrap.cc:654] T 631ff72613734913818a3cad24f20dbc P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:12.381171 9299 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 9ace4188d3494a05a5678381f8d8fd17 (table default.Table_9bf7b754a163446db873b9b81a700e4d [id=54ab773e30384ae98a0b0539284c85b4], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9ace4188d3494a05a5678381f8d8fd17 consensus state has no leader
I20251027 21:08:12.355233 11521 tablet.cc:1722] T b8856c5dc8f144358445165324bfb477 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251027 21:08:12.335832 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 03986aea266e45d49a969ed5cad865d6 (DEFAULT_TABLE table=default.Table_db3806bdc5c2498294b20102a1767b93 [id=2fe4f21376ce4594a6c51d39e373cbfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.382380 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03986aea266e45d49a969ed5cad865d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.389434 9745 tablet_service.cc:1505] Processing CreateTablet for tablet c34e487f57bd41bca440b0ad0179a075 (DEFAULT_TABLE table=default.Table_3c794af372fc4ed8ab15aecdf30067f9 [id=859536014530491888672609efaa6327]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.389860 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34e487f57bd41bca440b0ad0179a075. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.390383 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_ad34618843dc4a708617166fb4909e93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:12.390913 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad34618843dc4a708617166fb4909e93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:12.390033 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4ecc813240f9453ab4ec46a231f2b0d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:12.392439 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ecc813240f9453ab4ec46a231f2b0d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:12.404724 10043 ts_tablet_manager.cc:1929] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:12.404891 10043 log.cc:1199] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/9861a0efcba64141aae450663c5d83de
I20251027 21:08:12.405237 10043 ts_tablet_manager.cc:1950] T 9861a0efcba64141aae450663c5d83de P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:12.406113 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1096) took 3790 ms. Trace:
I20251027 21:08:12.406800 9752 tablet_service.cc:1505] Processing CreateTablet for tablet fc7c554d37b1419d899cff37db371221 (DEFAULT_TABLE table=default.Table_548bc7accdaa4dd8bfb6373d3d15667d [id=89cc203955a9451aae42cc8477eec02b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.407225 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc7c554d37b1419d899cff37db371221. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.408506 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 283b72f626644ac2b9b61186387b3839
I20251027 21:08:12.408865 9309 catalog_manager.cc:6719] ReplaceTablet: tablet 283b72f626644ac2b9b61186387b3839 deleted and replaced by tablet f5bf7b066583425593b89b1d3eb2b82d
I20251027 21:08:12.410511 9737 tablet_service.cc:1552] Processing DeleteTablet for tablet 283b72f626644ac2b9b61186387b3839 with delete_type TABLET_DATA_DELETED (replaced by tablet f5bf7b066583425593b89b1d3eb2b82d) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:12.411873 9306 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet dd9e6656d8df43afbb7f70706bc14585 (table default.Table_f78d839900634631b6963b2d63f0d084 [id=3cdf432ec43d42e8bc2d253a92eac8d3]). Expected version 3 got 0
I20251027 21:08:12.415869 10040 ts_tablet_manager.cc:1929] T f775a7e42bdc4910bdb604b26a28a7f4 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:12.415980 10040 log.cc:1199] T f775a7e42bdc4910bdb604b26a28a7f4 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/f775a7e42bdc4910bdb604b26a28a7f4
I20251027 21:08:12.415918 9898 tablet_service.cc:1505] Processing CreateTablet for tablet fc7c554d37b1419d899cff37db371221 (DEFAULT_TABLE table=default.Table_548bc7accdaa4dd8bfb6373d3d15667d [id=89cc203955a9451aae42cc8477eec02b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.416324 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc7c554d37b1419d899cff37db371221. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.416368 9309 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet e6b58b1f4f63461aa96e416cae879be8 (table default.Table_f3c2de263e8849caaef87b570fc69459 [id=ef50ef2238804c98ad19c5b1c0b881c4]). Expected version 1 got 0
I20251027 21:08:12.417155 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 5e3b3d9d43e643f6828cb6078dd8650e (DEFAULT_TABLE table=default.Table_b7e7e18292df4076a52ea8b99c94f901 [id=e2ad8c077bc84fdeb54e41fa643d519e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.417205 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_36902a42180149cbb6d361295f2a32cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:12.417510 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e3b3d9d43e643f6828cb6078dd8650e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:12.418689 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36902a42180149cbb6d361295f2a32cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:12.420481 10040 ts_tablet_manager.cc:1950] T f775a7e42bdc4910bdb604b26a28a7f4 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:12.420673 9905 tablet_service.cc:1505] Processing CreateTablet for tablet c34e487f57bd41bca440b0ad0179a075 (DEFAULT_TABLE table=default.Table_3c794af372fc4ed8ab15aecdf30067f9 [id=859536014530491888672609efaa6327]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.420966 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34e487f57bd41bca440b0ad0179a075. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.421934 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 283b72f626644ac2b9b61186387b3839 with delete_type TABLET_DATA_DELETED (replaced by tablet f5bf7b066583425593b89b1d3eb2b82d) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:12.422281 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 054d78c606d64944b32618b93615befd (DEFAULT_TABLE table=default.Table_877147b58eff438190609ce0142c08f5 [id=05da7a8eb8ea441d90763f0bf6abd69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.422657 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 054d78c606d64944b32618b93615befd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.423393 9306 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet aaaf70c950044387a63820079db3c829 (table default.Table_e08f0cbb71674a44a697686c1b6effc9 [id=471f86ddb9de40a7ae58ec202926a1fd]). Expected version 1 got 0
I20251027 21:08:12.425453 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 054d78c606d64944b32618b93615befd (DEFAULT_TABLE table=default.Table_877147b58eff438190609ce0142c08f5 [id=05da7a8eb8ea441d90763f0bf6abd69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.425521 9736 tablet_service.cc:1505] Processing CreateTablet for tablet d3bd5f101f58440db7097a11ad1313bb (DEFAULT_TABLE table=default.Table_09274fe9916e428a82701f8bd406e392 [id=470d302c6ca341f198c71581a1fa4ae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.425832 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 054d78c606d64944b32618b93615befd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.425838 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 2dd3e77e26c74ac392274844bc3df181
I20251027 21:08:12.426050 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3bd5f101f58440db7097a11ad1313bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.429718 11400 raft_consensus.cc:493] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.429778 9303 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet f2710f265ae9485dafec389168328067 (table default.Table_eb44e5cb7bc24597a807d81c1207a162 [id=27d1cff8fc7940c9abb741c8c7d87e24]). Expected version 2 got 0
I20251027 21:08:12.429829 11400 raft_consensus.cc:515] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.429967 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 2dd3e77e26c74ac392274844bc3df181 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:12.429980 9303 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet e6b58b1f4f63461aa96e416cae879be8 (table default.Table_f3c2de263e8849caaef87b570fc69459 [id=ef50ef2238804c98ad19c5b1c0b881c4]). Expected version 1 got 0
I20251027 21:08:12.430131 9303 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 29693cf06bc24930870dc60457126519 (table default.Table_c09cc86fc2a64450af56fc584302da69 [id=d46947a29ddf4d94b5a9da22d9e5383e]). Expected version 3 got 0
I20251027 21:08:12.431558 10032 raft_consensus.cc:359] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.431774 10032 raft_consensus.cc:385] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.431818 10032 raft_consensus.cc:740] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.432391 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet cf7b5a0b6cfd460da98558baf887093e
I20251027 21:08:12.433074 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 04c1f18b611d45c9ae2c7b895240f545
I20251027 21:08:12.434882 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet cf7b5a0b6cfd460da98558baf887093e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:12.434976 9894 tablet_service.cc:1552] Processing DeleteTablet for tablet 04c1f18b611d45c9ae2c7b895240f545 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:12.435132 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 04c1f18b611d45c9ae2c7b895240f545 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:12.435340 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet cf7b5a0b6cfd460da98558baf887093e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:12.435741 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 1ba47e34f5464b47bc32b2776f37d97d
W20251027 21:08:12.435786 9860 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.65:44531, user_credentials={real_user=slave}} blocked reactor thread for 401368us
I20251027 21:08:12.435932 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_2ef7853cbd8b4f59a59cec83c91857c2" } new_table_name: "default.Table_644ca5e3c3a844e48a092c5831243e41" modify_external_catalogs: true
W20251027 21:08:12.435979 9860 leader_election.cc:343] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.436070 9860 leader_election.cc:304] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.437114 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ba47e34f5464b47bc32b2776f37d97d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:12.437260 9895 tablet_service.cc:1505] Processing CreateTablet for tablet d3bd5f101f58440db7097a11ad1313bb (DEFAULT_TABLE table=default.Table_09274fe9916e428a82701f8bd406e392 [id=470d302c6ca341f198c71581a1fa4ae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.437649 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3bd5f101f58440db7097a11ad1313bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.437649 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_406cb517a4d34f0c921987c19227668b" } new_table_name: "default.Table_c47ddf1ee1b944d28fb5a3b4db111de5" modify_external_catalogs: true
I20251027 21:08:12.437881 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0b4e42c5dcdb46209d2970fb037bfba8" } new_table_name: "default.Table_7d1fa92426de486bab2d77ea01ec1957" modify_external_catalogs: true
I20251027 21:08:12.438004 11498 raft_consensus.cc:2749] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.438378 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 9065a17bc7994cfa8acea98909c6aac7 (DEFAULT_TABLE table=default.Table_7975dedb2ea34a1aa8d4bd689438585c [id=13c8ae744e944af989ac225220636234]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:12.438441 9860 leader_election.cc:343] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.438540 9860 leader_election.cc:304] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.438737 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9065a17bc7994cfa8acea98909c6aac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.442996 11498 raft_consensus.cc:2749] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.443509 9903 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ba47e34f5464b47bc32b2776f37d97d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:12.443523 10043 rpcz_store.cc:276] 1027 21:08:08.615527 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.615573 (+ 46us) service_pool.cc:225] Handling call
1027 21:08:12.306560 (+3690987us) ts_tablet_manager.cc:1151] Deleting tablet 9861a0efcba64141aae450663c5d83de
1027 21:08:12.306571 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.374877 (+ 68306us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.404567 (+ 29690us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.406089 (+ 1522us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3690874}
I20251027 21:08:12.444154 10043 tablet_replica.cc:333] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:12.444291 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 2dd3e77e26c74ac392274844bc3df181 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:08:12.445124 9860 leader_election.cc:343] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.445286 9860 leader_election.cc:304] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.445580 10029 tablet_bootstrap.cc:492] T 823620e5486e4cf982fdf9105e8cc4f1 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:12.445771 11498 raft_consensus.cc:2749] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.447341 10029 tablet_bootstrap.cc:654] T 823620e5486e4cf982fdf9105e8cc4f1 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.447786 9738 tablet_service.cc:1505] Processing CreateTablet for tablet c734e20b87ce4c62a2c1b8851783c8b1 (DEFAULT_TABLE table=default.Table_7cf03d793a0548ef80ca9ce245e0fc1f [id=d7fb62804ebe44bcacccc1a18d936653]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.448316 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet a4d26f73b8534f08bcaa386d4de75c01 (table default.Table_a6cf2819ed9241cea09738477423c84a [id=f5ef094809e24355a7e021408e2ddd20]) successfully deleted
I20251027 21:08:12.450048 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c734e20b87ce4c62a2c1b8851783c8b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.450155 9919 raft_consensus.cc:2468] T 986bf148b3e9482fa2fe53bbac54750d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:12.450702 11400 leader_election.cc:290] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.451440 10032 consensus_queue.cc:260] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.451946 10032 ts_tablet_manager.cc:1434] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.211s user 0.001s sys 0.001s
I20251027 21:08:12.452237 10032 tablet_bootstrap.cc:492] T fb7fdc91156e41de80872b5c941c5d95 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:12.453733 9903 tablet_service.cc:1505] Processing CreateTablet for tablet 9065a17bc7994cfa8acea98909c6aac7 (DEFAULT_TABLE table=default.Table_7975dedb2ea34a1aa8d4bd689438585c [id=13c8ae744e944af989ac225220636234]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:12.446131 9860 leader_election.cc:343] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.454523 9860 leader_election.cc:304] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.456193 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet f775a7e42bdc4910bdb604b26a28a7f4 (table default.Table_7b970942803b42f29feb36d6d845167a [id=20a729dd21c440a882083323ddb2561e]) successfully deleted
I20251027 21:08:12.456561 10040 tablet_replica.cc:333] T a033551a591146039b017b4193b221ca P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:12.456825 10040 raft_consensus.cc:2243] T a033551a591146039b017b4193b221ca P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:12.457031 10040 raft_consensus.cc:2272] T a033551a591146039b017b4193b221ca P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.457643 10040 ts_tablet_manager.cc:1916] T a033551a591146039b017b4193b221ca P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.459031 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 03986aea266e45d49a969ed5cad865d6 (DEFAULT_TABLE table=default.Table_db3806bdc5c2498294b20102a1767b93 [id=2fe4f21376ce4594a6c51d39e373cbfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.459436 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03986aea266e45d49a969ed5cad865d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.459471 11400 raft_consensus.cc:2749] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.465142 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bc5fbdbbea0406a961e1a4df51b8762" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:12.466214 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 5e3b3d9d43e643f6828cb6078dd8650e (DEFAULT_TABLE table=default.Table_b7e7e18292df4076a52ea8b99c94f901 [id=e2ad8c077bc84fdeb54e41fa643d519e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.466338 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 2dd3e77e26c74ac392274844bc3df181 (DEFAULT_TABLE table=default.Table_14d4a985bc664c44b50d535a4757aa00 [id=197976254c434ad0ac29ffcc7ea7980f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.466579 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e3b3d9d43e643f6828cb6078dd8650e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.466720 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dd3e77e26c74ac392274844bc3df181. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.470391 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 8baffc084fdc448189d20d419b08d194 (DEFAULT_TABLE table=default.Table_5e855e21102b4ee5b98896cf48eacde6 [id=0e4846acfbbc4e26a853ddd837598349]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.470711 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8baffc084fdc448189d20d419b08d194. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.475138 9607 tablet_service.cc:1505] Processing CreateTablet for tablet fc7c554d37b1419d899cff37db371221 (DEFAULT_TABLE table=default.Table_548bc7accdaa4dd8bfb6373d3d15667d [id=89cc203955a9451aae42cc8477eec02b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.475374 10031 tablet_bootstrap.cc:492] T 631ff72613734913818a3cad24f20dbc P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:12.475354 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 1e4120d22d184fa588fd48642d93dd27 (DEFAULT_TABLE table=default.Table_7c14877bc32245418a930cf51449c007 [id=820d0e68ebe3424cbb3507306d805429]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.475476 10031 ts_tablet_manager.cc:1403] T 631ff72613734913818a3cad24f20dbc P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.105s user 0.001s sys 0.002s
I20251027 21:08:12.475497 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc7c554d37b1419d899cff37db371221. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.475817 9611 tablet_service.cc:1505] Processing CreateTablet for tablet c34e487f57bd41bca440b0ad0179a075 (DEFAULT_TABLE table=default.Table_3c794af372fc4ed8ab15aecdf30067f9 [id=859536014530491888672609efaa6327]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.476126 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34e487f57bd41bca440b0ad0179a075. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.476159 10031 raft_consensus.cc:359] T 631ff72613734913818a3cad24f20dbc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:12.476297 10031 raft_consensus.cc:385] T 631ff72613734913818a3cad24f20dbc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.476337 10031 raft_consensus.cc:740] T 631ff72613734913818a3cad24f20dbc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.476504 10031 consensus_queue.cc:260] T 631ff72613734913818a3cad24f20dbc P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:12.476969 10031 ts_tablet_manager.cc:1434] T 631ff72613734913818a3cad24f20dbc P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:12.477238 9736 tablet_service.cc:1505] Processing CreateTablet for tablet a9e16012ffb74ebe84288399f45f3799 (DEFAULT_TABLE table=default.Table_f40312591b4147acbe2ba36ff49b01f4 [id=3d1dc1fe20a04c8494617a0c7af078d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.477665 10031 tablet_bootstrap.cc:492] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:12.478291 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet 283b72f626644ac2b9b61186387b3839 with delete_type TABLET_DATA_DELETED (replaced by tablet f5bf7b066583425593b89b1d3eb2b82d) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:12.478631 9598 tablet_service.cc:1505] Processing CreateTablet for tablet 054d78c606d64944b32618b93615befd (DEFAULT_TABLE table=default.Table_877147b58eff438190609ce0142c08f5 [id=05da7a8eb8ea441d90763f0bf6abd69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.478763 11514 raft_consensus.cc:493] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.478870 11514 raft_consensus.cc:515] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:12.479094 9599 tablet_service.cc:1505] Processing CreateTablet for tablet d3bd5f101f58440db7097a11ad1313bb (DEFAULT_TABLE table=default.Table_09274fe9916e428a82701f8bd406e392 [id=470d302c6ca341f198c71581a1fa4ae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.479362 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9e16012ffb74ebe84288399f45f3799. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.479476 11514 leader_election.cc:290] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:12.480260 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96a680e8f16343b1908829a67e960b60" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:12.480479 9630 raft_consensus.cc:2393] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77b0513f17c0472088901a1ce8b08218 in current term 1: Already voted for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in this term.
I20251027 21:08:12.478303 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e4120d22d184fa588fd48642d93dd27. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.485388 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3bd5f101f58440db7097a11ad1313bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.486974 10031 tablet_bootstrap.cc:654] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.490164 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet f543e8bcf70c4acab9880ee01af85562
I20251027 21:08:12.490540 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 054d78c606d64944b32618b93615befd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.444613 10043 raft_consensus.cc:2243] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:12.490924 10043 pending_rounds.cc:70] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14: Trying to abort 1 pending ops.
I20251027 21:08:12.490861 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_384092146e0e472da1ce74725287cb18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:12.490973 10043 pending_rounds.cc:77] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510699913449472 op_type: NO_OP noop_request { }
I20251027 21:08:12.491050 10043 raft_consensus.cc:2889] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:12.491124 10043 raft_consensus.cc:2272] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:12.491340 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_384092146e0e472da1ce74725287cb18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:12.492564 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet eeb667b3b10746dbad351c4c123ffd97
I20251027 21:08:12.492882 9310 catalog_manager.cc:6719] ReplaceTablet: tablet eeb667b3b10746dbad351c4c123ffd97 deleted and replaced by tablet 56cea2d6c38b447ba070e777d0525810
I20251027 21:08:12.493108 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6d0fed3fe4374cd3b70258dd15d9b06e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:12.493531 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d0fed3fe4374cd3b70258dd15d9b06e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:12.494848 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2350c01a35bb4fbba38029e0a0c72d0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:12.495222 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2350c01a35bb4fbba38029e0a0c72d0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:12.496402 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f2e5f26bedf64d45b5d74662abb032e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:12.496857 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2e5f26bedf64d45b5d74662abb032e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:12.499104 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet f543e8bcf70c4acab9880ee01af85562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:12.499320 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet eeb667b3b10746dbad351c4c123ffd97 with delete_type TABLET_DATA_DELETED (replaced by tablet 56cea2d6c38b447ba070e777d0525810) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:12.500496 11419 raft_consensus.cc:493] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.500610 11419 raft_consensus.cc:515] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.501109 11419 leader_election.cc:290] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.501226 11419 raft_consensus.cc:493] T 842c1a1298234d9ea2719bab29c54ae3 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.501302 11419 raft_consensus.cc:515] T 842c1a1298234d9ea2719bab29c54ae3 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.501811 11419 leader_election.cc:290] T 842c1a1298234d9ea2719bab29c54ae3 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.501924 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet eeb667b3b10746dbad351c4c123ffd97 with delete_type TABLET_DATA_DELETED (replaced by tablet 56cea2d6c38b447ba070e777d0525810) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:12.503692 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "842c1a1298234d9ea2719bab29c54ae3" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:12.503839 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd9e6656d8df43afbb7f70706bc14585" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:12.504714 9894 tablet_service.cc:1505] Processing CreateTablet for tablet 8baffc084fdc448189d20d419b08d194 (DEFAULT_TABLE table=default.Table_5e855e21102b4ee5b98896cf48eacde6 [id=0e4846acfbbc4e26a853ddd837598349]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.505908 9899 tablet_service.cc:1505] Processing CreateTablet for tablet c734e20b87ce4c62a2c1b8851783c8b1 (DEFAULT_TABLE table=default.Table_7cf03d793a0548ef80ca9ce245e0fc1f [id=d7fb62804ebe44bcacccc1a18d936653]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:12.506006 9566 leader_election.cc:343] T 842c1a1298234d9ea2719bab29c54ae3 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:12.506243 9566 leader_election.cc:343] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.506729 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet f543e8bcf70c4acab9880ee01af85562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:12.475104 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 2dd3e77e26c74ac392274844bc3df181 (DEFAULT_TABLE table=default.Table_14d4a985bc664c44b50d535a4757aa00 [id=197976254c434ad0ac29ffcc7ea7980f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.508711 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9065a17bc7994cfa8acea98909c6aac7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:12.510071 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04a50ad9927e441faa3c57579c9cd215 consensus state has no leader
W20251027 21:08:12.510259 9711 leader_election.cc:343] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 96a680e8f16343b1908829a67e960b60
I20251027 21:08:12.510354 9711 leader_election.cc:304] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:12.510452 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_1d34cea4395c4a96b561f68c263f70cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:12.510564 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8baffc084fdc448189d20d419b08d194. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.510720 11514 raft_consensus.cc:3060] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:12.510805 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c734e20b87ce4c62a2c1b8851783c8b1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:12.510962 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d34cea4395c4a96b561f68c263f70cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:12.511067 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dd3e77e26c74ac392274844bc3df181. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.512019 9273 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu
/tmp/dist-test-taskcL_Mgg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.14.125:46313
--webserver_interface=127.9.14.125
--webserver_port=46529
--builtin_ntp_servers=127.9.14.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.14.126:34057,127.9.14.125:46313,127.9.14.124:45071
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251027 21:08:12.512075 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6a5071cb8c5a4ef59286b5970c68845b consensus state has no leader
I20251027 21:08:12.512351 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_81e4dd46fa6945bc956cc69e26dabcf1" } new_table_name: "default.Table_1fde66d660354a5a92f81b7bc1f0d511" modify_external_catalogs: true
I20251027 21:08:12.513435 11514 raft_consensus.cc:2749] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.514406 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d8da07f0725347338012d992737c5d04" } new_table_name: "default.Table_cc8a1775cbcb4e0eaeb8c90bf9d61c04" modify_external_catalogs: true
I20251027 21:08:12.517884 9742 tablet_service.cc:1505] Processing CreateTablet for tablet d5c1c9eafeb841d1834d87bccedd62c3 (DEFAULT_TABLE table=default.Table_bd014e37535443cda5805af5baebc27d [id=97ca5ce4c6724b1ebc2c90f667ffcd71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.518335 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5c1c9eafeb841d1834d87bccedd62c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.519979 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c842d5beeb2a4a43b96a61a2cf97a8a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:12.520426 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c842d5beeb2a4a43b96a61a2cf97a8a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:12.521067 9891 tablet_service.cc:1505] Processing CreateTablet for tablet d5c1c9eafeb841d1834d87bccedd62c3 (DEFAULT_TABLE table=default.Table_bd014e37535443cda5805af5baebc27d [id=97ca5ce4c6724b1ebc2c90f667ffcd71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.521301 10040 ts_tablet_manager.cc:1929] T a033551a591146039b017b4193b221ca P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:12.521414 10040 log.cc:1199] T a033551a591146039b017b4193b221ca P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/a033551a591146039b017b4193b221ca
I20251027 21:08:12.521466 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5c1c9eafeb841d1834d87bccedd62c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.521488 9306 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_ade21e4db36445fc8f91817f46606b51 [id=5da6bd4463e24c03accd9985f72b6ebb] alter complete (version 4)
I20251027 21:08:12.521714 10040 ts_tablet_manager.cc:1950] T a033551a591146039b017b4193b221ca P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:12.524472 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c34c8f1433b40d491ae05e7c6e60ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:12.509485 9904 tablet_service.cc:1505] Processing CreateTablet for tablet a9e16012ffb74ebe84288399f45f3799 (DEFAULT_TABLE table=default.Table_f40312591b4147acbe2ba36ff49b01f4 [id=3d1dc1fe20a04c8494617a0c7af078d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.527096 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9e16012ffb74ebe84288399f45f3799. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.527951 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 26303640f7de40e8a45c4e6f55f0e1c6 (DEFAULT_TABLE table=default.Table_243113a951ad4de7a77f46d9c89f5ce4 [id=44bc5aee9cbc4acdaefcab97bd13c097]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.524298 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet aabaa34787cf4d179822269b7f323314 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:08:12.528645 9566 consensus_peers.cc:597] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 980eb6755aee4bcfb2ea44d84009021f. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:12.529446 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26303640f7de40e8a45c4e6f55f0e1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.530886 10032 tablet_bootstrap.cc:654] T fb7fdc91156e41de80872b5c941c5d95 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.533106 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet 2dd3e77e26c74ac392274844bc3df181 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:12.533440 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet cf7b5a0b6cfd460da98558baf887093e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:12.533782 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet 04c1f18b611d45c9ae2c7b895240f545 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:12.534099 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ba47e34f5464b47bc32b2776f37d97d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:12.534600 9613 tablet_service.cc:1505] Processing CreateTablet for tablet 9065a17bc7994cfa8acea98909c6aac7 (DEFAULT_TABLE table=default.Table_7975dedb2ea34a1aa8d4bd689438585c [id=13c8ae744e944af989ac225220636234]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.534963 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9065a17bc7994cfa8acea98909c6aac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.537331 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 631ff72613734913818a3cad24f20dbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:12.539969 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ae222307a374a91916f560a0fbe8d8f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:12.540238 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cccd253ae2a46328dc9f767b8e9ee97" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:12.540480 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "133e474f9133487f89c44dbfa6266ca7" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:12.541411 11400 raft_consensus.cc:493] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.541538 11400 raft_consensus.cc:515] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:12.542534 9859 leader_election.cc:343] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:12.542764 9859 leader_election.cc:343] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:12.542959 9859 leader_election.cc:343] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.545750 10041 ts_tablet_manager.cc:1929] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:12.545851 10041 log.cc:1199] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/e10ae34eaaf3476594813de8e96c7e77
I20251027 21:08:12.546330 10041 ts_tablet_manager.cc:1950] T e10ae34eaaf3476594813de8e96c7e77 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:12.546599 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1136) took 2950 ms. Trace:
W20251027 21:08:12.546725 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 958dafe1e6844f19902524e8fa27d69a consensus state has no leader
I20251027 21:08:12.546690 10041 rpcz_store.cc:276] 1027 21:08:09.595788 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.596464 (+ 676us) service_pool.cc:225] Handling call
1027 21:08:12.352102 (+2755638us) ts_tablet_manager.cc:1151] Deleting tablet e10ae34eaaf3476594813de8e96c7e77
1027 21:08:12.352113 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.456047 (+103934us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.545664 (+ 89617us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.546585 (+ 921us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2755452}
I20251027 21:08:12.546828 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 1cbec3625e5f47448e944619db713c87 (DEFAULT_TABLE table=default.Table_e22a34d2f5e848fcbe891c4d0325fad4 [id=0885815acfc3445a9b9fa50359d697a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.547088 10041 tablet_replica.cc:333] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:12.547199 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cbec3625e5f47448e944619db713c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.547255 10041 raft_consensus.cc:2243] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:12.547371 10041 raft_consensus.cc:2272] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.547639 9613 tablet_service.cc:1505] Processing CreateTablet for tablet c734e20b87ce4c62a2c1b8851783c8b1 (DEFAULT_TABLE table=default.Table_7cf03d793a0548ef80ca9ce245e0fc1f [id=d7fb62804ebe44bcacccc1a18d936653]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.547830 10041 ts_tablet_manager.cc:1916] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.547981 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c734e20b87ce4c62a2c1b8851783c8b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.548038 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet e10ae34eaaf3476594813de8e96c7e77 (table default.Table_107ad52a34d44600aa08dbe6983ab081 [id=bbc85bf038474c93ae8312bdca3a7029]) successfully deleted
I20251027 21:08:12.545852 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 26303640f7de40e8a45c4e6f55f0e1c6 (DEFAULT_TABLE table=default.Table_243113a951ad4de7a77f46d9c89f5ce4 [id=44bc5aee9cbc4acdaefcab97bd13c097]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.552055 9773 raft_consensus.cc:3060] T 29693cf06bc24930870dc60457126519 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:12.552268 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26303640f7de40e8a45c4e6f55f0e1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.551576 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 1cbec3625e5f47448e944619db713c87 (DEFAULT_TABLE table=default.Table_e22a34d2f5e848fcbe891c4d0325fad4 [id=0885815acfc3445a9b9fa50359d697a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.559262 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cbec3625e5f47448e944619db713c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.563158 9736 tablet_service.cc:1505] Processing CreateTablet for tablet dce65917bce54a41a9c78cfc8cf46f8e (DEFAULT_TABLE table=default.Table_783ec376f9f34eefadffe56b522df2eb [id=37049176d78c48758d71ad7ec579cd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.563508 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dce65917bce54a41a9c78cfc8cf46f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.566825 10041 ts_tablet_manager.cc:1929] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:12.568862 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet a033551a591146039b017b4193b221ca (table default.Table_7a01697cdee04a929068e3297b598128 [id=84d2047a211e479e89130346cedd9fdf]) successfully deleted
W20251027 21:08:12.569873 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 138a5d18566240858b5c92c3c03b8087 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 138a5d18566240858b5c92c3c03b8087 already in progress: creating tablet
I20251027 21:08:12.571318 10040 tablet_replica.cc:333] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:12.574911 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 2dd3e77e26c74ac392274844bc3df181 (DEFAULT_TABLE table=default.Table_14d4a985bc664c44b50d535a4757aa00 [id=197976254c434ad0ac29ffcc7ea7980f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.572467 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 1df8132e9e38499fa711072b4b6966a2 (DEFAULT_TABLE table=default.Table_258950ebb36249149222ca71cb3278e4 [id=64e71db8c212429ba4519e5f4e3dbe62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.575282 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dd3e77e26c74ac392274844bc3df181. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.575282 10040 raft_consensus.cc:2243] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:12.575404 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df8132e9e38499fa711072b4b6966a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.575439 10040 raft_consensus.cc:2272] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.575961 10040 ts_tablet_manager.cc:1916] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.580229 9738 tablet_service.cc:1505] Processing CreateTablet for tablet cc84242b57c042ee9dd44231f9696f1e (DEFAULT_TABLE table=default.Table_38420733ed6d43799874a162313d5828 [id=f943cdf2b6824434beed734129127725]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.580612 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc84242b57c042ee9dd44231f9696f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.581451 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet eeb667b3b10746dbad351c4c123ffd97 with delete_type TABLET_DATA_DELETED (replaced by tablet 56cea2d6c38b447ba070e777d0525810) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:12.581840 9598 tablet_service.cc:1505] Processing CreateTablet for tablet a9e16012ffb74ebe84288399f45f3799 (DEFAULT_TABLE table=default.Table_f40312591b4147acbe2ba36ff49b01f4 [id=3d1dc1fe20a04c8494617a0c7af078d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.582180 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9e16012ffb74ebe84288399f45f3799. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.582650 10029 tablet_bootstrap.cc:492] T 823620e5486e4cf982fdf9105e8cc4f1 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:12.582764 10029 ts_tablet_manager.cc:1403] T 823620e5486e4cf982fdf9105e8cc4f1 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.137s user 0.003s sys 0.000s
I20251027 21:08:12.583508 10043 ts_tablet_manager.cc:1916] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.583438 10029 raft_consensus.cc:359] T 823620e5486e4cf982fdf9105e8cc4f1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.583581 10029 raft_consensus.cc:385] T 823620e5486e4cf982fdf9105e8cc4f1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.583622 10029 raft_consensus.cc:740] T 823620e5486e4cf982fdf9105e8cc4f1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.583854 10029 consensus_queue.cc:260] T 823620e5486e4cf982fdf9105e8cc4f1 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.585145 9607 tablet_service.cc:1505] Processing CreateTablet for tablet d5c1c9eafeb841d1834d87bccedd62c3 (DEFAULT_TABLE table=default.Table_bd014e37535443cda5805af5baebc27d [id=97ca5ce4c6724b1ebc2c90f667ffcd71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.586912 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet 842c1a1298234d9ea2719bab29c54ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:12.587270 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5c1c9eafeb841d1834d87bccedd62c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.589653 9615 tablet_service.cc:1505] Processing CreateTablet for tablet dce65917bce54a41a9c78cfc8cf46f8e (DEFAULT_TABLE table=default.Table_783ec376f9f34eefadffe56b522df2eb [id=37049176d78c48758d71ad7ec579cd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.590896 9604 tablet_service.cc:1505] Processing CreateTablet for tablet 1cbec3625e5f47448e944619db713c87 (DEFAULT_TABLE table=default.Table_e22a34d2f5e848fcbe891c4d0325fad4 [id=0885815acfc3445a9b9fa50359d697a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.591536 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dce65917bce54a41a9c78cfc8cf46f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.595369 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cbec3625e5f47448e944619db713c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.597872 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 1df8132e9e38499fa711072b4b6966a2 (DEFAULT_TABLE table=default.Table_258950ebb36249149222ca71cb3278e4 [id=64e71db8c212429ba4519e5f4e3dbe62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.598196 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df8132e9e38499fa711072b4b6966a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.598511 10029 ts_tablet_manager.cc:1434] T 823620e5486e4cf982fdf9105e8cc4f1 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20251027 21:08:12.598764 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c15b74f53bc45e1af33b043f2cf225f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:12.601910 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 9861a0efcba64141aae450663c5d83de (table default.Table_a8906ade8f93405bb76d1ca9b7a0afa6 [id=a0c9cfaafd6c46dfb9a999f25f8977fa]) successfully deleted
I20251027 21:08:12.602212 10029 tablet_bootstrap.cc:492] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:12.603912 10029 tablet_bootstrap.cc:654] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.613724 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet f543e8bcf70c4acab9880ee01af85562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:12.629554 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 8baffc084fdc448189d20d419b08d194 (DEFAULT_TABLE table=default.Table_5e855e21102b4ee5b98896cf48eacde6 [id=0e4846acfbbc4e26a853ddd837598349]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.630033 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8baffc084fdc448189d20d419b08d194. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.631688 9740 tablet_service.cc:1505] Processing CreateTablet for tablet 736c27b1adf040208e356137c40f9af4 (DEFAULT_TABLE table=default.Table_1100e497c4ed4782824ba1c9e058fdbc [id=ea1d9535346141898f7cb0c9e23e5732]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.631843 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 712e1763fe064fd49ee210d991dc56b3 (DEFAULT_TABLE table=default.Table_aef78f959ff04f84a9f5d8d736c7f596 [id=0215aa295a0e493ca6f193c5872cbc5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.632100 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 736c27b1adf040208e356137c40f9af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.635742 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 712e1763fe064fd49ee210d991dc56b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.639824 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_916903f431254b8c89108557bc5f37bf" } modify_external_catalogs: true
I20251027 21:08:12.639990 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_916903f431254b8c89108557bc5f37bf" } modify_external_catalogs: true
I20251027 21:08:12.641037 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_645e7862b8ef4097aa7152d5984b7fb7" } modify_external_catalogs: true
I20251027 21:08:12.641171 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_645e7862b8ef4097aa7152d5984b7fb7" } modify_external_catalogs: true
I20251027 21:08:12.641796 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_66880e44b20b4f4cb43d1b827409bbd8" } modify_external_catalogs: true
I20251027 21:08:12.641916 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_66880e44b20b4f4cb43d1b827409bbd8" } modify_external_catalogs: true
I20251027 21:08:12.642303 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_20fbc364a16e49e8b9b5a87e7c7a4065"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:12.642745 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20fbc364a16e49e8b9b5a87e7c7a4065 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:12.566883 11400 leader_election.cc:290] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.574033 10032 tablet_bootstrap.cc:492] T fb7fdc91156e41de80872b5c941c5d95 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:12.646927 10032 ts_tablet_manager.cc:1403] T fb7fdc91156e41de80872b5c941c5d95 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.195s user 0.003s sys 0.003s
I20251027 21:08:12.646906 9307 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 29693cf06bc24930870dc60457126519 (table default.Table_c09cc86fc2a64450af56fc584302da69 [id=d46947a29ddf4d94b5a9da22d9e5383e]). Expected version 3 got 0
I20251027 21:08:12.647583 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7aab2b63c44043aa9aea339ec0b5763e" } modify_external_catalogs: true
I20251027 21:08:12.647708 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7aab2b63c44043aa9aea339ec0b5763e" } modify_external_catalogs: true
I20251027 21:08:12.647722 10032 raft_consensus.cc:359] T fb7fdc91156e41de80872b5c941c5d95 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.647861 10032 raft_consensus.cc:385] T fb7fdc91156e41de80872b5c941c5d95 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.647902 10032 raft_consensus.cc:740] T fb7fdc91156e41de80872b5c941c5d95 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.648062 10032 consensus_queue.cc:260] T fb7fdc91156e41de80872b5c941c5d95 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.581947 9905 tablet_service.cc:1505] Processing CreateTablet for tablet dce65917bce54a41a9c78cfc8cf46f8e (DEFAULT_TABLE table=default.Table_783ec376f9f34eefadffe56b522df2eb [id=37049176d78c48758d71ad7ec579cd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.648464 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dce65917bce54a41a9c78cfc8cf46f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.590269 11517 raft_consensus.cc:493] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.654240 11517 raft_consensus.cc:515] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.654917 11517 leader_election.cc:290] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.655325 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 35ae35f748f547d689f8f341a4ba149f (DEFAULT_TABLE table=default.Table_c3120318fe0f4feda2d71e836f0e02e6 [id=312861325fb54c3aa87964476548ec7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.655555 10032 ts_tablet_manager.cc:1434] T fb7fdc91156e41de80872b5c941c5d95 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251027 21:08:12.655812 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ae35f748f547d689f8f341a4ba149f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.655884 10032 tablet_bootstrap.cc:492] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:12.636394 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 26303640f7de40e8a45c4e6f55f0e1c6 (DEFAULT_TABLE table=default.Table_243113a951ad4de7a77f46d9c89f5ce4 [id=44bc5aee9cbc4acdaefcab97bd13c097]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.648746 11512 raft_consensus.cc:493] T 9ed4f9c61ad54e1292281b30bcd19533 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.657514 11512 raft_consensus.cc:515] T 9ed4f9c61ad54e1292281b30bcd19533 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.657792 10032 tablet_bootstrap.cc:654] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.658000 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9b8f76c33444bcb9b488af65e4d3e6b" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:12.658089 11512 leader_election.cc:290] T 9ed4f9c61ad54e1292281b30bcd19533 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.590718 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 1df8132e9e38499fa711072b4b6966a2 (DEFAULT_TABLE table=default.Table_258950ebb36249149222ca71cb3278e4 [id=64e71db8c212429ba4519e5f4e3dbe62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.606096 9911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd9e6656d8df43afbb7f70706bc14585" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:12.658262 9911 raft_consensus.cc:2468] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:12.658264 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ae222307a374a91916f560a0fbe8d8f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:12.658361 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "823620e5486e4cf982fdf9105e8cc4f1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:12.658504 9634 raft_consensus.cc:2468] T 823620e5486e4cf982fdf9105e8cc4f1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 0.
I20251027 21:08:12.658740 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df8132e9e38499fa711072b4b6966a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.658872 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26303640f7de40e8a45c4e6f55f0e1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.659149 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cccd253ae2a46328dc9f767b8e9ee97" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:12.659682 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "133e474f9133487f89c44dbfa6266ca7" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:12.660262 11517 raft_consensus.cc:493] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.660367 11517 raft_consensus.cc:515] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.660877 11517 leader_election.cc:290] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.661936 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c15b74f53bc45e1af33b043f2cf225f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:12.615320 9859 leader_election.cc:343] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:12.662281 9859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 46995.5us
I20251027 21:08:12.662655 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c27c699cc2744f40811f7b9395ca76be" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:12.662956 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c27c699cc2744f40811f7b9395ca76be" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:12.662967 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61ba8d7df6fd45abb6f5231ee8c4ec38" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:12.628446 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 35ae35f748f547d689f8f341a4ba149f (DEFAULT_TABLE table=default.Table_c3120318fe0f4feda2d71e836f0e02e6 [id=312861325fb54c3aa87964476548ec7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.664428 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ae35f748f547d689f8f341a4ba149f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.667536 11512 raft_consensus.cc:493] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.667537 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61ba8d7df6fd45abb6f5231ee8c4ec38" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:12.667685 11512 raft_consensus.cc:515] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.668169 11512 leader_election.cc:290] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.668327 11425 raft_consensus.cc:493] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.668440 11425 raft_consensus.cc:515] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.668933 11425 leader_election.cc:290] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.669164 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaaf70c950044387a63820079db3c829" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:12.669961 9860 leader_election.cc:343] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.670112 9860 leader_election.cc:304] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:12.670374 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1050ac06e1e74d54ba4014b49cc84467 consensus state has no leader
I20251027 21:08:12.670439 11425 raft_consensus.cc:2749] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:12.670823 9860 leader_election.cc:343] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.671027 9860 leader_election.cc:304] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.671315 11425 raft_consensus.cc:2749] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.635661 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 736c27b1adf040208e356137c40f9af4 (DEFAULT_TABLE table=default.Table_1100e497c4ed4782824ba1c9e058fdbc [id=ea1d9535346141898f7cb0c9e23e5732]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.671808 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 736c27b1adf040208e356137c40f9af4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:12.672103 9566 leader_election.cc:343] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.673599 9752 tablet_service.cc:1505] Processing CreateTablet for tablet bd2d5c148e5f49ff94d79de3953444a6 (DEFAULT_TABLE table=default.Table_109b65ba33734a5f8ffb302c8fc8caaa [id=bee382f9a8cf4a68a92eb57e2d5bdbf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.673954 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd2d5c148e5f49ff94d79de3953444a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.635926 9895 tablet_service.cc:1505] Processing CreateTablet for tablet cc84242b57c042ee9dd44231f9696f1e (DEFAULT_TABLE table=default.Table_38420733ed6d43799874a162313d5828 [id=f943cdf2b6824434beed734129127725]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.674419 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc84242b57c042ee9dd44231f9696f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.636180 9890 tablet_service.cc:1505] Processing CreateTablet for tablet 712e1763fe064fd49ee210d991dc56b3 (DEFAULT_TABLE table=default.Table_aef78f959ff04f84a9f5d8d736c7f596 [id=0215aa295a0e493ca6f193c5872cbc5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.675174 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 712e1763fe064fd49ee210d991dc56b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.679476 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 8909318b2d3344a6a89a84dedff69a42 (DEFAULT_TABLE table=default.Table_8ddc08df525c4d329846f82ba45beeca [id=de0a1d479af3438fb03dba6fef61ff45]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.679811 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8909318b2d3344a6a89a84dedff69a42. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:12.680975 9859 leader_election.cc:343] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:12.681622 9859 leader_election.cc:343] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.650573 9568 leader_election.cc:304] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.686395 11512 raft_consensus.cc:2804] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:12.686514 11512 raft_consensus.cc:697] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:12.686564 11521 tablet.cc:1722] T 29693cf06bc24930870dc60457126519 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251027 21:08:12.686748 11512 consensus_queue.cc:237] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.686771 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59300e16b3e148a08a217fc24d1e572f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:12.687354 9859 leader_election.cc:343] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.689270 9755 tablet_service.cc:1505] Processing CreateTablet for tablet e90e2e01b9e94b0a97fc6aaf1153dd4e (DEFAULT_TABLE table=default.Table_4cca34ef6c5f47e285603a77b24501bb [id=289a47b40eaa4b81b2501efacf6eb1af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.689580 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e90e2e01b9e94b0a97fc6aaf1153dd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.694455 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ed4f9c61ad54e1292281b30bcd19533" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:12.695040 9566 leader_election.cc:343] T 9ed4f9c61ad54e1292281b30bcd19533 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.695335 9891 tablet_service.cc:1505] Processing CreateTablet for tablet e90e2e01b9e94b0a97fc6aaf1153dd4e (DEFAULT_TABLE table=default.Table_4cca34ef6c5f47e285603a77b24501bb [id=289a47b40eaa4b81b2501efacf6eb1af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.695719 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e90e2e01b9e94b0a97fc6aaf1153dd4e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:12.696429 9568 leader_election.cc:343] T 842c1a1298234d9ea2719bab29c54ae3 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 842c1a1298234d9ea2719bab29c54ae3
I20251027 21:08:12.696792 9568 leader_election.cc:304] T 842c1a1298234d9ea2719bab29c54ae3 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.697602 9773 raft_consensus.cc:3060] T f2710f265ae9485dafec389168328067 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:12.697669 10040 ts_tablet_manager.cc:1929] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:12.697764 11411 raft_consensus.cc:2749] T 842c1a1298234d9ea2719bab29c54ae3 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.697798 10040 log.cc:1199] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/96a680e8f16343b1908829a67e960b60
I20251027 21:08:12.698091 10040 ts_tablet_manager.cc:1950] T 96a680e8f16343b1908829a67e960b60 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:12.698397 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 8909318b2d3344a6a89a84dedff69a42 (DEFAULT_TABLE table=default.Table_8ddc08df525c4d329846f82ba45beeca [id=de0a1d479af3438fb03dba6fef61ff45]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.698778 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8909318b2d3344a6a89a84dedff69a42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.699100 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 96a680e8f16343b1908829a67e960b60 (table default.Table_a46825cc888a4a0898d6db70b02e5ab6 [id=8579d3ff46d64c75b1da213fdee9cc1b]) successfully deleted
W20251027 21:08:12.702924 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet f19b5e8e460a4c4ba7399b981c603b84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f19b5e8e460a4c4ba7399b981c603b84 already in progress: creating tablet
I20251027 21:08:12.707715 9568 leader_election.cc:304] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:12.708027 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 7057c55a820d4e379fe1d9abfa353821 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7057c55a820d4e379fe1d9abfa353821 already in progress: creating tablet
W20251027 21:08:12.708555 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 4bfa70ded2f54e3fb2ca744c723395e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4bfa70ded2f54e3fb2ca744c723395e2 already in progress: creating tablet
I20251027 21:08:12.708197 10029 tablet_bootstrap.cc:492] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
W20251027 21:08:12.708830 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet d27f9c498eca4e5f8d2bfe9c805d06b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d27f9c498eca4e5f8d2bfe9c805d06b3 already in progress: creating tablet
I20251027 21:08:12.708856 11411 raft_consensus.cc:2804] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:12.708922 11411 raft_consensus.cc:493] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:12.708987 11411 raft_consensus.cc:3060] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:12.708925 10029 ts_tablet_manager.cc:1403] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.107s user 0.004s sys 0.000s
W20251027 21:08:12.709245 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 5cf3dd1b5f4b4b4ba112df78bf0df8f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cf3dd1b5f4b4b4ba112df78bf0df8f3 already in progress: creating tablet
I20251027 21:08:12.709921 10029 raft_consensus.cc:359] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.710058 10029 raft_consensus.cc:385] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.713338 10040 tablet_replica.cc:333] T a4d26f73b8534f08bcaa386d4de75c01 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:12.713583 10040 raft_consensus.cc:2243] T a4d26f73b8534f08bcaa386d4de75c01 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:12.714126 10040 raft_consensus.cc:2272] T a4d26f73b8534f08bcaa386d4de75c01 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.711762 11411 raft_consensus.cc:515] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.714773 11411 leader_election.cc:290] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.713032 11519 tablet.cc:1722] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:12.715260 10029 raft_consensus.cc:740] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.715303 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd9e6656d8df43afbb7f70706bc14585" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:12.715562 10029 consensus_queue.cc:260] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:12.715756 9566 leader_election.cc:343] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.717419 10029 ts_tablet_manager.cc:1434] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20251027 21:08:12.718313 9860 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.65:44531, user_credentials={real_user=slave}} blocked reactor thread for 47491.3us
I20251027 21:08:12.718365 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5c1f633e896747d6a15a4998a12a1ef9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:12.718595 9860 leader_election.cc:343] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.718721 9860 leader_election.cc:304] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.718959 11517 raft_consensus.cc:2749] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.719278 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_44f9303ecfd745f99df2ffa39d8a0499" } modify_external_catalogs: true
I20251027 21:08:12.719425 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_44f9303ecfd745f99df2ffa39d8a0499" } modify_external_catalogs: true
I20251027 21:08:12.720314 10031 tablet_bootstrap.cc:492] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:12.720436 10031 ts_tablet_manager.cc:1403] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.243s user 0.003s sys 0.014s
I20251027 21:08:12.720737 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a823e860186944468dd775400fd777e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:12.721201 10031 raft_consensus.cc:359] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:12.721376 10031 raft_consensus.cc:385] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.721472 10031 raft_consensus.cc:740] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.721732 10031 consensus_queue.cc:260] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:12.722047 9308 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 29693cf06bc24930870dc60457126519 (table default.Table_c09cc86fc2a64450af56fc584302da69 [id=d46947a29ddf4d94b5a9da22d9e5383e]). Expected version 3 got 0
I20251027 21:08:12.721987 10040 ts_tablet_manager.cc:1916] T a4d26f73b8534f08bcaa386d4de75c01 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.722369 10031 ts_tablet_manager.cc:1434] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:12.722604 10031 tablet_bootstrap.cc:492] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:12.724392 10031 tablet_bootstrap.cc:654] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.724471 9308 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 67cb71ce43244d8ab9e71ead6bb315c9 (table default.Table_6327cacf4ed54d56b522291ad9594784 [id=3eb54d245ec64ee48022c541c2e49b17]). Expected version 2 got 0
W20251027 21:08:12.725103 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a823e860186944468dd775400fd777e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:12.725943 10029 tablet_bootstrap.cc:492] T 72c1dc3b9eff47ee81df821deb789307 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:12.725983 9860 leader_election.cc:304] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.726224 11517 raft_consensus.cc:2804] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:12.726291 11517 raft_consensus.cc:493] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:12.726342 11517 raft_consensus.cc:3060] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:12.726392 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c1f633e896747d6a15a4998a12a1ef9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:12.727499 10029 tablet_bootstrap.cc:654] T 72c1dc3b9eff47ee81df821deb789307 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.728662 11517 raft_consensus.cc:515] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.729199 11517 leader_election.cc:290] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:12.732100 9860 leader_election.cc:343] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.732242 9860 leader_election.cc:304] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.732363 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet fb7fdc91156e41de80872b5c941c5d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:12.732501 11517 raft_consensus.cc:2749] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.733180 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "823620e5486e4cf982fdf9105e8cc4f1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:12.733682 9859 leader_election.cc:343] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:12.734059 9860 leader_election.cc:343] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.734205 9860 leader_election.cc:304] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.734558 11517 raft_consensus.cc:2749] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.736577 9892 tablet_service.cc:1505] Processing CreateTablet for tablet bd2d5c148e5f49ff94d79de3953444a6 (DEFAULT_TABLE table=default.Table_109b65ba33734a5f8ffb302c8fc8caaa [id=bee382f9a8cf4a68a92eb57e2d5bdbf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.736994 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd2d5c148e5f49ff94d79de3953444a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.739246 9308 master_service.cc:954] ReplaceTablet: received request to replace tablet fe7b00e9cf984d12817c9ea860391199 from {username='slave'} at 127.0.0.1:41710
W20251027 21:08:12.741087 9860 leader_election.cc:343] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.741137 10040 ts_tablet_manager.cc:1929] T a4d26f73b8534f08bcaa386d4de75c01 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:12.741236 10040 log.cc:1199] T a4d26f73b8534f08bcaa386d4de75c01 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/a4d26f73b8534f08bcaa386d4de75c01
I20251027 21:08:12.741274 9860 leader_election.cc:304] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.741518 11517 raft_consensus.cc:2749] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:12.741921 9860 leader_election.cc:343] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.744593 9860 leader_election.cc:304] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.746006 11517 raft_consensus.cc:2749] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.746459 10040 ts_tablet_manager.cc:1950] T a4d26f73b8534f08bcaa386d4de75c01 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:12.747335 10041 log.cc:1199] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/371aa8a57aee466ab01652d6061db4c2
I20251027 21:08:12.747494 10031 tablet_bootstrap.cc:492] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:12.747635 10031 ts_tablet_manager.cc:1403] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20251027 21:08:12.747663 10041 ts_tablet_manager.cc:1950] T 371aa8a57aee466ab01652d6061db4c2 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:12.747933 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1137) took 3134 ms. Trace:
I20251027 21:08:12.748010 10041 rpcz_store.cc:276] 1027 21:08:09.613421 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.613739 (+ 318us) service_pool.cc:225] Handling call
1027 21:08:12.547043 (+2933304us) ts_tablet_manager.cc:1151] Deleting tablet 371aa8a57aee466ab01652d6061db4c2
1027 21:08:12.547052 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.550987 (+ 3935us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.566725 (+ 15738us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.747919 (+181194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2933125}
I20251027 21:08:12.748351 10031 raft_consensus.cc:359] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.748427 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 371aa8a57aee466ab01652d6061db4c2 (table default.Table_cd4bceadd9e4409c9ed9dc8365e4ed8e [id=9529122fab2c414e9d718265e42f26ec]) successfully deleted
I20251027 21:08:12.748510 10031 raft_consensus.cc:385] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.748582 10031 raft_consensus.cc:740] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.749020 10029 tablet_bootstrap.cc:492] T 72c1dc3b9eff47ee81df821deb789307 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:12.749190 10029 ts_tablet_manager.cc:1403] T 72c1dc3b9eff47ee81df821deb789307 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20251027 21:08:12.749568 11536 tablet.cc:1722] T 29693cf06bc24930870dc60457126519 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251027 21:08:12.749934 10029 raft_consensus.cc:359] T 72c1dc3b9eff47ee81df821deb789307 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.750183 10029 raft_consensus.cc:385] T 72c1dc3b9eff47ee81df821deb789307 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.750283 10029 raft_consensus.cc:740] T 72c1dc3b9eff47ee81df821deb789307 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.750578 10029 consensus_queue.cc:260] T 72c1dc3b9eff47ee81df821deb789307 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.751543 10029 ts_tablet_manager.cc:1434] T 72c1dc3b9eff47ee81df821deb789307 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20251027 21:08:12.751749 9860 leader_election.cc:343] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.751899 9860 leader_election.cc:304] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.752177 10032 tablet_bootstrap.cc:492] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:12.753147 11521 tablet.cc:1722] T f2710f265ae9485dafec389168328067 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:12.754303 10029 tablet_bootstrap.cc:492] T c9c86253ce914916bee6584675a82067 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:12.756135 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet a4d26f73b8534f08bcaa386d4de75c01 (table default.Table_a6cf2819ed9241cea09738477423c84a [id=f5ef094809e24355a7e021408e2ddd20]) successfully deleted
I20251027 21:08:12.752306 11517 raft_consensus.cc:2749] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.758512 10032 ts_tablet_manager.cc:1403] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.103s user 0.005s sys 0.001s
I20251027 21:08:12.759187 10032 raft_consensus.cc:359] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:12.762248 10029 tablet_bootstrap.cc:654] T c9c86253ce914916bee6584675a82067 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:12.762763 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 283b72f626644ac2b9b61186387b3839 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 283b72f626644ac2b9b61186387b3839 already in progress: creating tablet
W20251027 21:08:12.763334 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 2dd3e77e26c74ac392274844bc3df181 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2dd3e77e26c74ac392274844bc3df181 already in progress: creating tablet
I20251027 21:08:12.765921 10031 consensus_queue.cc:260] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
W20251027 21:08:12.766745 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 04c1f18b611d45c9ae2c7b895240f545 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04c1f18b611d45c9ae2c7b895240f545 already in progress: creating tablet
I20251027 21:08:12.765020 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59300e16b3e148a08a217fc24d1e572f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:12.765614 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "823620e5486e4cf982fdf9105e8cc4f1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:12.767045 9634 raft_consensus.cc:3060] T 823620e5486e4cf982fdf9105e8cc4f1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:12.767508 9303 catalog_manager.cc:5649] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
W20251027 21:08:12.768350 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet cf7b5a0b6cfd460da98558baf887093e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf7b5a0b6cfd460da98558baf887093e already in progress: creating tablet
W20251027 21:08:12.768857 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 1ba47e34f5464b47bc32b2776f37d97d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ba47e34f5464b47bc32b2776f37d97d already in progress: creating tablet
W20251027 21:08:12.769349 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet eeb667b3b10746dbad351c4c123ffd97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eeb667b3b10746dbad351c4c123ffd97 already in progress: creating tablet
I20251027 21:08:12.769658 9634 raft_consensus.cc:2468] T 823620e5486e4cf982fdf9105e8cc4f1 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 1.
W20251027 21:08:12.770093 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet f543e8bcf70c4acab9880ee01af85562 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f543e8bcf70c4acab9880ee01af85562 already in progress: creating tablet
I20251027 21:08:12.759497 10041 tablet_replica.cc:333] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:12.770402 10040 tablet_replica.cc:333] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:12.770607 10040 raft_consensus.cc:2243] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:12.770635 10041 raft_consensus.cc:2243] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:12.770915 10041 raft_consensus.cc:2272] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.771763 10041 ts_tablet_manager.cc:1916] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.774636 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaaf70c950044387a63820079db3c829" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:12.770248 10032 raft_consensus.cc:385] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.774889 9925 raft_consensus.cc:2468] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:12.775029 10032 raft_consensus.cc:740] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
W20251027 21:08:12.775251 9860 leader_election.cc:343] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.775604 9860 leader_election.cc:304] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.775296 10032 consensus_queue.cc:260] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:12.777281 11512 raft_consensus.cc:493] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.777474 11512 raft_consensus.cc:515] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.774454 11544 tablet.cc:1722] T 29693cf06bc24930870dc60457126519 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251027 21:08:12.778532 11512 leader_election.cc:290] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.779029 10043 ts_tablet_manager.cc:1929] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:12.781656 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22dbfa50996b47269983c107cc399000" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:12.782491 10040 raft_consensus.cc:2272] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.782584 10031 ts_tablet_manager.cc:1434] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.035s user 0.002s sys 0.001s
I20251027 21:08:12.782896 10031 tablet_bootstrap.cc:492] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:12.783248 11536 tablet.cc:1722] T b8856c5dc8f144358445165324bfb477 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251027 21:08:12.790359 11498 raft_consensus.cc:2749] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.791844 10032 ts_tablet_manager.cc:1434] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.033s user 0.001s sys 0.001s
I20251027 21:08:12.792241 10032 tablet_bootstrap.cc:492] T 57ef07c764664deaa4b2e9bcae7e7973 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:12.792807 9860 leader_election.cc:304] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.783150 10040 ts_tablet_manager.cc:1916] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.800100 10040 ts_tablet_manager.cc:1929] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:12.800244 10040 log.cc:1199] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/5c34c8f1433b40d491ae05e7c6e60ba1
I20251027 21:08:12.800601 10040 ts_tablet_manager.cc:1950] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:12.801601 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 5c34c8f1433b40d491ae05e7c6e60ba1 (table default.Table_5877315061df40eb80ba9d966632471a [id=c70fb6448778479fa17e811b21830ea5]) successfully deleted
I20251027 21:08:12.803376 10031 tablet_bootstrap.cc:654] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.793577 10032 tablet_bootstrap.cc:654] T 57ef07c764664deaa4b2e9bcae7e7973 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.794075 11536 tablet.cc:1722] T 67cb71ce43244d8ab9e71ead6bb315c9 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:12.805900 11498 raft_consensus.cc:2804] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:12.806005 11498 raft_consensus.cc:697] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Becoming Leader. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Running, Role: LEADER
I20251027 21:08:12.806231 11498 consensus_queue.cc:237] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.800223 10041 ts_tablet_manager.cc:1929] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:12.809566 10041 log.cc:1199] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/4ba13a89a33549af921d3b55023faf15
I20251027 21:08:12.810704 10041 ts_tablet_manager.cc:1950] T 4ba13a89a33549af921d3b55023faf15 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:12.811475 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1138) took 3187 ms. Trace:
I20251027 21:08:12.813210 10041 rpcz_store.cc:276] 1027 21:08:09.623961 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.624029 (+ 68us) service_pool.cc:225] Handling call
1027 21:08:12.759444 (+3135415us) ts_tablet_manager.cc:1151] Deleting tablet 4ba13a89a33549af921d3b55023faf15
1027 21:08:12.759455 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.787232 (+ 27777us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.790068 (+ 2836us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.811458 (+ 21390us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3135241}
I20251027 21:08:12.814028 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 4ba13a89a33549af921d3b55023faf15 (table default.Table_e763bc3cda90472897f0f798424199b4 [id=714f3076ca9f4d399fcc7a36e701ac85]) successfully deleted
I20251027 21:08:12.814488 10041 tablet_replica.cc:333] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:12.814972 10041 raft_consensus.cc:2243] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:12.815667 10041 raft_consensus.cc:2272] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.817009 10041 ts_tablet_manager.cc:1916] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.782341 10029 tablet_bootstrap.cc:492] T c9c86253ce914916bee6584675a82067 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:12.822335 10029 ts_tablet_manager.cc:1403] T c9c86253ce914916bee6584675a82067 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.001s
I20251027 21:08:12.822947 11498 raft_consensus.cc:493] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.823046 11498 raft_consensus.cc:515] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.823022 10029 raft_consensus.cc:359] T c9c86253ce914916bee6584675a82067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.823165 10029 raft_consensus.cc:385] T c9c86253ce914916bee6584675a82067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.823212 10029 raft_consensus.cc:740] T c9c86253ce914916bee6584675a82067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.823355 10029 consensus_queue.cc:260] T c9c86253ce914916bee6584675a82067 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.823498 11498 leader_election.cc:290] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.823745 10029 ts_tablet_manager.cc:1434] T c9c86253ce914916bee6584675a82067 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 21:08:12.823973 10029 tablet_bootstrap.cc:492] T cdbde6272af147829ecadd67d634f049 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:12.824744 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72c1dc3b9eff47ee81df821deb789307" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:12.824913 9630 raft_consensus.cc:2468] T 72c1dc3b9eff47ee81df821deb789307 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 0.
I20251027 21:08:12.825410 10029 tablet_bootstrap.cc:654] T cdbde6272af147829ecadd67d634f049 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.782917 11519 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:59712 (request call id 1240) took 1001 ms. Trace:
I20251027 21:08:12.827010 11519 rpcz_store.cc:276] 1027 21:08:11.781602 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:11.781844 (+ 242us) service_pool.cc:225] Handling call
1027 21:08:12.782903 (+1001059us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:08:11.868920 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1027 21:08:11.869034 (+ 114us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1027 21:08:11.869042 (+ 8us) alter_schema_op.cc:66] Acquired schema lock
1027 21:08:11.869048 (+ 6us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1027 21:08:11.869082 (+ 34us) alter_schema_op.cc:128] START. Timestamp: 1761599291869047
1027 21:08:11.869085 (+ 3us) op_driver.cc:348] REPLICATION: starting
1027 21:08:11.869245 (+ 160us) log.cc:844] Serialized 173 byte log entry
1027 21:08:12.712693 (+843448us) op_driver.cc:464] REPLICATION: finished
1027 21:08:12.713010 (+ 317us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1027 21:08:12.715174 (+ 2164us) log.cc:844] Serialized 5 byte log entry
1027 21:08:12.781616 (+ 66442us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.781871 (+ 255us) log.cc:844] Serialized 14 byte log entry
1027 21:08:12.782862 (+ 991us) alter_schema_op.cc:73] Released schema lock
1027 21:08:12.782868 (+ 6us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":266,"mutex_wait_us":323,"prepare.queue_time_us":124,"prepare.run_cpu_time_us":427,"prepare.run_wall_time_us":426,"raft.queue_time_us":775,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":293,"replication_time_us":843578,"spinlock_wait_cycles":5888}]]}
W20251027 21:08:12.783648 9566 leader_election.cc:343] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.827821 9860 leader_election.cc:304] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
W20251027 21:08:12.827855 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 44227.6us
I20251027 21:08:12.811213 9613 tablet_service.cc:1505] Processing CreateTablet for tablet cc84242b57c042ee9dd44231f9696f1e (DEFAULT_TABLE table=default.Table_38420733ed6d43799874a162313d5828 [id=f943cdf2b6824434beed734129127725]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.828202 11543 raft_consensus.cc:2804] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:12.828231 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc84242b57c042ee9dd44231f9696f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.828269 11543 raft_consensus.cc:493] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:12.828346 11543 raft_consensus.cc:3060] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:12.813740 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 712e1763fe064fd49ee210d991dc56b3 (DEFAULT_TABLE table=default.Table_aef78f959ff04f84a9f5d8d736c7f596 [id=0215aa295a0e493ca6f193c5872cbc5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.813740 9604 tablet_service.cc:1505] Processing CreateTablet for tablet 35ae35f748f547d689f8f341a4ba149f (DEFAULT_TABLE table=default.Table_c3120318fe0f4feda2d71e836f0e02e6 [id=312861325fb54c3aa87964476548ec7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.814199 9603 tablet_service.cc:1505] Processing CreateTablet for tablet 736c27b1adf040208e356137c40f9af4 (DEFAULT_TABLE table=default.Table_1100e497c4ed4782824ba1c9e058fdbc [id=ea1d9535346141898f7cb0c9e23e5732]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.830751 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 712e1763fe064fd49ee210d991dc56b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.830940 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 736c27b1adf040208e356137c40f9af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.831110 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ae35f748f547d689f8f341a4ba149f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.833467 11543 raft_consensus.cc:515] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.834534 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72c1dc3b9eff47ee81df821deb789307" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:12.834734 9630 raft_consensus.cc:3060] T 72c1dc3b9eff47ee81df821deb789307 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:12.838512 9630 raft_consensus.cc:2468] T 72c1dc3b9eff47ee81df821deb789307 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 1.
I20251027 21:08:12.817013 10040 tablet_replica.cc:333] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:12.823979 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72c1dc3b9eff47ee81df821deb789307" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:12.840246 10041 ts_tablet_manager.cc:1929] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:12.840963 10040 raft_consensus.cc:2243] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:12.841208 10040 raft_consensus.cc:2272] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.841776 10040 ts_tablet_manager.cc:1916] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.847601 10043 log.cc:1199] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/a033551a591146039b017b4193b221ca
I20251027 21:08:12.848021 10043 ts_tablet_manager.cc:1950] T a033551a591146039b017b4193b221ca P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:12.848791 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1097) took 4233 ms. Trace:
W20251027 21:08:12.840677 9859 leader_election.cc:343] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.842115 10032 tablet_bootstrap.cc:492] T 57ef07c764664deaa4b2e9bcae7e7973 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:12.850703 10032 ts_tablet_manager.cc:1403] T 57ef07c764664deaa4b2e9bcae7e7973 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.008s
I20251027 21:08:12.845916 9860 leader_election.cc:304] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:12.851341 10032 raft_consensus.cc:359] T 57ef07c764664deaa4b2e9bcae7e7973 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.851637 10032 raft_consensus.cc:385] T 57ef07c764664deaa4b2e9bcae7e7973 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.851708 10032 raft_consensus.cc:740] T 57ef07c764664deaa4b2e9bcae7e7973 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.851908 10032 consensus_queue.cc:260] T 57ef07c764664deaa4b2e9bcae7e7973 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.852126 11498 raft_consensus.cc:2804] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:12.852355 10032 ts_tablet_manager.cc:1434] T 57ef07c764664deaa4b2e9bcae7e7973 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 21:08:12.852602 10032 tablet_bootstrap.cc:492] T dafbb806923c4cf595abde7b7daa1637 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:12.852260 11498 raft_consensus.cc:697] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Becoming Leader. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Running, Role: LEADER
I20251027 21:08:12.852907 11498 consensus_queue.cc:237] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.854035 10032 tablet_bootstrap.cc:654] T dafbb806923c4cf595abde7b7daa1637 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.855381 10041 log.cc:1199] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/0d2d977e360b4ed29c9ed63a309783d6
I20251027 21:08:12.855952 10041 ts_tablet_manager.cc:1950] T 0d2d977e360b4ed29c9ed63a309783d6 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:12.856289 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1139) took 3215 ms. Trace:
I20251027 21:08:12.856403 10041 rpcz_store.cc:276] 1027 21:08:09.641039 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.641108 (+ 69us) service_pool.cc:225] Handling call
1027 21:08:12.814433 (+3173325us) ts_tablet_manager.cc:1151] Deleting tablet 0d2d977e360b4ed29c9ed63a309783d6
1027 21:08:12.814446 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.824868 (+ 10422us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.840167 (+ 15299us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.856275 (+ 16108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3173126}
I20251027 21:08:12.857551 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 0d2d977e360b4ed29c9ed63a309783d6 (table default.Table_fbc3168ad4454ab3a6a83c1226e329f5 [id=9ebc5854bbdc40bfacda411c948f98d6]) successfully deleted
I20251027 21:08:12.857981 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1140) took 3212 ms. Trace:
I20251027 21:08:12.858908 10041 rpcz_store.cc:276] 1027 21:08:09.645201 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.645256 (+ 55us) service_pool.cc:225] Handling call
1027 21:08:12.857848 (+3212592us) ts_tablet_manager.cc:1151] Deleting tablet ece28e82383540bea3457b1cbc65558b
1027 21:08:12.857861 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.857962 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3212410}
W20251027 21:08:12.859447 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet ece28e82383540bea3457b1cbc65558b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ece28e82383540bea3457b1cbc65558b already in progress: creating tablet
I20251027 21:08:12.859827 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1141) took 3213 ms. Trace:
I20251027 21:08:12.859959 10041 rpcz_store.cc:276] 1027 21:08:09.646423 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.646494 (+ 71us) service_pool.cc:225] Handling call
1027 21:08:12.859726 (+3213232us) ts_tablet_manager.cc:1151] Deleting tablet 784df834432244e8812d1802f9e1a1bc
1027 21:08:12.859735 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.859812 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3213071}
I20251027 21:08:12.861016 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd9e6656d8df43afbb7f70706bc14585" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:12.862046 9909 raft_consensus.cc:3060] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:12.861845 9922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22dbfa50996b47269983c107cc399000" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:12.862268 9922 raft_consensus.cc:2468] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:12.865070 9909 raft_consensus.cc:2468] T dd9e6656d8df43afbb7f70706bc14585 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:12.865900 9310 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 0aa4de30e69d4c22a8fd147b62906744
W20251027 21:08:12.866012 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 784df834432244e8812d1802f9e1a1bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 784df834432244e8812d1802f9e1a1bc already in progress: creating tablet
I20251027 21:08:12.866199 10031 tablet_bootstrap.cc:492] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:12.866324 10031 ts_tablet_manager.cc:1403] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.002s
I20251027 21:08:12.866374 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1143) took 3219 ms. Trace:
I20251027 21:08:12.866433 10041 rpcz_store.cc:276] 1027 21:08:09.646696 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.646757 (+ 61us) service_pool.cc:225] Handling call
1027 21:08:12.866274 (+3219517us) ts_tablet_manager.cc:1151] Deleting tablet e24312671b4b4106a1790c61a5548a76
1027 21:08:12.866283 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.866357 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3219403}
W20251027 21:08:12.866802 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet e24312671b4b4106a1790c61a5548a76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e24312671b4b4106a1790c61a5548a76 already in progress: creating tablet
I20251027 21:08:12.867123 10041 tablet_replica.cc:333] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:12.867059 10031 raft_consensus.cc:359] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.867379 10041 raft_consensus.cc:2243] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:12.867583 10041 raft_consensus.cc:2272] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.868180 10041 ts_tablet_manager.cc:1916] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.870935 10032 tablet_bootstrap.cc:492] T dafbb806923c4cf595abde7b7daa1637 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:12.874315 10032 ts_tablet_manager.cc:1403] T dafbb806923c4cf595abde7b7daa1637 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20251027 21:08:12.875156 10032 raft_consensus.cc:359] T dafbb806923c4cf595abde7b7daa1637 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.875371 10032 raft_consensus.cc:385] T dafbb806923c4cf595abde7b7daa1637 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.875480 10032 raft_consensus.cc:740] T dafbb806923c4cf595abde7b7daa1637 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.875766 10032 consensus_queue.cc:260] T dafbb806923c4cf595abde7b7daa1637 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.876259 10032 ts_tablet_manager.cc:1434] T dafbb806923c4cf595abde7b7daa1637 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:12.877817 9310 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet f2710f265ae9485dafec389168328067 (table default.Table_eb44e5cb7bc24597a807d81c1207a162 [id=27d1cff8fc7940c9abb741c8c7d87e24]). Expected version 2 got 0
I20251027 21:08:12.876562 10032 tablet_bootstrap.cc:492] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:12.879446 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c76870d7ca1b43308f0ab94d4ab39b5f
I20251027 21:08:12.880213 10032 tablet_bootstrap.cc:654] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.886185 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet c76870d7ca1b43308f0ab94d4ab39b5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:12.889446 11543 leader_election.cc:290] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.874867 11514 raft_consensus.cc:493] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.858389 9603 tablet_service.cc:1505] Processing CreateTablet for tablet 8909318b2d3344a6a89a84dedff69a42 (DEFAULT_TABLE table=default.Table_8ddc08df525c4d329846f82ba45beeca [id=de0a1d479af3438fb03dba6fef61ff45]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.880930 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet c76870d7ca1b43308f0ab94d4ab39b5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:12.889271 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72c1dc3b9eff47ee81df821deb789307" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:12.889864 11514 raft_consensus.cc:515] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:12.890053 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8909318b2d3344a6a89a84dedff69a42. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:12.890610 9859 leader_election.cc:343] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.891322 11514 leader_election.cc:290] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:12.892076 9711 leader_election.cc:343] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: fde0980aa64c4496861a662f7ed3ef75
I20251027 21:08:12.892400 10041 ts_tablet_manager.cc:1929] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:12.892729 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fde0980aa64c4496861a662f7ed3ef75" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:12.859014 9607 tablet_service.cc:1505] Processing CreateTablet for tablet e90e2e01b9e94b0a97fc6aaf1153dd4e (DEFAULT_TABLE table=default.Table_4cca34ef6c5f47e285603a77b24501bb [id=289a47b40eaa4b81b2501efacf6eb1af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:12.893370 9710 leader_election.cc:304] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:12.893735 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e90e2e01b9e94b0a97fc6aaf1153dd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.893824 11514 raft_consensus.cc:2749] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.894035 10031 raft_consensus.cc:385] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.894104 10031 raft_consensus.cc:740] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.894083 10041 log.cc:1199] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/0e64d21abef342de86555e876f020787
I20251027 21:08:12.894594 10041 ts_tablet_manager.cc:1950] T 0e64d21abef342de86555e876f020787 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:12.895534 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1142) took 3248 ms. Trace:
I20251027 21:08:12.896311 10041 rpcz_store.cc:276] 1027 21:08:09.646585 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.646795 (+ 210us) service_pool.cc:225] Handling call
1027 21:08:12.867079 (+3220284us) ts_tablet_manager.cc:1151] Deleting tablet 0e64d21abef342de86555e876f020787
1027 21:08:12.867089 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.873724 (+ 6635us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.892308 (+ 18584us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.895518 (+ 3210us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3219540}
I20251027 21:08:12.896917 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 0e64d21abef342de86555e876f020787 (table default.Table_8ebde3403af34f4589ee6222700b2283 [id=cfdf2d0e9745403ea4bbeca6b4ed7439]) successfully deleted
I20251027 21:08:12.897248 10041 tablet_replica.cc:333] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:12.897549 10031 consensus_queue.cc:260] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.899214 10031 ts_tablet_manager.cc:1434] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20251027 21:08:12.863611 11512 raft_consensus.cc:493] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.899916 11512 raft_consensus.cc:515] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.864238 11519 tablet.cc:1722] T d672118ecd6f4094855adf106e5f7115 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:12.902644 10041 raft_consensus.cc:2243] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:12.902951 10041 raft_consensus.cc:2272] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.903607 10041 ts_tablet_manager.cc:1916] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.905295 11512 leader_election.cc:290] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.910216 10043 rpcz_store.cc:276] 1027 21:08:08.615781 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.615818 (+ 37us) service_pool.cc:225] Handling call
1027 21:08:12.444106 (+3828288us) ts_tablet_manager.cc:1151] Deleting tablet a033551a591146039b017b4193b221ca
1027 21:08:12.444116 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.683335 (+239219us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.778930 (+ 95595us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.848775 (+ 69845us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3828169}
I20251027 21:08:12.910585 10043 tablet_replica.cc:333] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:12.910818 10043 raft_consensus.cc:2243] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:12.911170 10043 pending_rounds.cc:70] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14: Trying to abort 1 pending ops.
I20251027 21:08:12.911250 10043 pending_rounds.cc:77] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510701060292608 op_type: NO_OP noop_request { }
I20251027 21:08:12.911368 10043 raft_consensus.cc:2889] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:12.911479 10043 raft_consensus.cc:2272] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.912055 10041 ts_tablet_manager.cc:1929] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:12.900876 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "952583fd7fde4b4baccdb98c01bdf9e1" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:12.912979 10031 tablet_bootstrap.cc:492] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:08:12.913509 9566 leader_election.cc:343] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.917061 10031 tablet_bootstrap.cc:654] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.918280 10041 log.cc:1199] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/952583fd7fde4b4baccdb98c01bdf9e1
I20251027 21:08:12.918638 10041 ts_tablet_manager.cc:1950] T 952583fd7fde4b4baccdb98c01bdf9e1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:12.919332 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1144) took 3271 ms. Trace:
I20251027 21:08:12.919456 10041 rpcz_store.cc:276] 1027 21:08:09.647667 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.649116 (+ 1449us) service_pool.cc:225] Handling call
1027 21:08:12.897208 (+3248092us) ts_tablet_manager.cc:1151] Deleting tablet 952583fd7fde4b4baccdb98c01bdf9e1
1027 21:08:12.897217 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.909182 (+ 11965us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.911987 (+ 2805us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.919314 (+ 7327us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3231934}
I20251027 21:08:12.919905 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 952583fd7fde4b4baccdb98c01bdf9e1 (table default.Table_df5f4de40f7f46728f38580d23124544 [id=7527117eef9d46a694977cbf2c073974]) successfully deleted
I20251027 21:08:12.915300 11543 raft_consensus.cc:493] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.920311 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1145) took 3195 ms. Trace:
I20251027 21:08:12.920321 11543 raft_consensus.cc:515] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.920986 10032 tablet_bootstrap.cc:492] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:12.921096 10032 ts_tablet_manager.cc:1403] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.007s
I20251027 21:08:12.921619 11543 leader_election.cc:290] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.921715 10032 raft_consensus.cc:359] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.921847 10032 raft_consensus.cc:385] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.921901 10032 raft_consensus.cc:740] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.922150 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "334c580862bd4bf09be3f1710aa6a092" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:12.922209 10032 consensus_queue.cc:260] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.922632 10032 ts_tablet_manager.cc:1434] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 21:08:12.922909 10032 tablet_bootstrap.cc:492] T d0c53b7b32c84a51a967e2a6a22258c6 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:12.924064 10032 tablet_bootstrap.cc:654] T d0c53b7b32c84a51a967e2a6a22258c6 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.920442 10041 rpcz_store.cc:276] 1027 21:08:09.724904 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.724966 (+ 62us) service_pool.cc:225] Handling call
1027 21:08:12.920191 (+3195225us) ts_tablet_manager.cc:1151] Deleting tablet 280ee23d80d64587b18811ff06340eee
1027 21:08:12.920202 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.920296 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3195052}
I20251027 21:08:12.924876 9602 tablet_service.cc:1505] Processing CreateTablet for tablet bd2d5c148e5f49ff94d79de3953444a6 (DEFAULT_TABLE table=default.Table_109b65ba33734a5f8ffb302c8fc8caaa [id=bee382f9a8cf4a68a92eb57e2d5bdbf6]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:12.925256 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 280ee23d80d64587b18811ff06340eee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 280ee23d80d64587b18811ff06340eee already in progress: creating tablet
I20251027 21:08:12.925401 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd2d5c148e5f49ff94d79de3953444a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:12.925698 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1146) took 3200 ms. Trace:
W20251027 21:08:12.922745 9860 leader_election.cc:343] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.926647 10041 rpcz_store.cc:276] 1027 21:08:09.725070 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.725169 (+ 99us) service_pool.cc:225] Handling call
1027 21:08:12.925557 (+3200388us) ts_tablet_manager.cc:1151] Deleting tablet 2a9c6bf4f75144f487305520b3714b84
1027 21:08:12.925567 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.925676 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3200232}
W20251027 21:08:12.927000 9568 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 63230us
W20251027 21:08:12.927309 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 2a9c6bf4f75144f487305520b3714b84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a9c6bf4f75144f487305520b3714b84 already in progress: creating tablet
W20251027 21:08:12.927485 9568 leader_election.cc:343] T 9ed4f9c61ad54e1292281b30bcd19533 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 9ed4f9c61ad54e1292281b30bcd19533
I20251027 21:08:12.927582 9568 leader_election.cc:304] T 9ed4f9c61ad54e1292281b30bcd19533 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.927783 11411 raft_consensus.cc:2749] T 9ed4f9c61ad54e1292281b30bcd19533 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.930305 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1150) took 3005 ms. Trace:
I20251027 21:08:12.930516 10043 ts_tablet_manager.cc:1916] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.930624 10041 rpcz_store.cc:276] 1027 21:08:09.925245 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.925287 (+ 42us) service_pool.cc:225] Handling call
1027 21:08:12.930181 (+3004894us) ts_tablet_manager.cc:1151] Deleting tablet bc387163e2d64532b87c32c5842a285a
1027 21:08:12.930207 (+ 26us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.930287 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3002096}
W20251027 21:08:12.931282 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet bc387163e2d64532b87c32c5842a285a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc387163e2d64532b87c32c5842a285a already in progress: creating tablet
I20251027 21:08:12.931793 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1151) took 3002 ms. Trace:
I20251027 21:08:12.932219 10041 rpcz_store.cc:276] 1027 21:08:09.929327 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.929371 (+ 44us) service_pool.cc:225] Handling call
1027 21:08:12.931661 (+3002290us) ts_tablet_manager.cc:1151] Deleting tablet 53948dc94c334329bdf81f1eba94863c
1027 21:08:12.931684 (+ 23us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.931774 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2972262}
W20251027 21:08:12.932929 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 53948dc94c334329bdf81f1eba94863c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53948dc94c334329bdf81f1eba94863c already in progress: creating tablet
I20251027 21:08:12.933418 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1158) took 3001 ms. Trace:
I20251027 21:08:12.933876 11543 raft_consensus.cc:493] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.934109 11543 raft_consensus.cc:515] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.934376 9568 leader_election.cc:304] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.934684 11411 raft_consensus.cc:2804] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:12.934794 11411 raft_consensus.cc:493] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:12.934895 11411 raft_consensus.cc:3060] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:12.935133 11543 leader_election.cc:290] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.935144 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "371051fb3491426e8cf7797317baa3db" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:12.933840 10041 rpcz_store.cc:276] 1027 21:08:09.931761 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.931811 (+ 50us) service_pool.cc:225] Handling call
1027 21:08:12.933297 (+3001486us) ts_tablet_manager.cc:1151] Deleting tablet d042a15408484fbc9ee9666476b103d5
1027 21:08:12.933309 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.933401 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2954899}
W20251027 21:08:12.935644 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet d042a15408484fbc9ee9666476b103d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d042a15408484fbc9ee9666476b103d5 already in progress: creating tablet
W20251027 21:08:12.935700 9860 leader_election.cc:343] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.936170 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1173) took 2757 ms. Trace:
I20251027 21:08:12.936600 10041 rpcz_store.cc:276] 1027 21:08:10.184551 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.184589 (+ 38us) service_pool.cc:225] Handling call
1027 21:08:12.936047 (+2751458us) ts_tablet_manager.cc:1151] Deleting tablet befdd3a485a1426b8a3adf340d4eae4f
1027 21:08:12.936059 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.936152 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2751201}
W20251027 21:08:12.937307 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet befdd3a485a1426b8a3adf340d4eae4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet befdd3a485a1426b8a3adf340d4eae4f already in progress: creating tablet
I20251027 21:08:12.937433 11411 raft_consensus.cc:515] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.937772 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1176) took 2681 ms. Trace:
I20251027 21:08:12.938099 10041 rpcz_store.cc:276] 1027 21:08:10.255870 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.262314 (+ 6444us) service_pool.cc:225] Handling call
1027 21:08:12.937678 (+2675364us) ts_tablet_manager.cc:1151] Deleting tablet ca14d51a66de4a0d8476117e0984d1e9
1027 21:08:12.937687 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.937756 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2675154}
W20251027 21:08:12.938663 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet ca14d51a66de4a0d8476117e0984d1e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca14d51a66de4a0d8476117e0984d1e9 already in progress: creating tablet
I20251027 21:08:12.938797 11411 leader_election.cc:290] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.939013 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1175) took 2683 ms. Trace:
I20251027 21:08:12.940192 10041 rpcz_store.cc:276] 1027 21:08:10.255757 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.266268 (+ 10511us) service_pool.cc:225] Handling call
1027 21:08:12.938929 (+2672661us) ts_tablet_manager.cc:1151] Deleting tablet 8d61afc9ce6b4b80b5e64259b9c04f85
1027 21:08:12.938937 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.939000 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2672454}
W20251027 21:08:12.940719 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 8d61afc9ce6b4b80b5e64259b9c04f85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d61afc9ce6b4b80b5e64259b9c04f85 already in progress: creating tablet
I20251027 21:08:12.941040 10041 tablet_replica.cc:333] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:12.941421 10041 raft_consensus.cc:2243] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:12.941799 10041 raft_consensus.cc:2272] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.946659 10041 ts_tablet_manager.cc:1916] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.946844 9306 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 7d6095dddfad41ec9036024f33a43b6e
I20251027 21:08:12.947149 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_423b90a6cbf94715b7d8344c550bb10f" } new_table_name: "default.Table_f8fba89abfb24219bea08e91f56fb019" modify_external_catalogs: true
I20251027 21:08:12.949118 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 7062871e266c4603bab851a837ef9aec
I20251027 21:08:12.950701 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet 7062871e266c4603bab851a837ef9aec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:12.951017 10032 tablet_bootstrap.cc:492] T d0c53b7b32c84a51a967e2a6a22258c6 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:12.951050 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_adad32ac270d4323b3de1c871664d15d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:12.951304 10032 ts_tablet_manager.cc:1403] T d0c53b7b32c84a51a967e2a6a22258c6 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.002s
W20251027 21:08:12.951546 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adad32ac270d4323b3de1c871664d15d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:12.952147 10032 raft_consensus.cc:359] T d0c53b7b32c84a51a967e2a6a22258c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.952381 10032 raft_consensus.cc:385] T d0c53b7b32c84a51a967e2a6a22258c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.952469 10032 raft_consensus.cc:740] T d0c53b7b32c84a51a967e2a6a22258c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.952584 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3c794af372fc4ed8ab15aecdf30067f9" } new_table_name: "default.Table_1eded5e5214c4d79820e0ef2df2fbb1f" modify_external_catalogs: true
I20251027 21:08:12.952734 10032 consensus_queue.cc:260] T d0c53b7b32c84a51a967e2a6a22258c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.954106 10032 ts_tablet_manager.cc:1434] T d0c53b7b32c84a51a967e2a6a22258c6 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:08:12.921501 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "334c580862bd4bf09be3f1710aa6a092" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:12.924320 11514 raft_consensus.cc:493] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:12.954923 11514 raft_consensus.cc:515] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.955199 10032 tablet_bootstrap.cc:492] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:12.955732 11514 leader_election.cc:290] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
W20251027 21:08:12.956207 9859 leader_election.cc:343] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.956331 9859 leader_election.cc:304] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:12.956665 9711 leader_election.cc:343] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 1d64a13667194c12864fa388eb88a380
I20251027 21:08:12.956903 10032 tablet_bootstrap.cc:654] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:12.957101 11543 raft_consensus.cc:2749] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.928169 10040 ts_tablet_manager.cc:1929] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:12.935223 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "371051fb3491426e8cf7797317baa3db" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:12.938488 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaaf70c950044387a63820079db3c829" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:12.950212 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 7062871e266c4603bab851a837ef9aec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:08:12.958058 9566 leader_election.cc:343] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.958274 10040 log.cc:1199] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/aabaa34787cf4d179822269b7f323314
I20251027 21:08:12.958561 10040 ts_tablet_manager.cc:1950] T aabaa34787cf4d179822269b7f323314 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:12.959292 10040 tablet_replica.cc:333] T 631ff72613734913818a3cad24f20dbc P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:12.959314 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d64a13667194c12864fa388eb88a380" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:12.959467 10040 raft_consensus.cc:2243] T 631ff72613734913818a3cad24f20dbc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:12.959579 10040 raft_consensus.cc:2272] T 631ff72613734913818a3cad24f20dbc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.960150 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet aabaa34787cf4d179822269b7f323314 (table default.Table_0eecf2c874144e7594c825f01fcc5770 [id=a0a10aefbf4c4ab083982e9e46bba513]) successfully deleted
W20251027 21:08:12.960427 9859 leader_election.cc:343] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.960548 9859 leader_election.cc:304] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.960974 11543 raft_consensus.cc:2749] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.961311 9710 leader_election.cc:304] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:12.961714 11514 raft_consensus.cc:2749] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:12.962788 9568 leader_election.cc:304] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.963056 10041 ts_tablet_manager.cc:1929] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:12.964120 10041 log.cc:1199] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/423047dfbf084bfcb881efeb0352f88d
I20251027 21:08:12.964797 10041 ts_tablet_manager.cc:1950] T 423047dfbf084bfcb881efeb0352f88d P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:12.970846 10040 ts_tablet_manager.cc:1916] T 631ff72613734913818a3cad24f20dbc P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.970911 10029 tablet_bootstrap.cc:492] T cdbde6272af147829ecadd67d634f049 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:12.971050 10029 ts_tablet_manager.cc:1403] T cdbde6272af147829ecadd67d634f049 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.147s user 0.000s sys 0.008s
I20251027 21:08:12.972956 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 4c60797e8cd5442b9eef86ee51d4b9c6
I20251027 21:08:12.972956 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1174) took 2717 ms. Trace:
I20251027 21:08:12.973034 10041 rpcz_store.cc:276] 1027 21:08:10.255614 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.310970 (+ 55356us) service_pool.cc:225] Handling call
1027 21:08:12.940991 (+2630021us) ts_tablet_manager.cc:1151] Deleting tablet 423047dfbf084bfcb881efeb0352f88d
1027 21:08:12.941000 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.959840 (+ 18840us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.962968 (+ 3128us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.972937 (+ 9969us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2629861}
I20251027 21:08:12.973390 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 423047dfbf084bfcb881efeb0352f88d (table default.Table_ad2a3d4173ca45948828915983300f00 [id=25d8790f144f475d92de13eae144bc44]) successfully deleted
I20251027 21:08:12.973601 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e22a34d2f5e848fcbe891c4d0325fad4" } new_table_name: "default.Table_4a9b706e9ba0478081f45c3b26f6ad57" modify_external_catalogs: true
I20251027 21:08:12.973884 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c60797e8cd5442b9eef86ee51d4b9c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:12.973937 10041 tablet_replica.cc:333] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:12.974116 10041 raft_consensus.cc:2243] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:12.970909 10031 tablet_bootstrap.cc:492] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:12.974289 10041 raft_consensus.cc:2272] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.974332 10031 ts_tablet_manager.cc:1403] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.000s
I20251027 21:08:12.974771 10041 ts_tablet_manager.cc:1916] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.975901 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c60797e8cd5442b9eef86ee51d4b9c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:12.976342 11512 raft_consensus.cc:2804] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:12.976476 11512 raft_consensus.cc:493] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:12.976590 11512 raft_consensus.cc:3060] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:12.979285 11521 tablet.cc:1722] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251027 21:08:12.980350 11512 raft_consensus.cc:515] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.980671 10041 ts_tablet_manager.cc:1929] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:12.980767 10041 log.cc:1199] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/980eb6755aee4bcfb2ea44d84009021f
I20251027 21:08:12.981086 10041 ts_tablet_manager.cc:1950] T 980eb6755aee4bcfb2ea44d84009021f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:12.981174 11512 leader_election.cc:290] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:12.981549 10029 raft_consensus.cc:359] T cdbde6272af147829ecadd67d634f049 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.981680 10029 raft_consensus.cc:385] T cdbde6272af147829ecadd67d634f049 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.981725 10029 raft_consensus.cc:740] T cdbde6272af147829ecadd67d634f049 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:12.981720 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1189) took 2610 ms. Trace:
I20251027 21:08:12.981791 10041 rpcz_store.cc:276] 1027 21:08:10.371344 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.372224 (+ 880us) service_pool.cc:225] Handling call
1027 21:08:12.973893 (+2601669us) ts_tablet_manager.cc:1151] Deleting tablet 980eb6755aee4bcfb2ea44d84009021f
1027 21:08:12.973904 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.977861 (+ 3957us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.980585 (+ 2724us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.981703 (+ 1118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2601503}
I20251027 21:08:12.981921 10029 consensus_queue.cc:260] T cdbde6272af147829ecadd67d634f049 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.982183 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 980eb6755aee4bcfb2ea44d84009021f (table default.Table_bb56319b6fe0405991a4d171477621e7 [id=42c8fedcabe0435dbaa11427141228ed]) successfully deleted
I20251027 21:08:12.982401 10029 ts_tablet_manager.cc:1434] T cdbde6272af147829ecadd67d634f049 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20251027 21:08:12.982388 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22dbfa50996b47269983c107cc399000" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:12.982635 10029 tablet_bootstrap.cc:492] T 0e4d35f229bc46e1859ded7c8196ab9d P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
W20251027 21:08:12.983130 9566 leader_election.cc:343] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:12.983629 11519 tablet.cc:1722] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251027 21:08:12.984601 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1190) took 2553 ms. Trace:
I20251027 21:08:12.984688 10041 rpcz_store.cc:276] 1027 21:08:10.431179 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.431245 (+ 66us) service_pool.cc:225] Handling call
1027 21:08:12.984495 (+2553250us) ts_tablet_manager.cc:1151] Deleting tablet 44804ae1c85749cb9286777e7b5c5707
1027 21:08:12.984504 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.984586 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2505433}
I20251027 21:08:12.984848 10029 tablet_bootstrap.cc:654] T 0e4d35f229bc46e1859ded7c8196ab9d P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:12.985069 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 44804ae1c85749cb9286777e7b5c5707 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44804ae1c85749cb9286777e7b5c5707 already in progress: creating tablet
I20251027 21:08:12.985850 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1192) took 2501 ms. Trace:
I20251027 21:08:12.986202 10041 rpcz_store.cc:276] 1027 21:08:10.484718 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.484797 (+ 79us) service_pool.cc:225] Handling call
1027 21:08:12.985753 (+2500956us) ts_tablet_manager.cc:1151] Deleting tablet c97cba61e08d42a7bdb3f51db0a63ec8
1027 21:08:12.985762 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.985836 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2500794}
W20251027 21:08:12.986769 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet c97cba61e08d42a7bdb3f51db0a63ec8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c97cba61e08d42a7bdb3f51db0a63ec8 already in progress: creating tablet
I20251027 21:08:12.987154 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1191) took 2513 ms. Trace:
I20251027 21:08:12.987485 10041 rpcz_store.cc:276] 1027 21:08:10.473997 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.474827 (+ 830us) service_pool.cc:225] Handling call
1027 21:08:12.987066 (+2512239us) ts_tablet_manager.cc:1151] Deleting tablet 93e8b973b39244be9de1107b14d6d93f
1027 21:08:12.987074 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.987141 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2496308}
W20251027 21:08:12.988111 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 93e8b973b39244be9de1107b14d6d93f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93e8b973b39244be9de1107b14d6d93f already in progress: creating tablet
I20251027 21:08:12.988495 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1193) took 2469 ms. Trace:
I20251027 21:08:12.988821 10041 rpcz_store.cc:276] 1027 21:08:10.519126 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.519179 (+ 53us) service_pool.cc:225] Handling call
1027 21:08:12.988414 (+2469235us) ts_tablet_manager.cc:1151] Deleting tablet 824ff6913b0142d5889c75aa8e3cba3e
1027 21:08:12.988422 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.988481 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2469006}
I20251027 21:08:12.988914 10031 raft_consensus.cc:359] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.989114 10031 raft_consensus.cc:385] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:12.989174 10031 raft_consensus.cc:740] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
W20251027 21:08:12.989483 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 824ff6913b0142d5889c75aa8e3cba3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 824ff6913b0142d5889c75aa8e3cba3e already in progress: creating tablet
I20251027 21:08:12.989817 10041 tablet_replica.cc:333] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:12.990202 10041 raft_consensus.cc:2243] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:12.990624 10041 raft_consensus.cc:2272] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:12.991258 9568 leader_election.cc:304] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:12.991578 11373 raft_consensus.cc:2804] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:12.991674 11373 raft_consensus.cc:697] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:12.991848 10041 ts_tablet_manager.cc:1916] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:12.991863 11373 consensus_queue.cc:237] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:12.993518 10031 consensus_queue.cc:260] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:12.995110 10031 ts_tablet_manager.cc:1434] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20251027 21:08:12.997674 10041 ts_tablet_manager.cc:1929] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:12.997789 10041 log.cc:1199] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/9351a369dc1e4ac49f6ad419eab3a4c5
I20251027 21:08:12.998072 10041 ts_tablet_manager.cc:1950] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:13.004175 11543 raft_consensus.cc:493] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.004334 11543 raft_consensus.cc:515] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:13.004920 11543 leader_election.cc:290] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.005123 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1194) took 2405 ms. Trace:
I20251027 21:08:13.005188 10041 rpcz_store.cc:276] 1027 21:08:10.600112 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.600186 (+ 74us) service_pool.cc:225] Handling call
1027 21:08:12.989777 (+2389591us) ts_tablet_manager.cc:1151] Deleting tablet 9351a369dc1e4ac49f6ad419eab3a4c5
1027 21:08:12.989785 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.994733 (+ 4948us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:12.997584 (+ 2851us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.005110 (+ 7526us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2389381}
I20251027 21:08:13.005540 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 9351a369dc1e4ac49f6ad419eab3a4c5 (table default.Table_3b49d647ca194ad9bf5524812c6837a2 [id=a12d4ab777c34431beaa8f87ddbc7135]) successfully deleted
I20251027 21:08:13.006266 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cb8ca3a50c148e6b602e949650353f8" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:13.006749 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1195) took 2322 ms. Trace:
I20251027 21:08:13.006824 10041 rpcz_store.cc:276] 1027 21:08:10.683965 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.684025 (+ 60us) service_pool.cc:225] Handling call
1027 21:08:13.005390 (+2321365us) ts_tablet_manager.cc:1151] Deleting tablet f954daa93d5a42bfabfbf8fd4b72172b
1027 21:08:13.005399 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.006734 (+ 1335us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2321169}
I20251027 21:08:13.007126 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1197) took 2259 ms. Trace:
W20251027 21:08:13.007171 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet f954daa93d5a42bfabfbf8fd4b72172b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f954daa93d5a42bfabfbf8fd4b72172b already in progress: creating tablet
W20251027 21:08:13.007194 9860 leader_election.cc:343] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.007189 10041 rpcz_store.cc:276] 1027 21:08:10.747167 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.747230 (+ 63us) service_pool.cc:225] Handling call
1027 21:08:13.007037 (+2259807us) ts_tablet_manager.cc:1151] Deleting tablet 0166ba1e634747cd866fabc4e9acf1b5
1027 21:08:13.007047 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.007113 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2259637}
W20251027 21:08:13.008177 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 0166ba1e634747cd866fabc4e9acf1b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0166ba1e634747cd866fabc4e9acf1b5 already in progress: creating tablet
I20251027 21:08:13.008586 10041 tablet_replica.cc:333] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:13.008775 10041 raft_consensus.cc:2243] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.008910 10041 raft_consensus.cc:2272] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.009343 10041 ts_tablet_manager.cc:1916] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.010620 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cb8ca3a50c148e6b602e949650353f8" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:13.011034 9859 leader_election.cc:343] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.011132 9859 leader_election.cc:304] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.011459 11543 raft_consensus.cc:2749] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.013803 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7768a03566534f8bbcacc53a14a957db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:13.014462 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7768a03566534f8bbcacc53a14a957db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:13.015039 10041 ts_tablet_manager.cc:1929] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:13.015122 10041 log.cc:1199] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/22dbfa50996b47269983c107cc399000
I20251027 21:08:13.015375 10041 ts_tablet_manager.cc:1950] T 22dbfa50996b47269983c107cc399000 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:13.016021 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1196) took 2269 ms. Trace:
I20251027 21:08:13.016103 10041 rpcz_store.cc:276] 1027 21:08:10.747052 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.747390 (+ 338us) service_pool.cc:225] Handling call
1027 21:08:13.008544 (+2261154us) ts_tablet_manager.cc:1151] Deleting tablet 22dbfa50996b47269983c107cc399000
1027 21:08:13.008554 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.012180 (+ 3626us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.014973 (+ 2793us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.016003 (+ 1030us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2260989}
I20251027 21:08:13.016304 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet a033551a591146039b017b4193b221ca (table default.Table_7a01697cdee04a929068e3297b598128 [id=84d2047a211e479e89130346cedd9fdf]) successfully deleted
I20251027 21:08:13.016438 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 22dbfa50996b47269983c107cc399000 (table default.Table_cb55ac1fcbad4075b84ec473bf89aa2b [id=be549850a5104d288311e062e1362583]) successfully deleted
I20251027 21:08:13.016671 10031 tablet_bootstrap.cc:492] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:13.016865 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1206) took 2181 ms. Trace:
I20251027 21:08:13.016937 10041 rpcz_store.cc:276] 1027 21:08:10.835227 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.846586 (+ 11359us) service_pool.cc:225] Handling call
1027 21:08:13.016755 (+2170169us) ts_tablet_manager.cc:1151] Deleting tablet 953b137f14a44955ae41ab503d04b43c
1027 21:08:13.016763 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.016850 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2167586}
W20251027 21:08:13.017288 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 953b137f14a44955ae41ab503d04b43c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 953b137f14a44955ae41ab503d04b43c already in progress: creating tablet
I20251027 21:08:13.017617 10041 tablet_replica.cc:333] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:13.017843 10041 raft_consensus.cc:2243] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.017776 11546 tablet.cc:1722] T d672118ecd6f4094855adf106e5f7115 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:13.022353 10041 raft_consensus.cc:2272] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.023360 10041 ts_tablet_manager.cc:1916] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.024082 10032 tablet_bootstrap.cc:492] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:13.024215 10032 ts_tablet_manager.cc:1403] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.069s user 0.008s sys 0.017s
I20251027 21:08:13.024986 10032 raft_consensus.cc:359] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.025134 10032 raft_consensus.cc:385] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.025180 10032 raft_consensus.cc:740] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.025350 10032 consensus_queue.cc:260] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.025846 10032 ts_tablet_manager.cc:1434] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:13.027539 10031 tablet_bootstrap.cc:654] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.032393 10041 ts_tablet_manager.cc:1929] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:13.032495 10041 log.cc:1199] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/5cb8ca3a50c148e6b602e949650353f8
I20251027 21:08:13.032807 10041 ts_tablet_manager.cc:1950] T 5cb8ca3a50c148e6b602e949650353f8 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:13.033291 10032 tablet_bootstrap.cc:492] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:13.034770 10032 tablet_bootstrap.cc:654] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.036180 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaaf70c950044387a63820079db3c829" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:13.036329 9909 raft_consensus.cc:3060] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:13.039243 10040 ts_tablet_manager.cc:1929] T 631ff72613734913818a3cad24f20dbc P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:13.039332 10040 log.cc:1199] T 631ff72613734913818a3cad24f20dbc P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/631ff72613734913818a3cad24f20dbc
I20251027 21:08:13.039372 9909 raft_consensus.cc:2468] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:13.049562 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1205) took 2214 ms. Trace:
I20251027 21:08:13.049655 10041 rpcz_store.cc:276] 1027 21:08:10.835102 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.846881 (+ 11779us) service_pool.cc:225] Handling call
1027 21:08:13.017579 (+2170698us) ts_tablet_manager.cc:1151] Deleting tablet 5cb8ca3a50c148e6b602e949650353f8
1027 21:08:13.017587 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.029561 (+ 11974us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.032322 (+ 2761us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.049546 (+ 17224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2162134}
I20251027 21:08:13.049991 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 5cb8ca3a50c148e6b602e949650353f8 (table default.Table_b7607fa1c70d431d97f9e3f8a7c3d667 [id=080bf50609c045bd9b05f4e9144feac5]) successfully deleted
I20251027 21:08:13.050657 10041 tablet_replica.cc:333] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:13.050835 10041 raft_consensus.cc:2243] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.050972 10041 raft_consensus.cc:2272] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.051430 10041 ts_tablet_manager.cc:1916] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.052796 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_ea0fba68a1e2447bb0379255e2aa61d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:13.053311 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea0fba68a1e2447bb0379255e2aa61d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:13.053486 10031 tablet_bootstrap.cc:492] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:13.053597 10031 ts_tablet_manager.cc:1403] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20251027 21:08:13.054253 10031 raft_consensus.cc:359] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:13.054453 10031 raft_consensus.cc:385] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.054503 10031 raft_consensus.cc:740] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.054870 10031 consensus_queue.cc:260] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:13.057718 9771 raft_consensus.cc:3060] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:13.058907 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 7062871e266c4603bab851a837ef9aec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:13.059070 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c60797e8cd5442b9eef86ee51d4b9c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:13.065294 10041 ts_tablet_manager.cc:1929] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:13.065387 10041 log.cc:1199] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/e7df6f0acec34d399aa25b78c3ed7c7a
I20251027 21:08:13.065673 10041 ts_tablet_manager.cc:1950] T e7df6f0acec34d399aa25b78c3ed7c7a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:13.066380 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1208) took 2113 ms. Trace:
I20251027 21:08:13.066462 10041 rpcz_store.cc:276] 1027 21:08:10.953332 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.958065 (+ 4733us) service_pool.cc:225] Handling call
1027 21:08:13.050606 (+2092541us) ts_tablet_manager.cc:1151] Deleting tablet e7df6f0acec34d399aa25b78c3ed7c7a
1027 21:08:13.050615 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.061026 (+ 10411us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.065211 (+ 4185us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.066359 (+ 1148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2092264}
I20251027 21:08:13.066870 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet e7df6f0acec34d399aa25b78c3ed7c7a (table default.Table_82b8023e5cb34761981a49e164eb36c4 [id=635f029832b249c590d149e5c44ca8f8]) successfully deleted
I20251027 21:08:13.066939 10029 tablet_bootstrap.cc:492] T 0e4d35f229bc46e1859ded7c8196ab9d P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:13.067054 10029 ts_tablet_manager.cc:1403] T 0e4d35f229bc46e1859ded7c8196ab9d P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.000s
I20251027 21:08:13.067752 10041 tablet_replica.cc:333] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:13.068131 10041 raft_consensus.cc:2243] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.068500 10041 raft_consensus.cc:2272] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.069186 10041 ts_tablet_manager.cc:1916] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.072741 10031 ts_tablet_manager.cc:1434] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
W20251027 21:08:13.074047 9568 leader_election.cc:343] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 952583fd7fde4b4baccdb98c01bdf9e1
I20251027 21:08:13.077930 10031 tablet_bootstrap.cc:492] T d672118ecd6f4094855adf106e5f7115 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:13.080945 10041 ts_tablet_manager.cc:1929] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:13.081049 10041 log.cc:1199] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/2752b86151a6428c88bfaac1ff72cfb2
I20251027 21:08:13.081329 10041 ts_tablet_manager.cc:1950] T 2752b86151a6428c88bfaac1ff72cfb2 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:13.082003 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1207) took 2140 ms. Trace:
I20251027 21:08:13.082089 10041 rpcz_store.cc:276] 1027 21:08:10.941633 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:11.019967 (+ 78334us) service_pool.cc:225] Handling call
1027 21:08:13.067709 (+2047742us) ts_tablet_manager.cc:1151] Deleting tablet 2752b86151a6428c88bfaac1ff72cfb2
1027 21:08:13.067718 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.073586 (+ 5868us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.080863 (+ 7277us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.081987 (+ 1124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2047361}
I20251027 21:08:13.082479 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 2752b86151a6428c88bfaac1ff72cfb2 (table default.Table_400ccee830744304b52118f36ab774ff [id=ba21340acd344a59865d26d398ee1937]) successfully deleted
I20251027 21:08:13.082976 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet fe7b00e9cf984d12817c9ea860391199
I20251027 21:08:13.083408 9308 catalog_manager.cc:6719] ReplaceTablet: tablet fe7b00e9cf984d12817c9ea860391199 deleted and replaced by tablet 023eb8eb693d446caef49e0eaecd147c
I20251027 21:08:13.084441 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet fe7b00e9cf984d12817c9ea860391199 with delete_type TABLET_DATA_DELETED (replaced by tablet 023eb8eb693d446caef49e0eaecd147c) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:13.084656 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet fe7b00e9cf984d12817c9ea860391199 with delete_type TABLET_DATA_DELETED (replaced by tablet 023eb8eb693d446caef49e0eaecd147c) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:13.085848 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1211) took 1835 ms. Trace:
I20251027 21:08:13.085925 10041 rpcz_store.cc:276] 1027 21:08:11.250655 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:11.250717 (+ 62us) service_pool.cc:225] Handling call
1027 21:08:13.085734 (+1835017us) ts_tablet_manager.cc:1151] Deleting tablet bd6152f7ac8a4123872098955b8837ac
1027 21:08:13.085744 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.085833 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1824401}
W20251027 21:08:13.086337 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet bd6152f7ac8a4123872098955b8837ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd6152f7ac8a4123872098955b8837ac already in progress: creating tablet
I20251027 21:08:13.087095 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1213) took 1597 ms. Trace:
I20251027 21:08:13.087167 10041 rpcz_store.cc:276] 1027 21:08:11.490083 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:11.490148 (+ 65us) service_pool.cc:225] Handling call
1027 21:08:13.087000 (+1596852us) ts_tablet_manager.cc:1151] Deleting tablet 0d0099e7b3bd403590ca5c9c73198853
1027 21:08:13.087009 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.087080 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1596650}
I20251027 21:08:13.087345 10031 tablet_bootstrap.cc:654] T d672118ecd6f4094855adf106e5f7115 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:13.087538 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 0d0099e7b3bd403590ca5c9c73198853 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d0099e7b3bd403590ca5c9c73198853 already in progress: creating tablet
I20251027 21:08:13.087987 11514 raft_consensus.cc:493] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.088135 11514 raft_consensus.cc:515] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:13.088270 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1214) took 1581 ms. Trace:
I20251027 21:08:13.088330 10041 rpcz_store.cc:276] 1027 21:08:11.506564 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:11.506617 (+ 53us) service_pool.cc:225] Handling call
1027 21:08:13.088177 (+1581560us) ts_tablet_manager.cc:1151] Deleting tablet 191009830c30467b9c5ab39e13b7c267
1027 21:08:13.088186 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.088256 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1581373}
W20251027 21:08:13.088652 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 191009830c30467b9c5ab39e13b7c267 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 191009830c30467b9c5ab39e13b7c267 already in progress: creating tablet
I20251027 21:08:13.088694 11514 leader_election.cc:290] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:13.089407 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1215) took 1577 ms. Trace:
I20251027 21:08:13.089743 10041 rpcz_store.cc:276] 1027 21:08:11.512310 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:11.512375 (+ 65us) service_pool.cc:225] Handling call
1027 21:08:13.089308 (+1576933us) ts_tablet_manager.cc:1151] Deleting tablet 278d8f448ef144148cabfc3a57bd24fd
1027 21:08:13.089318 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.089391 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1576716}
W20251027 21:08:13.090186 9711 leader_election.cc:343] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 9c2d713d72fd4b5ca34cb3e828c75761
W20251027 21:08:13.090303 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 278d8f448ef144148cabfc3a57bd24fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 278d8f448ef144148cabfc3a57bd24fd already in progress: creating tablet
I20251027 21:08:13.090641 10041 tablet_replica.cc:333] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:13.091187 10041 raft_consensus.cc:2243] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:13.091696 10041 raft_consensus.cc:2272] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.092607 10041 ts_tablet_manager.cc:1916] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.039609 10040 ts_tablet_manager.cc:1950] T 631ff72613734913818a3cad24f20dbc P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:13.094265 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 631ff72613734913818a3cad24f20dbc (table default.Table_ed23972576b74b5fbc7167cf7b541bfc [id=e7b7bdb2ea7f468c98bfe88f900a5be8]) successfully deleted
I20251027 21:08:13.089674 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c2d713d72fd4b5ca34cb3e828c75761" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:13.094661 9630 raft_consensus.cc:2393] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77b0513f17c0472088901a1ce8b08218 in current term 1: Already voted for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in this term.
I20251027 21:08:13.095108 9710 leader_election.cc:304] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:13.095326 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_27667baca7d8497a8f70a38f1ed37a30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:13.095520 11514 raft_consensus.cc:3060] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:13.095878 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27667baca7d8497a8f70a38f1ed37a30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:13.094559 9568 leader_election.cc:304] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.098389 9568 leader_election.cc:304] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.098515 11411 raft_consensus.cc:2749] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.098764 11411 raft_consensus.cc:2804] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:13.098865 11411 raft_consensus.cc:697] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:13.099058 11411 consensus_queue.cc:237] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.099665 10041 ts_tablet_manager.cc:1929] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 21:08:13.099752 10041 log.cc:1199] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/90f96c21da9944f1ade05d35787d47e2
I20251027 21:08:13.100101 10041 ts_tablet_manager.cc:1950] T 90f96c21da9944f1ade05d35787d47e2 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:13.100808 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1216) took 1572 ms. Trace:
I20251027 21:08:13.100893 10041 rpcz_store.cc:276] 1027 21:08:11.528812 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:11.528868 (+ 56us) service_pool.cc:225] Handling call
1027 21:08:13.090586 (+1561718us) ts_tablet_manager.cc:1151] Deleting tablet 90f96c21da9944f1ade05d35787d47e2
1027 21:08:13.090595 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.096701 (+ 6106us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.099583 (+ 2882us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.100789 (+ 1206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1561519}
I20251027 21:08:13.101274 11514 raft_consensus.cc:2749] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.101258 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 90f96c21da9944f1ade05d35787d47e2 (table default.Table_cdf514bc48a4478da802d1bbfd0f1c71 [id=ec9aa8523bf44f2f9486b14f4cf7639c]) successfully deleted
I20251027 21:08:13.101382 10029 raft_consensus.cc:359] T 0e4d35f229bc46e1859ded7c8196ab9d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.101527 10029 raft_consensus.cc:385] T 0e4d35f229bc46e1859ded7c8196ab9d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.101543 10041 tablet_replica.cc:333] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:13.101576 10029 raft_consensus.cc:740] T 0e4d35f229bc46e1859ded7c8196ab9d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.101758 10041 raft_consensus.cc:2243] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.101922 10041 raft_consensus.cc:2272] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.102563 10041 ts_tablet_manager.cc:1916] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.105863 10032 tablet_bootstrap.cc:492] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:13.105993 10032 ts_tablet_manager.cc:1403] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.073s user 0.024s sys 0.007s
W20251027 21:08:13.106457 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c76870d7ca1b43308f0ab94d4ab39b5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c76870d7ca1b43308f0ab94d4ab39b5f already in progress: creating tablet
W20251027 21:08:13.106663 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 7062871e266c4603bab851a837ef9aec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7062871e266c4603bab851a837ef9aec already in progress: creating tablet
W20251027 21:08:13.106837 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 4c60797e8cd5442b9eef86ee51d4b9c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c60797e8cd5442b9eef86ee51d4b9c6 already in progress: creating tablet
W20251027 21:08:13.107012 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet fe7b00e9cf984d12817c9ea860391199 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe7b00e9cf984d12817c9ea860391199 already in progress: creating tablet
I20251027 21:08:13.107827 10032 raft_consensus.cc:359] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.108001 10032 raft_consensus.cc:385] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.108059 10032 raft_consensus.cc:740] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.108278 10032 consensus_queue.cc:260] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.108744 10032 ts_tablet_manager.cc:1434] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:08:13.109032 10032 tablet_bootstrap.cc:492] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:13.114248 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0655ada3126c49049155b9671bcbe409"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:13.114789 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0655ada3126c49049155b9671bcbe409 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:13.116315 10032 tablet_bootstrap.cc:654] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.117106 10043 ts_tablet_manager.cc:1929] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:13.117218 10043 log.cc:1199] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/aabaa34787cf4d179822269b7f323314
I20251027 21:08:13.117558 10043 ts_tablet_manager.cc:1950] T aabaa34787cf4d179822269b7f323314 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:13.118393 10029 consensus_queue.cc:260] T 0e4d35f229bc46e1859ded7c8196ab9d P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.118736 11411 raft_consensus.cc:493] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.118881 10029 ts_tablet_manager.cc:1434] T 0e4d35f229bc46e1859ded7c8196ab9d P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.052s user 0.001s sys 0.001s
I20251027 21:08:13.118849 11411 raft_consensus.cc:515] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.119140 10029 tablet_bootstrap.cc:492] T 27bd3b2a011648b2b8f2f92944513751 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:13.119712 11419 raft_consensus.cc:493] T cdbde6272af147829ecadd67d634f049 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251027 21:08:13.098771 9568 leader_election.cc:343] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 22dbfa50996b47269983c107cc399000
I20251027 21:08:13.119817 11419 raft_consensus.cc:515] T cdbde6272af147829ecadd67d634f049 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.119884 9568 leader_election.cc:304] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.120316 11419 leader_election.cc:290] T cdbde6272af147829ecadd67d634f049 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.120440 11419 raft_consensus.cc:2749] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.122236 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cdbde6272af147829ecadd67d634f049" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:13.122799 9566 leader_election.cc:343] T cdbde6272af147829ecadd67d634f049 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.123466 11411 leader_election.cc:290] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.123869 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28a2e85021394f9d8a9ea524959b52a1" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:13.124300 9566 leader_election.cc:343] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.134896 10032 tablet_bootstrap.cc:492] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:13.135206 10032 ts_tablet_manager.cc:1403] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.003s
I20251027 21:08:13.135986 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet fe7b00e9cf984d12817c9ea860391199 with delete_type TABLET_DATA_DELETED (replaced by tablet 023eb8eb693d446caef49e0eaecd147c) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:13.136142 10032 raft_consensus.cc:359] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:13.136341 10032 raft_consensus.cc:385] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.137012 11546 tablet.cc:1722] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251027 21:08:13.136404 10032 raft_consensus.cc:740] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.137795 10029 tablet_bootstrap.cc:654] T 27bd3b2a011648b2b8f2f92944513751 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.137863 10032 consensus_queue.cc:260] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:08:13.138906 9568 leader_election.cc:343] T cdbde6272af147829ecadd67d634f049 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: cdbde6272af147829ecadd67d634f049
I20251027 21:08:13.139014 9568 leader_election.cc:304] T cdbde6272af147829ecadd67d634f049 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.139211 11373 raft_consensus.cc:2749] T cdbde6272af147829ecadd67d634f049 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.145776 11543 raft_consensus.cc:493] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.145920 11543 raft_consensus.cc:515] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:13.146548 11543 leader_election.cc:290] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.147205 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e419e588b31d48c8bc19d7948081cc81" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:13.147436 9310 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_c09cc86fc2a64450af56fc584302da69 [id=d46947a29ddf4d94b5a9da22d9e5383e] alter complete (version 3)
W20251027 21:08:13.147527 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a6edd6a5c4434f0497667b1a2771f885 consensus state has no leader
W20251027 21:08:13.147662 9860 leader_election.cc:343] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.148049 10032 ts_tablet_manager.cc:1434] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251027 21:08:13.148452 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_87ac731a20bf4f6bbce8d82c1ad3bd97" } new_table_name: "default.Table_c93714f3039c415d80e39b7ed977450f" modify_external_catalogs: true
I20251027 21:08:13.149235 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e419e588b31d48c8bc19d7948081cc81" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:13.149652 9859 leader_election.cc:343] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.149739 9859 leader_election.cc:304] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.149883 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_45b899e2d3ab48ec90355341e1648881"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:13.150038 11543 raft_consensus.cc:2749] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:13.150452 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45b899e2d3ab48ec90355341e1648881 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:13.151750 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet fde0980aa64c4496861a662f7ed3ef75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:13.153079 10040 tablet_replica.cc:333] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:13.153245 10040 raft_consensus.cc:2243] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.153384 10040 raft_consensus.cc:2272] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.154062 10040 ts_tablet_manager.cc:1916] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.150813 10032 tablet_bootstrap.cc:492] T 54c2b10e79e149c2a637475e1f518907 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:13.156275 10032 tablet_bootstrap.cc:654] T 54c2b10e79e149c2a637475e1f518907 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.164113 10041 ts_tablet_manager.cc:1929] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:13.164215 10041 log.cc:1199] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/532ea47b15b544dfb05ddb2bdacf3953
I20251027 21:08:13.164522 10041 ts_tablet_manager.cc:1950] T 532ea47b15b544dfb05ddb2bdacf3953 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:13.165226 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1217) took 1636 ms. Trace:
I20251027 21:08:13.165313 10041 rpcz_store.cc:276] 1027 21:08:11.528899 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:11.532952 (+ 4053us) service_pool.cc:225] Handling call
1027 21:08:13.101498 (+1568546us) ts_tablet_manager.cc:1151] Deleting tablet 532ea47b15b544dfb05ddb2bdacf3953
1027 21:08:13.101507 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.122569 (+ 21062us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.164031 (+ 41462us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.165210 (+ 1179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1568368}
I20251027 21:08:13.165705 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1219) took 1587 ms. Trace:
I20251027 21:08:13.165746 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 532ea47b15b544dfb05ddb2bdacf3953 (table default.Table_c9a81e9dabad4a7d9fbd427439a7587c [id=40ce2a0befe54d698759beff3bfdf0f6]) successfully deleted
I20251027 21:08:13.165788 10041 rpcz_store.cc:276] 1027 21:08:11.577830 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:11.577882 (+ 52us) service_pool.cc:225] Handling call
1027 21:08:13.165575 (+1587693us) ts_tablet_manager.cc:1151] Deleting tablet b162734c536a40c08755201a85255f4c
1027 21:08:13.165584 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.165689 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1587502}
W20251027 21:08:13.166411 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet b162734c536a40c08755201a85255f4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b162734c536a40c08755201a85255f4c already in progress: creating tablet
I20251027 21:08:13.167366 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1220) took 1589 ms. Trace:
I20251027 21:08:13.167454 10041 rpcz_store.cc:276] 1027 21:08:11.577932 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:11.578163 (+ 231us) service_pool.cc:225] Handling call
1027 21:08:13.167262 (+1589099us) ts_tablet_manager.cc:1151] Deleting tablet b6c4fc648fa04d9aa4a10b5b6b07cd21
1027 21:08:13.167270 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.167350 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1588911}
W20251027 21:08:13.167840 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet b6c4fc648fa04d9aa4a10b5b6b07cd21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6c4fc648fa04d9aa4a10b5b6b07cd21 already in progress: creating tablet
I20251027 21:08:13.168377 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1221) took 1590 ms. Trace:
I20251027 21:08:13.168792 10041 rpcz_store.cc:276] 1027 21:08:11.578023 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:11.581440 (+ 3417us) service_pool.cc:225] Handling call
1027 21:08:13.168285 (+1586845us) ts_tablet_manager.cc:1151] Deleting tablet e4b84d8c0ee74dcbaee8aeb49aa30c57
1027 21:08:13.168293 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.168362 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1585865}
W20251027 21:08:13.169440 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet e4b84d8c0ee74dcbaee8aeb49aa30c57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4b84d8c0ee74dcbaee8aeb49aa30c57 already in progress: creating tablet
I20251027 21:08:13.169754 10041 tablet_replica.cc:333] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
W20251027 21:08:13.170671 9568 leader_election.cc:343] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 28a2e85021394f9d8a9ea524959b52a1
I20251027 21:08:13.170890 9568 leader_election.cc:304] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.170992 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet c76870d7ca1b43308f0ab94d4ab39b5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:12 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:13.171116 11512 raft_consensus.cc:2749] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.172961 10032 tablet_bootstrap.cc:492] T 54c2b10e79e149c2a637475e1f518907 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:13.173085 10032 ts_tablet_manager.cc:1403] T 54c2b10e79e149c2a637475e1f518907 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.001s
I20251027 21:08:13.173415 11543 raft_consensus.cc:493] T 57ef07c764664deaa4b2e9bcae7e7973 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.173523 11543 raft_consensus.cc:515] T 57ef07c764664deaa4b2e9bcae7e7973 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.173852 10032 raft_consensus.cc:359] T 54c2b10e79e149c2a637475e1f518907 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.173969 10032 raft_consensus.cc:385] T 54c2b10e79e149c2a637475e1f518907 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.174012 10032 raft_consensus.cc:740] T 54c2b10e79e149c2a637475e1f518907 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.174259 10041 raft_consensus.cc:2243] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.174386 10041 raft_consensus.cc:2272] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.174448 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57ef07c764664deaa4b2e9bcae7e7973" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:13.174588 10032 consensus_queue.cc:260] T 54c2b10e79e149c2a637475e1f518907 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.174994 10032 ts_tablet_manager.cc:1434] T 54c2b10e79e149c2a637475e1f518907 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:13.175679 10041 ts_tablet_manager.cc:1916] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:13.176072 9860 leader_election.cc:343] T 57ef07c764664deaa4b2e9bcae7e7973 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.174016 11543 leader_election.cc:290] T 57ef07c764664deaa4b2e9bcae7e7973 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.177367 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57ef07c764664deaa4b2e9bcae7e7973" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:13.177816 9859 leader_election.cc:343] T 57ef07c764664deaa4b2e9bcae7e7973 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.177901 9859 leader_election.cc:304] T 57ef07c764664deaa4b2e9bcae7e7973 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.178283 11543 raft_consensus.cc:2749] T 57ef07c764664deaa4b2e9bcae7e7973 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.175215 10032 tablet_bootstrap.cc:492] T 93bebcb636ec466b81453d90aa1bbd0f P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:13.183102 10032 tablet_bootstrap.cc:654] T 93bebcb636ec466b81453d90aa1bbd0f P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.184808 10041 ts_tablet_manager.cc:1929] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:13.184917 10041 log.cc:1199] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/e04f4c5cf724419ab0be0e84d30a9a57
I20251027 21:08:13.185185 10041 ts_tablet_manager.cc:1950] T e04f4c5cf724419ab0be0e84d30a9a57 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:13.185881 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1224) took 1545 ms. Trace:
I20251027 21:08:13.185983 10041 rpcz_store.cc:276] 1027 21:08:11.640340 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:11.640514 (+ 174us) service_pool.cc:225] Handling call
1027 21:08:13.169715 (+1529201us) ts_tablet_manager.cc:1151] Deleting tablet e04f4c5cf724419ab0be0e84d30a9a57
1027 21:08:13.169723 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.178714 (+ 8991us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.184725 (+ 6011us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.185863 (+ 1138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3775,"tablet-delete.queue_time_us":1529021}
I20251027 21:08:13.186398 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet e04f4c5cf724419ab0be0e84d30a9a57 (table default.Table_5315576a7f87464dbbfded0081ddef2d [id=ca2d675d800a46728d7b11ee50f2e056]) successfully deleted
W20251027 21:08:13.186920 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c34e487f57bd41bca440b0ad0179a075 consensus state has no leader
I20251027 21:08:13.188390 10041 tablet_replica.cc:333] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:13.188755 10041 raft_consensus.cc:2243] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.189070 10041 raft_consensus.cc:2272] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.189653 10041 ts_tablet_manager.cc:1916] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.196623 10032 tablet_bootstrap.cc:492] T 93bebcb636ec466b81453d90aa1bbd0f P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:13.196795 10032 ts_tablet_manager.cc:1403] T 93bebcb636ec466b81453d90aa1bbd0f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.001s
I20251027 21:08:13.197503 10032 raft_consensus.cc:359] T 93bebcb636ec466b81453d90aa1bbd0f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.197707 10032 raft_consensus.cc:385] T 93bebcb636ec466b81453d90aa1bbd0f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.197804 10032 raft_consensus.cc:740] T 93bebcb636ec466b81453d90aa1bbd0f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.198041 10032 consensus_queue.cc:260] T 93bebcb636ec466b81453d90aa1bbd0f P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.198580 10032 ts_tablet_manager.cc:1434] T 93bebcb636ec466b81453d90aa1bbd0f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:13.198853 10032 tablet_bootstrap.cc:492] T f19b5e8e460a4c4ba7399b981c603b84 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:13.199100 11543 raft_consensus.cc:493] T 93bebcb636ec466b81453d90aa1bbd0f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.199191 11543 raft_consensus.cc:515] T 93bebcb636ec466b81453d90aa1bbd0f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.199661 11543 leader_election.cc:290] T 93bebcb636ec466b81453d90aa1bbd0f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.200549 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93bebcb636ec466b81453d90aa1bbd0f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:13.201138 9860 leader_election.cc:343] T 93bebcb636ec466b81453d90aa1bbd0f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.201431 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93bebcb636ec466b81453d90aa1bbd0f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:13.201910 9859 leader_election.cc:343] T 93bebcb636ec466b81453d90aa1bbd0f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.201997 9859 leader_election.cc:304] T 93bebcb636ec466b81453d90aa1bbd0f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.202216 11543 raft_consensus.cc:2749] T 93bebcb636ec466b81453d90aa1bbd0f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.203675 10041 ts_tablet_manager.cc:1929] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:13.203773 10041 log.cc:1199] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/e9b8f76c33444bcb9b488af65e4d3e6b
I20251027 21:08:13.204034 10041 ts_tablet_manager.cc:1950] T e9b8f76c33444bcb9b488af65e4d3e6b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
W20251027 21:08:13.204587 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1cbec3625e5f47448e944619db713c87 consensus state has no leader
I20251027 21:08:13.204680 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1225) took 1564 ms. Trace:
I20251027 21:08:13.201184 10032 tablet_bootstrap.cc:654] T f19b5e8e460a4c4ba7399b981c603b84 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.207710 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1098) took 4591 ms. Trace:
I20251027 21:08:13.207878 10043 rpcz_store.cc:276] 1027 21:08:08.616045 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.616083 (+ 38us) service_pool.cc:225] Handling call
1027 21:08:12.910547 (+4294464us) ts_tablet_manager.cc:1151] Deleting tablet aabaa34787cf4d179822269b7f323314
1027 21:08:12.910555 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:12.934099 (+ 23544us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.116993 (+182894us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.207667 (+ 90674us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4294349}
I20251027 21:08:13.208345 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet aabaa34787cf4d179822269b7f323314 (table default.Table_0eecf2c874144e7594c825f01fcc5770 [id=a0a10aefbf4c4ab083982e9e46bba513]) successfully deleted
I20251027 21:08:13.208643 10043 tablet_replica.cc:333] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:13.208835 10031 tablet_bootstrap.cc:492] T d672118ecd6f4094855adf106e5f7115 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:13.208846 10043 raft_consensus.cc:2243] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:13.209051 10031 ts_tablet_manager.cc:1403] T d672118ecd6f4094855adf106e5f7115 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.131s user 0.006s sys 0.000s
I20251027 21:08:13.209106 10043 pending_rounds.cc:70] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14: Trying to abort 1 pending ops.
I20251027 21:08:13.209195 10043 pending_rounds.cc:77] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510684689440768 op_type: NO_OP noop_request { }
I20251027 21:08:13.209321 10043 raft_consensus.cc:2889] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:13.209548 10043 raft_consensus.cc:2272] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.209806 10031 raft_consensus.cc:359] T d672118ecd6f4094855adf106e5f7115 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:13.210026 10041 rpcz_store.cc:276] 1027 21:08:11.640467 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:11.640729 (+ 262us) service_pool.cc:225] Handling call
1027 21:08:13.188349 (+1547620us) ts_tablet_manager.cc:1151] Deleting tablet e9b8f76c33444bcb9b488af65e4d3e6b
1027 21:08:13.188357 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.194412 (+ 6055us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.203583 (+ 9171us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.204663 (+ 1080us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1547473}
I20251027 21:08:13.210425 10043 ts_tablet_manager.cc:1916] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.210510 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet e9b8f76c33444bcb9b488af65e4d3e6b (table default.Table_3d031fe204804a95b5bc07e8b54573d8 [id=855df1b953454ddeac8efc6e5869db2b]) successfully deleted
I20251027 21:08:13.210820 10041 tablet_replica.cc:333] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:13.211036 10041 raft_consensus.cc:2243] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.211205 10029 tablet_bootstrap.cc:492] T 27bd3b2a011648b2b8f2f92944513751 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:13.211215 10041 raft_consensus.cc:2272] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.211556 10031 raft_consensus.cc:385] T d672118ecd6f4094855adf106e5f7115 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.211622 10029 ts_tablet_manager.cc:1403] T 27bd3b2a011648b2b8f2f92944513751 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.000s
I20251027 21:08:13.211701 10031 raft_consensus.cc:740] T d672118ecd6f4094855adf106e5f7115 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.211969 10031 consensus_queue.cc:260] T d672118ecd6f4094855adf106e5f7115 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:13.212322 10029 raft_consensus.cc:359] T 27bd3b2a011648b2b8f2f92944513751 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.212527 10029 raft_consensus.cc:385] T 27bd3b2a011648b2b8f2f92944513751 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.212636 10029 raft_consensus.cc:740] T 27bd3b2a011648b2b8f2f92944513751 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.212935 10040 ts_tablet_manager.cc:1929] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:13.213030 10040 log.cc:1199] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/fde0980aa64c4496861a662f7ed3ef75
I20251027 21:08:13.213056 10029 consensus_queue.cc:260] T 27bd3b2a011648b2b8f2f92944513751 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.213316 10040 ts_tablet_manager.cc:1950] T fde0980aa64c4496861a662f7ed3ef75 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:13.213840 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet fde0980aa64c4496861a662f7ed3ef75 (table default.Table_096b7cf08dbf49279d8d5c11b3b687ab [id=20b788f5a2d14b31a58e1b29fa954951]) successfully deleted
I20251027 21:08:13.214188 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_db03a0faacfc4ecfad89dbf48def9140" } new_table_name: "default.Table_b857920bfc22497b8869a10d0702f52c" modify_external_catalogs: true
I20251027 21:08:13.214339 10029 ts_tablet_manager.cc:1434] T 27bd3b2a011648b2b8f2f92944513751 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251027 21:08:13.214617 10029 tablet_bootstrap.cc:492] T e6ed9d7a700446edafcd0daa02aa7ef6 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:13.216423 10032 tablet_bootstrap.cc:492] T f19b5e8e460a4c4ba7399b981c603b84 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:13.216687 10032 ts_tablet_manager.cc:1403] T f19b5e8e460a4c4ba7399b981c603b84 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20251027 21:08:13.217379 10032 raft_consensus.cc:359] T f19b5e8e460a4c4ba7399b981c603b84 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.217573 10032 raft_consensus.cc:385] T f19b5e8e460a4c4ba7399b981c603b84 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.217649 10032 raft_consensus.cc:740] T f19b5e8e460a4c4ba7399b981c603b84 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.217875 10032 consensus_queue.cc:260] T f19b5e8e460a4c4ba7399b981c603b84 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.218397 10032 ts_tablet_manager.cc:1434] T f19b5e8e460a4c4ba7399b981c603b84 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:13.218701 10032 tablet_bootstrap.cc:492] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:13.220615 10032 tablet_bootstrap.cc:654] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.222108 10031 ts_tablet_manager.cc:1434] T d672118ecd6f4094855adf106e5f7115 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20251027 21:08:13.222544 10041 ts_tablet_manager.cc:1916] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.226854 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9a1b725b46c94ae7a2831ad1e047cb39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:13.227391 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a1b725b46c94ae7a2831ad1e047cb39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:13.230778 10029 tablet_bootstrap.cc:654] T e6ed9d7a700446edafcd0daa02aa7ef6 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.232898 10031 tablet_bootstrap.cc:492] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:13.235141 9305 master_service.cc:954] ReplaceTablet: received request to replace tablet c3f4e692c08244afa2c79e0a075bbcdb from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:13.235178 11373 raft_consensus.cc:493] T c9c86253ce914916bee6584675a82067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.235294 11373 raft_consensus.cc:515] T c9c86253ce914916bee6584675a82067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.235553 10031 tablet_bootstrap.cc:654] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.235889 11373 leader_election.cc:290] T c9c86253ce914916bee6584675a82067 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.236420 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9c86253ce914916bee6584675a82067" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:13.236964 9568 leader_election.cc:343] T c9c86253ce914916bee6584675a82067 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: c9c86253ce914916bee6584675a82067
W20251027 21:08:13.237419 9566 leader_election.cc:343] T c9c86253ce914916bee6584675a82067 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.237519 9566 leader_election.cc:304] T c9c86253ce914916bee6584675a82067 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.237897 11373 raft_consensus.cc:2749] T c9c86253ce914916bee6584675a82067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.241396 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_47e6de266b544e31a4d781f8c9377717" } modify_external_catalogs: true
I20251027 21:08:13.241583 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_47e6de266b544e31a4d781f8c9377717" } modify_external_catalogs: true
I20251027 21:08:13.242861 10041 ts_tablet_manager.cc:1929] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:13.243002 10041 log.cc:1199] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/c27c699cc2744f40811f7b9395ca76be
I20251027 21:08:13.243227 10032 tablet_bootstrap.cc:492] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:13.243317 10041 ts_tablet_manager.cc:1950] T c27c699cc2744f40811f7b9395ca76be P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:13.243364 10032 ts_tablet_manager.cc:1403] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.001s
I20251027 21:08:13.244123 10032 raft_consensus.cc:359] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.244284 10032 raft_consensus.cc:385] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.244369 10032 raft_consensus.cc:740] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.244562 10032 consensus_queue.cc:260] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.250430 10032 ts_tablet_manager.cc:1434] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251027 21:08:13.250797 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_06ef936a484b4bfe88cee764e5aeb190" } modify_external_catalogs: true
I20251027 21:08:13.250931 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_06ef936a484b4bfe88cee764e5aeb190" } modify_external_catalogs: true
I20251027 21:08:13.252288 10032 tablet_bootstrap.cc:492] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:13.259160 10032 tablet_bootstrap.cc:654] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.260582 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1228) took 1579 ms. Trace:
I20251027 21:08:13.260697 10041 rpcz_store.cc:276] 1027 21:08:11.681161 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:11.681215 (+ 54us) service_pool.cc:225] Handling call
1027 21:08:13.210761 (+1529546us) ts_tablet_manager.cc:1151] Deleting tablet c27c699cc2744f40811f7b9395ca76be
1027 21:08:13.210771 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.239942 (+ 29171us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.242755 (+ 2813us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.260525 (+ 17770us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1529310}
I20251027 21:08:13.261019 10041 tablet_replica.cc:333] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:13.261197 10041 raft_consensus.cc:2243] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.261272 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet c27c699cc2744f40811f7b9395ca76be (table default.Table_691240a058cc4f879e5737f73a4c12e7 [id=697732b24e12465d8caac4e23de633bf]) successfully deleted
I20251027 21:08:13.261353 10041 raft_consensus.cc:2272] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.261802 10041 ts_tablet_manager.cc:1916] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.263494 10032 tablet_bootstrap.cc:492] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:13.263664 10032 ts_tablet_manager.cc:1403] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20251027 21:08:13.264420 10032 raft_consensus.cc:359] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.264614 10032 raft_consensus.cc:385] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.264694 10032 raft_consensus.cc:740] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.264942 10032 consensus_queue.cc:260] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.265448 10032 ts_tablet_manager.cc:1434] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:13.265765 10032 tablet_bootstrap.cc:492] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:13.266165 11543 raft_consensus.cc:493] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.266366 11543 raft_consensus.cc:515] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.266872 11543 leader_election.cc:290] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.267393 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46185d382f0a4dea803aad692ed3c89a" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:13.267912 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46185d382f0a4dea803aad692ed3c89a" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:13.268074 9859 leader_election.cc:343] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:13.268684 9860 leader_election.cc:343] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.268817 9860 leader_election.cc:304] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.269485 11543 raft_consensus.cc:2749] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.277473 11514 raft_consensus.cc:493] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.277626 11514 raft_consensus.cc:515] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:13.278283 11514 leader_election.cc:290] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
W20251027 21:08:13.278951 9711 leader_election.cc:343] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 83f5fbf5757a49df93ab949b7657c61f
W20251027 21:08:13.279178 9710 leader_election.cc:343] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 83f5fbf5757a49df93ab949b7657c61f
I20251027 21:08:13.279271 9710 leader_election.cc:304] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:13.280009 11514 raft_consensus.cc:2749] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.280045 10041 ts_tablet_manager.cc:1929] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:13.280009 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c6f57d0c3113494cb54754d67cfefa71" } modify_external_catalogs: true
I20251027 21:08:13.280148 10041 log.cc:1199] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/133e474f9133487f89c44dbfa6266ca7
I20251027 21:08:13.280164 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c6f57d0c3113494cb54754d67cfefa71" } modify_external_catalogs: true
I20251027 21:08:13.282197 10041 ts_tablet_manager.cc:1950] T 133e474f9133487f89c44dbfa6266ca7 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:13.286213 11543 raft_consensus.cc:493] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.286501 11543 raft_consensus.cc:515] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.287096 11543 leader_election.cc:290] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.287500 10032 tablet_bootstrap.cc:654] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.287838 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93a3b22cdc1543aebcc7fe72f2f2c0ae" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:13.288400 9859 leader_election.cc:343] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.291296 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93a3b22cdc1543aebcc7fe72f2f2c0ae" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:13.292068 9860 leader_election.cc:343] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.292188 9860 leader_election.cc:304] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.292564 11543 raft_consensus.cc:2749] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.298985 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_305463d05bd84f69883cb6165ec89446"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:13.299678 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_305463d05bd84f69883cb6165ec89446 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:13.309062 11543 raft_consensus.cc:493] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.309226 11543 raft_consensus.cc:515] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.309770 11543 leader_election.cc:290] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.310586 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d2c40f983e743b0b53f987b6a7ba437" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:13.310621 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d2c40f983e743b0b53f987b6a7ba437" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:13.311219 9860 leader_election.cc:343] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:13.311513 9859 leader_election.cc:343] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.311655 9859 leader_election.cc:304] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.312083 11543 raft_consensus.cc:2749] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.314200 10031 tablet_bootstrap.cc:492] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:13.315516 10031 ts_tablet_manager.cc:1403] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.090s user 0.019s sys 0.000s
I20251027 21:08:13.316058 10043 ts_tablet_manager.cc:1929] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:13.316844 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5dee7205c58b4b70beadf92f948aea58" } modify_external_catalogs: true
I20251027 21:08:13.317005 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5dee7205c58b4b70beadf92f948aea58" } modify_external_catalogs: true
I20251027 21:08:13.317482 10031 raft_consensus.cc:359] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:13.317642 10031 raft_consensus.cc:385] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.317694 10031 raft_consensus.cc:740] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.318248 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1232) took 1539 ms. Trace:
I20251027 21:08:13.318353 10041 rpcz_store.cc:276] 1027 21:08:11.779176 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:11.786925 (+ 7749us) service_pool.cc:225] Handling call
1027 21:08:13.260938 (+1474013us) ts_tablet_manager.cc:1151] Deleting tablet 133e474f9133487f89c44dbfa6266ca7
1027 21:08:13.260947 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.270485 (+ 9538us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.279911 (+ 9426us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.318196 (+ 38285us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1473841}
I20251027 21:08:13.318876 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 133e474f9133487f89c44dbfa6266ca7 (table default.Table_3ef4dcb270d64cbbb4ad719b91ab1f10 [id=3b80e606aa374fd9ad1e7b0c5c20bb6a]) successfully deleted
I20251027 21:08:13.318957 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1236) took 1526 ms. Trace:
I20251027 21:08:13.319077 10041 rpcz_store.cc:276] 1027 21:08:11.792084 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:11.792508 (+ 424us) service_pool.cc:225] Handling call
1027 21:08:13.318828 (+1526320us) ts_tablet_manager.cc:1151] Deleting tablet 138a5d18566240858b5c92c3c03b8087
1027 21:08:13.318841 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.318940 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1526141}
W20251027 21:08:13.319434 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 138a5d18566240858b5c92c3c03b8087 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 138a5d18566240858b5c92c3c03b8087 already in progress: creating tablet
I20251027 21:08:13.319509 10041 tablet_replica.cc:333] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:13.319552 10031 consensus_queue.cc:260] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:13.319715 10041 raft_consensus.cc:2243] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.319891 10041 raft_consensus.cc:2272] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.320672 10041 ts_tablet_manager.cc:1916] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.323936 11543 raft_consensus.cc:493] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.324106 11543 raft_consensus.cc:515] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.324687 11543 leader_election.cc:290] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.325548 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e477ca4b69c84327b04618d90406e19a" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:13.326076 9859 leader_election.cc:343] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.328742 10041 ts_tablet_manager.cc:1929] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:13.328873 10041 log.cc:1199] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/a7d8d060916941a5b7e944e8b08a2882
I20251027 21:08:13.329192 10041 ts_tablet_manager.cc:1950] T a7d8d060916941a5b7e944e8b08a2882 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:13.329927 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1239) took 1527 ms. Trace:
I20251027 21:08:13.330410 10031 ts_tablet_manager.cc:1434] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251027 21:08:13.334717 10031 tablet_bootstrap.cc:492] T 9fa39697360c46138ef4cd21e8c7a950 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:13.337313 10031 tablet_bootstrap.cc:654] T 9fa39697360c46138ef4cd21e8c7a950 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:13.338065 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 016e1aa5266d44b5b35a3f47e8f11a46 consensus state has no leader
I20251027 21:08:13.347733 10031 tablet_bootstrap.cc:492] T 9fa39697360c46138ef4cd21e8c7a950 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:13.347862 10031 ts_tablet_manager.cc:1403] T 9fa39697360c46138ef4cd21e8c7a950 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.001s
I20251027 21:08:13.348533 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3b8f6db604d34f05b8dde4b29bd4cd3f" } modify_external_catalogs: true
I20251027 21:08:13.348575 10031 raft_consensus.cc:359] T 9fa39697360c46138ef4cd21e8c7a950 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:13.348690 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3b8f6db604d34f05b8dde4b29bd4cd3f" } modify_external_catalogs: true
I20251027 21:08:13.348730 10031 raft_consensus.cc:385] T 9fa39697360c46138ef4cd21e8c7a950 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.348776 10031 raft_consensus.cc:740] T 9fa39697360c46138ef4cd21e8c7a950 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.351648 10031 consensus_queue.cc:260] T 9fa39697360c46138ef4cd21e8c7a950 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:13.352690 9775 raft_consensus.cc:3060] T d672118ecd6f4094855adf106e5f7115 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:13.356415 10031 ts_tablet_manager.cc:1434] T 9fa39697360c46138ef4cd21e8c7a950 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251027 21:08:13.359680 11514 raft_consensus.cc:493] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.359833 11514 raft_consensus.cc:515] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:13.360724 11546 tablet.cc:1722] T 9fde4fd91943498d8888d1783005e492 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251027 21:08:13.361052 9711 leader_election.cc:343] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 9861a0efcba64141aae450663c5d83de
I20251027 21:08:13.360338 11514 leader_election.cc:290] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:13.360280 10031 tablet_bootstrap.cc:492] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:08:13.362771 9710 leader_election.cc:343] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 9861a0efcba64141aae450663c5d83de
I20251027 21:08:13.362879 9710 leader_election.cc:304] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:13.363291 11543 raft_consensus.cc:493] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.363418 11543 raft_consensus.cc:515] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.363807 11514 raft_consensus.cc:2749] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.363911 11543 leader_election.cc:290] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.364564 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76bee322d0b94abe9166170733608853" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:13.365078 9859 leader_election.cc:343] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.367648 10031 tablet_bootstrap.cc:654] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.373389 11521 tablet.cc:1722] T 9fde4fd91943498d8888d1783005e492 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251027 21:08:13.383055 9297 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 016e1aa5266d44b5b35a3f47e8f11a46 (table default.Table_b857920bfc22497b8869a10d0702f52c [id=409b668e9fed4a748a4cb43c5c69551a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 016e1aa5266d44b5b35a3f47e8f11a46 consensus state has no leader
I20251027 21:08:13.393128 9304 master_service.cc:954] ReplaceTablet: received request to replace tablet aaaf70c950044387a63820079db3c829 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:13.330039 10041 rpcz_store.cc:276] 1027 21:08:11.802721 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:11.802841 (+ 120us) service_pool.cc:225] Handling call
1027 21:08:13.319456 (+1516615us) ts_tablet_manager.cc:1151] Deleting tablet a7d8d060916941a5b7e944e8b08a2882
1027 21:08:13.319464 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.324056 (+ 4592us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.328636 (+ 4580us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.329865 (+ 1229us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1516439}
I20251027 21:08:13.399567 10041 tablet_replica.cc:333] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:13.399673 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet a7d8d060916941a5b7e944e8b08a2882 (table default.Table_a4ae12c512d645e58eadc2daa1bb7b0a [id=e1f4a77dfc0a497080c3c0ad4fe2e370]) successfully deleted
I20251027 21:08:13.399771 10041 raft_consensus.cc:2243] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.399940 10041 raft_consensus.cc:2272] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.400593 10041 ts_tablet_manager.cc:1916] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.325290 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e477ca4b69c84327b04618d90406e19a" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:13.342885 10029 tablet_bootstrap.cc:492] T e6ed9d7a700446edafcd0daa02aa7ef6 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:13.407186 10029 ts_tablet_manager.cc:1403] T e6ed9d7a700446edafcd0daa02aa7ef6 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.193s user 0.014s sys 0.019s
W20251027 21:08:13.407488 9860 leader_election.cc:343] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.407593 9860 leader_election.cc:304] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.407948 11543 raft_consensus.cc:2749] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.407943 10029 raft_consensus.cc:359] T e6ed9d7a700446edafcd0daa02aa7ef6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.408115 10029 raft_consensus.cc:385] T e6ed9d7a700446edafcd0daa02aa7ef6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.408192 10029 raft_consensus.cc:740] T e6ed9d7a700446edafcd0daa02aa7ef6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.408317 10043 log.cc:1199] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/fde0980aa64c4496861a662f7ed3ef75
I20251027 21:08:13.408408 10029 consensus_queue.cc:260] T e6ed9d7a700446edafcd0daa02aa7ef6 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.358954 11519 tablet.cc:1722] T 9fde4fd91943498d8888d1783005e492 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:13.408908 10029 ts_tablet_manager.cc:1434] T e6ed9d7a700446edafcd0daa02aa7ef6 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:13.409214 10029 tablet_bootstrap.cc:492] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:13.364714 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76bee322d0b94abe9166170733608853" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:13.410959 10031 tablet_bootstrap.cc:492] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:13.411088 10031 ts_tablet_manager.cc:1403] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.051s user 0.013s sys 0.021s
I20251027 21:08:13.411177 10029 tablet_bootstrap.cc:654] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.411756 10031 raft_consensus.cc:359] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.411903 10031 raft_consensus.cc:385] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.411947 10031 raft_consensus.cc:740] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.412163 10031 consensus_queue.cc:260] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.412699 10031 ts_tablet_manager.cc:1434] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:13.412910 10031 tablet_bootstrap.cc:492] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:13.414363 10031 tablet_bootstrap.cc:654] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:13.414310 9860 leader_election.cc:343] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.414435 9860 leader_election.cc:304] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.414953 11543 raft_consensus.cc:2749] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.422264 10031 tablet_bootstrap.cc:492] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:13.422392 10031 ts_tablet_manager.cc:1403] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20251027 21:08:13.423055 10031 raft_consensus.cc:359] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.423239 10031 raft_consensus.cc:385] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.423283 10031 raft_consensus.cc:740] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.423496 10031 consensus_queue.cc:260] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.423575 10032 tablet_bootstrap.cc:492] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:13.423768 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_8a73cb7d9a4849a4aae95965ddf1e738"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:13.423970 10032 ts_tablet_manager.cc:1403] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.158s user 0.013s sys 0.029s
I20251027 21:08:13.424021 10031 ts_tablet_manager.cc:1434] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:13.424248 10031 tablet_bootstrap.cc:492] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:08:13.424324 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a73cb7d9a4849a4aae95965ddf1e738 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:13.424649 10032 raft_consensus.cc:359] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.424795 10032 raft_consensus.cc:385] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.424847 10032 raft_consensus.cc:740] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.425017 10032 consensus_queue.cc:260] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.425669 10031 tablet_bootstrap.cc:654] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.429000 10031 tablet_bootstrap.cc:492] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:13.429105 10031 ts_tablet_manager.cc:1403] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251027 21:08:13.429280 10032 ts_tablet_manager.cc:1434] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251027 21:08:13.429728 10031 raft_consensus.cc:359] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.429870 10031 raft_consensus.cc:385] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.429914 10031 raft_consensus.cc:740] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.430099 10031 consensus_queue.cc:260] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.430568 10032 tablet_bootstrap.cc:492] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:13.430611 10031 ts_tablet_manager.cc:1434] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251027 21:08:13.430831 10031 tablet_bootstrap.cc:492] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:13.431838 10032 tablet_bootstrap.cc:654] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.432102 10031 tablet_bootstrap.cc:654] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.436316 10031 tablet_bootstrap.cc:492] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:13.436439 10031 ts_tablet_manager.cc:1403] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20251027 21:08:13.437060 10031 raft_consensus.cc:359] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.437219 10031 raft_consensus.cc:385] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.437268 10031 raft_consensus.cc:740] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.437479 10031 consensus_queue.cc:260] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.438076 10031 ts_tablet_manager.cc:1434] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:13.438336 10031 tablet_bootstrap.cc:492] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:13.439802 10031 tablet_bootstrap.cc:654] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.408908 10043 ts_tablet_manager.cc:1950] T fde0980aa64c4496861a662f7ed3ef75 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:13.441154 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1099) took 4824 ms. Trace:
I20251027 21:08:13.441283 10043 rpcz_store.cc:276] 1027 21:08:08.616290 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.616326 (+ 36us) service_pool.cc:225] Handling call
1027 21:08:13.208587 (+4592261us) ts_tablet_manager.cc:1151] Deleting tablet fde0980aa64c4496861a662f7ed3ef75
1027 21:08:13.208597 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.312760 (+104163us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.315952 (+ 3192us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.441110 (+125158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4592161}
I20251027 21:08:13.441779 10043 tablet_replica.cc:333] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:13.442094 10043 raft_consensus.cc:2243] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:13.444851 11512 raft_consensus.cc:493] T 0e4d35f229bc46e1859ded7c8196ab9d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.444965 11512 raft_consensus.cc:515] T 0e4d35f229bc46e1859ded7c8196ab9d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.445526 11512 leader_election.cc:290] T 0e4d35f229bc46e1859ded7c8196ab9d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.445727 11521 tablet.cc:1722] T d672118ecd6f4094855adf106e5f7115 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:13.446328 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e4d35f229bc46e1859ded7c8196ab9d" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:13.446786 9568 leader_election.cc:343] T 0e4d35f229bc46e1859ded7c8196ab9d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 0e4d35f229bc46e1859ded7c8196ab9d
W20251027 21:08:13.448482 9566 leader_election.cc:343] T 0e4d35f229bc46e1859ded7c8196ab9d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.448603 9566 leader_election.cc:304] T 0e4d35f229bc46e1859ded7c8196ab9d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.448932 11512 raft_consensus.cc:2749] T 0e4d35f229bc46e1859ded7c8196ab9d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.450865 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_8496081464c049219f4c7e4de97d11c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:13.451464 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8496081464c049219f4c7e4de97d11c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:13.469031 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7be07f1db7b548f3b1fb67fd289088f5" } new_table_name: "default.Table_21ee713fbe10497bbeb1a43df84d1774" modify_external_catalogs: true
I20251027 21:08:13.476017 10041 ts_tablet_manager.cc:1929] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:13.476171 10041 log.cc:1199] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/61ba8d7df6fd45abb6f5231ee8c4ec38
I20251027 21:08:13.476558 10041 ts_tablet_manager.cc:1950] T 61ba8d7df6fd45abb6f5231ee8c4ec38 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
W20251027 21:08:13.477995 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2741d50c7654407194488e14b55460ad consensus state has no leader
W20251027 21:08:13.479912 9298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.125:46313, user_credentials={real_user=slave}} blocked reactor thread for 80859.4us
I20251027 21:08:13.481226 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet fde0980aa64c4496861a662f7ed3ef75 (table default.Table_096b7cf08dbf49279d8d5c11b3b687ab [id=20b788f5a2d14b31a58e1b29fa954951]) successfully deleted
I20251027 21:08:13.442806 10043 raft_consensus.cc:2272] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.482275 10043 ts_tablet_manager.cc:1916] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.492259 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_bd8533529e7a49a681bbc785aa61c597"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:13.492939 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd8533529e7a49a681bbc785aa61c597 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:13.494619 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1238) took 1692 ms. Trace:
I20251027 21:08:13.494737 10041 rpcz_store.cc:276] 1027 21:08:11.802574 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:11.806196 (+ 3622us) service_pool.cc:225] Handling call
1027 21:08:13.399517 (+1593321us) ts_tablet_manager.cc:1151] Deleting tablet 61ba8d7df6fd45abb6f5231ee8c4ec38
1027 21:08:13.399526 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.437249 (+ 37723us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.475882 (+ 38633us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.494592 (+ 18710us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1593011}
I20251027 21:08:13.495195 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 61ba8d7df6fd45abb6f5231ee8c4ec38 (table default.Table_3f462ba7e3ab4705a1daa311661c4f3d [id=39097e9cb16c449682ec4fa71836e4ec]) successfully deleted
I20251027 21:08:13.498379 10041 tablet_replica.cc:333] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:13.498633 10041 raft_consensus.cc:2243] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.498801 10041 raft_consensus.cc:2272] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.498880 10032 tablet_bootstrap.cc:492] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:13.499020 10032 ts_tablet_manager.cc:1403] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.069s user 0.014s sys 0.009s
I20251027 21:08:13.499436 10041 ts_tablet_manager.cc:1916] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.499657 10032 raft_consensus.cc:359] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.499820 10032 raft_consensus.cc:385] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.499884 10032 raft_consensus.cc:740] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.500120 10032 consensus_queue.cc:260] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.503448 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_318d41ccfb2c4d2c99bafcea875143a1" } new_table_name: "default.Table_df13ae1f688a44f8be79d5db36fbd976" modify_external_catalogs: true
I20251027 21:08:13.507294 10032 ts_tablet_manager.cc:1434] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251027 21:08:13.507424 10029 tablet_bootstrap.cc:492] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:13.507601 10029 ts_tablet_manager.cc:1403] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.098s user 0.002s sys 0.006s
I20251027 21:08:13.509423 10029 raft_consensus.cc:359] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.510210 10029 raft_consensus.cc:385] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.510314 10029 raft_consensus.cc:740] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.510555 10029 consensus_queue.cc:260] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.511539 10029 ts_tablet_manager.cc:1434] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251027 21:08:13.511857 10029 tablet_bootstrap.cc:492] T d2203ee62f7e404c95c5f86b5ae85feb P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:13.513860 10032 tablet_bootstrap.cc:492] T e3654aa289154b3ea613b6ff94cbd57e P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:13.514412 10029 tablet_bootstrap.cc:654] T d2203ee62f7e404c95c5f86b5ae85feb P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.515237 10032 tablet_bootstrap.cc:654] T e3654aa289154b3ea613b6ff94cbd57e P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.521108 11512 raft_consensus.cc:493] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.521227 11512 raft_consensus.cc:515] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.521730 11512 leader_election.cc:290] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.521752 10031 tablet_bootstrap.cc:492] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:13.521878 10031 ts_tablet_manager.cc:1403] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.084s user 0.046s sys 0.034s
I20251027 21:08:13.522658 10031 raft_consensus.cc:359] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:13.522760 9568 leader_election.cc:343] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: b3abc5f78c4e41e29128e2d790319c75
I20251027 21:08:13.522826 10031 raft_consensus.cc:385] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.522876 10031 raft_consensus.cc:740] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.522900 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_554983eff7d04fefb9f5f9a7bf6ccc55" } new_table_name: "default.Table_0ac970d5cfbe495782ae79db9675aad0" modify_external_catalogs: true
I20251027 21:08:13.523077 10031 consensus_queue.cc:260] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.523504 10031 ts_tablet_manager.cc:1434] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:13.523782 10031 tablet_bootstrap.cc:492] T 882c58a45d46402eabe96591a7c1de71 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:13.525318 10031 tablet_bootstrap.cc:654] T 882c58a45d46402eabe96591a7c1de71 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.527797 10041 ts_tablet_manager.cc:1929] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:13.527935 10041 log.cc:1199] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/8ae222307a374a91916f560a0fbe8d8f
I20251027 21:08:13.528297 10041 ts_tablet_manager.cc:1950] T 8ae222307a374a91916f560a0fbe8d8f P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:13.529047 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3abc5f78c4e41e29128e2d790319c75" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:13.529186 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1240) took 1704 ms. Trace:
I20251027 21:08:13.529297 10041 rpcz_store.cc:276] 1027 21:08:11.824397 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:11.825170 (+ 773us) service_pool.cc:225] Handling call
1027 21:08:13.498300 (+1673130us) ts_tablet_manager.cc:1151] Deleting tablet 8ae222307a374a91916f560a0fbe8d8f
1027 21:08:13.498315 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.524101 (+ 25786us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.527678 (+ 3577us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.529161 (+ 1483us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1672946}
W20251027 21:08:13.529561 9566 leader_election.cc:343] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.529671 9566 leader_election.cc:304] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.529723 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 8ae222307a374a91916f560a0fbe8d8f (table default.Table_8ec40deea5fd4803a9ee3f6c295a7439 [id=d89790f6beb646f8b2580feeddea18f8]) successfully deleted
I20251027 21:08:13.530154 11512 raft_consensus.cc:2749] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251027 21:08:13.535733 11514 raft_consensus.cc:493] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.535856 11514 raft_consensus.cc:515] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.536391 11514 leader_election.cc:290] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:13.537213 9711 leader_election.cc:343] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: f6789dc2f7cb4a3c826ac507d5c9a239
I20251027 21:08:13.538548 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f6789dc2f7cb4a3c826ac507d5c9a239" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:13.539068 9710 leader_election.cc:304] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:13.539291 11514 raft_consensus.cc:2749] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.549448 10041 tablet_replica.cc:333] T f19b5e8e460a4c4ba7399b981c603b84 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:13.549724 10041 raft_consensus.cc:2243] T f19b5e8e460a4c4ba7399b981c603b84 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.549918 10041 raft_consensus.cc:2272] T f19b5e8e460a4c4ba7399b981c603b84 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.553355 10041 ts_tablet_manager.cc:1916] T f19b5e8e460a4c4ba7399b981c603b84 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:13.569708 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 594dba3c9d0b4c0ea3089073c2ea8ad8 consensus state has no leader
I20251027 21:08:13.571537 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d4dbfb9efe5f4b00a0279400110e5779"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:13.572919 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4dbfb9efe5f4b00a0279400110e5779 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:13.574920 10031 tablet_bootstrap.cc:492] T 882c58a45d46402eabe96591a7c1de71 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:13.575099 10031 ts_tablet_manager.cc:1403] T 882c58a45d46402eabe96591a7c1de71 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.051s user 0.013s sys 0.012s
I20251027 21:08:13.575809 10031 raft_consensus.cc:359] T 882c58a45d46402eabe96591a7c1de71 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.575963 10031 raft_consensus.cc:385] T 882c58a45d46402eabe96591a7c1de71 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.576011 10031 raft_consensus.cc:740] T 882c58a45d46402eabe96591a7c1de71 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.576182 10031 consensus_queue.cc:260] T 882c58a45d46402eabe96591a7c1de71 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.576725 10031 ts_tablet_manager.cc:1434] T 882c58a45d46402eabe96591a7c1de71 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:13.577677 10041 ts_tablet_manager.cc:1929] T f19b5e8e460a4c4ba7399b981c603b84 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:13.577775 10041 log.cc:1199] T f19b5e8e460a4c4ba7399b981c603b84 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/f19b5e8e460a4c4ba7399b981c603b84
I20251027 21:08:13.578030 10041 ts_tablet_manager.cc:1950] T f19b5e8e460a4c4ba7399b981c603b84 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:13.578712 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1242) took 1596 ms. Trace:
I20251027 21:08:13.578800 10041 rpcz_store.cc:276] 1027 21:08:11.981831 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:11.981915 (+ 84us) service_pool.cc:225] Handling call
1027 21:08:13.549374 (+1567459us) ts_tablet_manager.cc:1151] Deleting tablet f19b5e8e460a4c4ba7399b981c603b84
1027 21:08:13.549391 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.568002 (+ 18611us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.577573 (+ 9571us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.578694 (+ 1121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1522033}
I20251027 21:08:13.579185 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1243) took 1397 ms. Trace:
I20251027 21:08:13.579250 10041 rpcz_store.cc:276] 1027 21:08:12.181548 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:12.181599 (+ 51us) service_pool.cc:225] Handling call
1027 21:08:13.579075 (+1397476us) ts_tablet_manager.cc:1151] Deleting tablet 7057c55a820d4e379fe1d9abfa353821
1027 21:08:13.579084 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.579170 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1397070}
I20251027 21:08:13.579500 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet f19b5e8e460a4c4ba7399b981c603b84 (table default.Table_7d20b20b1f274f4f94a9497e2d030235 [id=ab4e770ed197499ba9751f3061b2aeba]) successfully deleted
I20251027 21:08:13.580473 10029 tablet_bootstrap.cc:492] T d2203ee62f7e404c95c5f86b5ae85feb P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:13.580642 10029 ts_tablet_manager.cc:1403] T d2203ee62f7e404c95c5f86b5ae85feb P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.069s user 0.001s sys 0.002s
I20251027 21:08:13.581357 10029 raft_consensus.cc:359] T d2203ee62f7e404c95c5f86b5ae85feb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:13.581552 10029 raft_consensus.cc:385] T d2203ee62f7e404c95c5f86b5ae85feb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.581645 10029 raft_consensus.cc:740] T d2203ee62f7e404c95c5f86b5ae85feb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.581868 10029 consensus_queue.cc:260] T d2203ee62f7e404c95c5f86b5ae85feb P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:13.584702 10031 tablet_bootstrap.cc:492] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:13.588194 10031 tablet_bootstrap.cc:654] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:13.597653 9568 consensus_peers.cc:597] T 980eb6755aee4bcfb2ea44d84009021f P 51902b6b61ff4d5584ff21896e847e14 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 980eb6755aee4bcfb2ea44d84009021f. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:13.600000 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1245) took 1414 ms. Trace:
W20251027 21:08:13.600250 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 7057c55a820d4e379fe1d9abfa353821 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7057c55a820d4e379fe1d9abfa353821 already in progress: creating tablet
I20251027 21:08:13.600090 10041 rpcz_store.cc:276] 1027 21:08:12.185457 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:12.185527 (+ 70us) service_pool.cc:225] Handling call
1027 21:08:13.599892 (+1414365us) ts_tablet_manager.cc:1151] Deleting tablet 4bfa70ded2f54e3fb2ca744c723395e2
1027 21:08:13.599902 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.599984 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1414199}
I20251027 21:08:13.603360 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1244) took 1421 ms. Trace:
I20251027 21:08:13.603438 10041 rpcz_store.cc:276] 1027 21:08:12.181647 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:12.181711 (+ 64us) service_pool.cc:225] Handling call
1027 21:08:13.603255 (+1421544us) ts_tablet_manager.cc:1151] Deleting tablet d27f9c498eca4e5f8d2bfe9c805d06b3
1027 21:08:13.603265 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.603345 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1414735}
I20251027 21:08:13.603745 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1246) took 1415 ms. Trace:
I20251027 21:08:13.603807 10041 rpcz_store.cc:276] 1027 21:08:12.188696 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:12.188840 (+ 144us) service_pool.cc:225] Handling call
1027 21:08:13.603649 (+1414809us) ts_tablet_manager.cc:1151] Deleting tablet 5cf3dd1b5f4b4b4ba112df78bf0df8f3
1027 21:08:13.603659 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.603731 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1414672}
I20251027 21:08:13.604080 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1265) took 1188 ms. Trace:
I20251027 21:08:13.604137 10041 rpcz_store.cc:276] 1027 21:08:12.415586 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:12.415661 (+ 75us) service_pool.cc:225] Handling call
1027 21:08:13.604003 (+1188342us) ts_tablet_manager.cc:1151] Deleting tablet 283b72f626644ac2b9b61186387b3839
1027 21:08:13.604010 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.604069 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1181946}
I20251027 21:08:13.604386 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1269) took 1169 ms. Trace:
I20251027 21:08:13.604442 10041 rpcz_store.cc:276] 1027 21:08:12.434772 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:12.434834 (+ 62us) service_pool.cc:225] Handling call
1027 21:08:13.604305 (+1169471us) ts_tablet_manager.cc:1151] Deleting tablet cf7b5a0b6cfd460da98558baf887093e
1027 21:08:13.604313 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.604373 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1169304}
I20251027 21:08:13.604724 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1272) took 1167 ms. Trace:
I20251027 21:08:13.604785 10041 rpcz_store.cc:276] 1027 21:08:12.437118 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:12.437302 (+ 184us) service_pool.cc:225] Handling call
1027 21:08:13.604603 (+1167301us) ts_tablet_manager.cc:1151] Deleting tablet 1ba47e34f5464b47bc32b2776f37d97d
1027 21:08:13.604622 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.604710 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1160980}
I20251027 21:08:13.605044 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1268) took 1177 ms. Trace:
I20251027 21:08:13.605110 10041 rpcz_store.cc:276] 1027 21:08:12.427662 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:12.430227 (+ 2565us) service_pool.cc:225] Handling call
1027 21:08:13.604957 (+1174730us) ts_tablet_manager.cc:1151] Deleting tablet 2dd3e77e26c74ac392274844bc3df181
1027 21:08:13.604965 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.605031 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1160511}
I20251027 21:08:13.605389 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1270) took 1170 ms. Trace:
I20251027 21:08:13.605464 10041 rpcz_store.cc:276] 1027 21:08:12.434899 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:12.434946 (+ 47us) service_pool.cc:225] Handling call
1027 21:08:13.605291 (+1170345us) ts_tablet_manager.cc:1151] Deleting tablet 04c1f18b611d45c9ae2c7b895240f545
1027 21:08:13.605301 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.605373 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1157079}
W20251027 21:08:13.605870 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 4bfa70ded2f54e3fb2ca744c723395e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4bfa70ded2f54e3fb2ca744c723395e2 already in progress: creating tablet
W20251027 21:08:13.606099 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet d27f9c498eca4e5f8d2bfe9c805d06b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d27f9c498eca4e5f8d2bfe9c805d06b3 already in progress: creating tablet
W20251027 21:08:13.606277 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 5cf3dd1b5f4b4b4ba112df78bf0df8f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cf3dd1b5f4b4b4ba112df78bf0df8f3 already in progress: creating tablet
W20251027 21:08:13.606429 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 283b72f626644ac2b9b61186387b3839 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 283b72f626644ac2b9b61186387b3839 already in progress: creating tablet
W20251027 21:08:13.606575 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet cf7b5a0b6cfd460da98558baf887093e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf7b5a0b6cfd460da98558baf887093e already in progress: creating tablet
W20251027 21:08:13.606720 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 1ba47e34f5464b47bc32b2776f37d97d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ba47e34f5464b47bc32b2776f37d97d already in progress: creating tablet
W20251027 21:08:13.606899 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 2dd3e77e26c74ac392274844bc3df181 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2dd3e77e26c74ac392274844bc3df181 already in progress: creating tablet
W20251027 21:08:13.618897 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 04c1f18b611d45c9ae2c7b895240f545 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04c1f18b611d45c9ae2c7b895240f545 already in progress: creating tablet
I20251027 21:08:13.619446 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1277) took 1120 ms. Trace:
I20251027 21:08:13.619665 11543 raft_consensus.cc:493] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.620294 11543 raft_consensus.cc:515] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.621008 11543 leader_election.cc:290] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.621543 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "618f1657a10947fd98976820564c20b3" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:13.621515 10041 rpcz_store.cc:276] 1027 21:08:12.498955 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:12.499034 (+ 79us) service_pool.cc:225] Handling call
1027 21:08:13.618745 (+1119711us) ts_tablet_manager.cc:1151] Deleting tablet f543e8bcf70c4acab9880ee01af85562
1027 21:08:13.619324 (+ 579us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.619424 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1119496}
W20251027 21:08:13.621917 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet f543e8bcf70c4acab9880ee01af85562 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f543e8bcf70c4acab9880ee01af85562 already in progress: creating tablet
I20251027 21:08:13.621941 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "618f1657a10947fd98976820564c20b3" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:13.622421 9859 leader_election.cc:343] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:13.622529 9860 leader_election.cc:343] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.622629 9860 leader_election.cc:304] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.622968 11543 raft_consensus.cc:2749] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.623116 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1278) took 1124 ms. Trace:
I20251027 21:08:13.623621 10041 rpcz_store.cc:276] 1027 21:08:12.499093 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:12.499287 (+ 194us) service_pool.cc:225] Handling call
1027 21:08:13.623015 (+1123728us) ts_tablet_manager.cc:1151] Deleting tablet eeb667b3b10746dbad351c4c123ffd97
1027 21:08:13.623026 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.623102 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1122370}
I20251027 21:08:13.624511 10041 tablet_replica.cc:333] T fb7fdc91156e41de80872b5c941c5d95 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:13.624691 10041 raft_consensus.cc:2243] T fb7fdc91156e41de80872b5c941c5d95 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.624817 10041 raft_consensus.cc:2272] T fb7fdc91156e41de80872b5c941c5d95 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:13.625303 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet eeb667b3b10746dbad351c4c123ffd97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eeb667b3b10746dbad351c4c123ffd97 already in progress: creating tablet
I20251027 21:08:13.626526 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a01b887ede5f492b9175988cbfec56c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:13.627220 10029 ts_tablet_manager.cc:1434] T d2203ee62f7e404c95c5f86b5ae85feb P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.046s user 0.001s sys 0.001s
I20251027 21:08:13.627573 10041 ts_tablet_manager.cc:1916] T fb7fdc91156e41de80872b5c941c5d95 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.644137 11411 raft_consensus.cc:493] T d2203ee62f7e404c95c5f86b5ae85feb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.644265 11411 raft_consensus.cc:515] T d2203ee62f7e404c95c5f86b5ae85feb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:08:13.644526 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a01b887ede5f492b9175988cbfec56c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:13.644829 11411 leader_election.cc:290] T d2203ee62f7e404c95c5f86b5ae85feb P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:13.645569 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2203ee62f7e404c95c5f86b5ae85feb" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:13.645720 9568 leader_election.cc:343] T d2203ee62f7e404c95c5f86b5ae85feb P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: d2203ee62f7e404c95c5f86b5ae85feb
W20251027 21:08:13.646198 9566 leader_election.cc:343] T d2203ee62f7e404c95c5f86b5ae85feb P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.646327 9566 leader_election.cc:304] T d2203ee62f7e404c95c5f86b5ae85feb P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.647173 11411 raft_consensus.cc:2749] T d2203ee62f7e404c95c5f86b5ae85feb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.647722 10029 tablet_bootstrap.cc:492] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:13.648898 11543 raft_consensus.cc:493] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.649021 11543 raft_consensus.cc:515] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:13.649503 11543 leader_election.cc:290] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.649973 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "815e12edcd3847ff8485ba768291a824" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:13.650638 9860 leader_election.cc:343] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.651587 10041 ts_tablet_manager.cc:1929] T fb7fdc91156e41de80872b5c941c5d95 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:13.651716 10041 log.cc:1199] T fb7fdc91156e41de80872b5c941c5d95 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/fb7fdc91156e41de80872b5c941c5d95
I20251027 21:08:13.652025 10041 ts_tablet_manager.cc:1950] T fb7fdc91156e41de80872b5c941c5d95 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:13.652972 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet fb7fdc91156e41de80872b5c941c5d95 (table default.Table_f1103102a7544791874c67fa7cadbbbe [id=08bb0e89b4514dafb2758847b79e64c2]) successfully deleted
W20251027 21:08:13.654374 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet c76870d7ca1b43308f0ab94d4ab39b5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c76870d7ca1b43308f0ab94d4ab39b5f already in progress: creating tablet
W20251027 21:08:13.654563 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 7062871e266c4603bab851a837ef9aec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7062871e266c4603bab851a837ef9aec already in progress: creating tablet
W20251027 21:08:13.654728 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 4c60797e8cd5442b9eef86ee51d4b9c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c60797e8cd5442b9eef86ee51d4b9c6 already in progress: creating tablet
W20251027 21:08:13.654870 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet fe7b00e9cf984d12817c9ea860391199 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe7b00e9cf984d12817c9ea860391199 already in progress: creating tablet
I20251027 21:08:13.660693 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 3b2c479d11794a6a99b5e05d3954ad54
I20251027 21:08:13.661581 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2a79f5c1ef84456fa6c96ec6a45a321d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:13.662221 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a79f5c1ef84456fa6c96ec6a45a321d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:13.665671 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b2c479d11794a6a99b5e05d3954ad54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:13 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:08:13.666293 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 3b2c479d11794a6a99b5e05d3954ad54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b2c479d11794a6a99b5e05d3954ad54 already in progress: creating tablet
I20251027 21:08:13.669742 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b2c479d11794a6a99b5e05d3954ad54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:13 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:13.672832 11519 tablet.cc:1722] T 7f7ad06a15c04efbb701634e14904351 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:13.673521 11546 tablet.cc:1722] T 7f7ad06a15c04efbb701634e14904351 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:13.674084 10029 tablet_bootstrap.cc:654] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.680502 11543 consensus_queue.cc:1048] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:13.698370 9602 tablet_service.cc:1505] Processing CreateTablet for tablet ea579a862af14dd4b71ead79354b8d5e (DEFAULT_TABLE table=default.Table_2350c01a35bb4fbba38029e0a0c72d0d [id=90639b80770647668829fd35731aebe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.698809 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea579a862af14dd4b71ead79354b8d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.699450 9607 tablet_service.cc:1505] Processing CreateTablet for tablet a482bd4d4a3247e9baa9e9f13f7aa0d3 (DEFAULT_TABLE table=default.Table_6d0fed3fe4374cd3b70258dd15d9b06e [id=d8e3b8d9075c4ce3a8d4472be44de656]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.699769 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a482bd4d4a3247e9baa9e9f13f7aa0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.700282 10043 ts_tablet_manager.cc:1929] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 21:08:13.700395 10043 log.cc:1199] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/94f2819dd0394530ae93754bdd1e482f
I20251027 21:08:13.700706 10043 ts_tablet_manager.cc:1950] T 94f2819dd0394530ae93754bdd1e482f P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:13.701392 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1100) took 5084 ms. Trace:
I20251027 21:08:13.701463 10043 rpcz_store.cc:276] 1027 21:08:08.616533 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.616571 (+ 38us) service_pool.cc:225] Handling call
1027 21:08:13.441732 (+4825161us) ts_tablet_manager.cc:1151] Deleting tablet 94f2819dd0394530ae93754bdd1e482f
1027 21:08:13.441742 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.620428 (+178686us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.700197 (+ 79769us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.701363 (+ 1166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4825030}
I20251027 21:08:13.701843 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 94f2819dd0394530ae93754bdd1e482f (table default.Table_f46fff92cdc244f6a020795467099e74 [id=63147f61013c4d6dbc96fd25769b6136]) successfully deleted
I20251027 21:08:13.702138 10043 tablet_replica.cc:333] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:13.702339 10043 raft_consensus.cc:2243] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.702546 10043 raft_consensus.cc:2272] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.703006 10043 ts_tablet_manager.cc:1916] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.703193 9892 tablet_service.cc:1505] Processing CreateTablet for tablet ea579a862af14dd4b71ead79354b8d5e (DEFAULT_TABLE table=default.Table_2350c01a35bb4fbba38029e0a0c72d0d [id=90639b80770647668829fd35731aebe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.703533 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea579a862af14dd4b71ead79354b8d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.707417 9773 raft_consensus.cc:3060] T 420ccfaaa7f24ad0aea9fdfea24fc5dd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:13.712203 11514 raft_consensus.cc:493] T 882c58a45d46402eabe96591a7c1de71 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.712332 11514 raft_consensus.cc:515] T 882c58a45d46402eabe96591a7c1de71 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.712832 11514 leader_election.cc:290] T 882c58a45d46402eabe96591a7c1de71 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:13.713449 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882c58a45d46402eabe96591a7c1de71" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:13.714359 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b2c479d11794a6a99b5e05d3954ad54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:13 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:13.714453 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882c58a45d46402eabe96591a7c1de71" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:13.714645 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "815e12edcd3847ff8485ba768291a824" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:13.715101 9742 tablet_service.cc:1505] Processing CreateTablet for tablet ea579a862af14dd4b71ead79354b8d5e (DEFAULT_TABLE table=default.Table_2350c01a35bb4fbba38029e0a0c72d0d [id=90639b80770647668829fd35731aebe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.715433 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea579a862af14dd4b71ead79354b8d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.715737 11603 raft_consensus.cc:493] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.715860 11603 raft_consensus.cc:515] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.716424 11603 leader_election.cc:290] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.717443 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc4635c00e9b40c49ce04debc76a04bf" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:13.720521 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc4635c00e9b40c49ce04debc76a04bf" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:13.720727 9711 leader_election.cc:304] T 882c58a45d46402eabe96591a7c1de71 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
W20251027 21:08:13.721716 9859 leader_election.cc:343] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.721814 9859 leader_election.cc:304] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:13.722190 9859 leader_election.cc:343] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.722747 11543 raft_consensus.cc:2749] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.723430 9736 tablet_service.cc:1505] Processing CreateTablet for tablet a482bd4d4a3247e9baa9e9f13f7aa0d3 (DEFAULT_TABLE table=default.Table_6d0fed3fe4374cd3b70258dd15d9b06e [id=d8e3b8d9075c4ce3a8d4472be44de656]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.723841 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a482bd4d4a3247e9baa9e9f13f7aa0d3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:13.725670 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 3b2c479d11794a6a99b5e05d3954ad54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b2c479d11794a6a99b5e05d3954ad54 already in progress: creating tablet
I20251027 21:08:13.725893 11514 raft_consensus.cc:2749] T 882c58a45d46402eabe96591a7c1de71 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:13.726714 9859 consensus_peers.cc:597] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:13.731137 9860 leader_election.cc:343] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.731416 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 46185d382f0a4dea803aad692ed3c89a
I20251027 21:08:13.731537 9860 leader_election.cc:304] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.732197 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_802fea5b939f43549f0e180cd01cfa4b" } new_table_name: "default.Table_b9a3394bb7e245198ec2ebc14d9d7fd6" modify_external_catalogs: true
I20251027 21:08:13.733968 11543 raft_consensus.cc:2749] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.734774 9893 tablet_service.cc:1505] Processing CreateTablet for tablet a482bd4d4a3247e9baa9e9f13f7aa0d3 (DEFAULT_TABLE table=default.Table_6d0fed3fe4374cd3b70258dd15d9b06e [id=d8e3b8d9075c4ce3a8d4472be44de656]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.735116 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a482bd4d4a3247e9baa9e9f13f7aa0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.735146 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet 46185d382f0a4dea803aad692ed3c89a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:13 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:13.735606 9607 tablet_service.cc:1505] Processing CreateTablet for tablet ca2deb7ad115495cb6dba8037a936b13 (DEFAULT_TABLE table=default.Table_384092146e0e472da1ce74725287cb18 [id=9dfe7339fd8f4d47a306536262c193d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.736255 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4a368b4b21ba46b083d78e0c1f3cceb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:13.736730 9752 tablet_service.cc:1505] Processing CreateTablet for tablet ca2deb7ad115495cb6dba8037a936b13 (DEFAULT_TABLE table=default.Table_384092146e0e472da1ce74725287cb18 [id=9dfe7339fd8f4d47a306536262c193d7]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:13.736778 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a368b4b21ba46b083d78e0c1f3cceb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:13.737054 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca2deb7ad115495cb6dba8037a936b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.738233 9603 tablet_service.cc:1505] Processing CreateTablet for tablet 62203e2f19644c218c36768336af4b9f (DEFAULT_TABLE table=default.Table_0d90eb17ab224b21b7a594324d408c7e [id=89f338f09b7f4d1ab26013be97bf2f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.738813 9887 tablet_service.cc:1505] Processing CreateTablet for tablet ca2deb7ad115495cb6dba8037a936b13 (DEFAULT_TABLE table=default.Table_384092146e0e472da1ce74725287cb18 [id=9dfe7339fd8f4d47a306536262c193d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.739140 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca2deb7ad115495cb6dba8037a936b13. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:13.739423 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 46185d382f0a4dea803aad692ed3c89a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46185d382f0a4dea803aad692ed3c89a already in progress: creating tablet
I20251027 21:08:13.740675 9611 tablet_service.cc:1505] Processing CreateTablet for tablet d2b3852873ac4153aa76890c0a807872 (DEFAULT_TABLE table=default.Table_196251158be7437dada70c9ff3938aa5 [id=e35ba40cc5d54b338d9ac898cd55d52f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.740891 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 46185d382f0a4dea803aad692ed3c89a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:13 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:13.741086 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet 46185d382f0a4dea803aad692ed3c89a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:13 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:13.741683 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 62203e2f19644c218c36768336af4b9f (DEFAULT_TABLE table=default.Table_0d90eb17ab224b21b7a594324d408c7e [id=89f338f09b7f4d1ab26013be97bf2f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.741976 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62203e2f19644c218c36768336af4b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.742249 10041 tablet_replica.cc:333] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:13.742417 10041 raft_consensus.cc:2243] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.742558 10041 raft_consensus.cc:2272] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.743075 10041 ts_tablet_manager.cc:1916] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.743830 9745 tablet_service.cc:1505] Processing CreateTablet for tablet d2b3852873ac4153aa76890c0a807872 (DEFAULT_TABLE table=default.Table_196251158be7437dada70c9ff3938aa5 [id=e35ba40cc5d54b338d9ac898cd55d52f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.744110 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2b3852873ac4153aa76890c0a807872. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.746050 10032 tablet_bootstrap.cc:492] T e3654aa289154b3ea613b6ff94cbd57e P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:13.746186 10032 ts_tablet_manager.cc:1403] T e3654aa289154b3ea613b6ff94cbd57e P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.232s user 0.035s sys 0.075s
I20251027 21:08:13.746845 10032 raft_consensus.cc:359] T e3654aa289154b3ea613b6ff94cbd57e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.746991 10032 raft_consensus.cc:385] T e3654aa289154b3ea613b6ff94cbd57e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.747033 10032 raft_consensus.cc:740] T e3654aa289154b3ea613b6ff94cbd57e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.747226 10032 consensus_queue.cc:260] T e3654aa289154b3ea613b6ff94cbd57e P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.747712 10032 ts_tablet_manager.cc:1434] T e3654aa289154b3ea613b6ff94cbd57e P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 21:08:13.747967 10032 tablet_bootstrap.cc:492] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:13.749455 10032 tablet_bootstrap.cc:654] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.750401 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 62203e2f19644c218c36768336af4b9f (DEFAULT_TABLE table=default.Table_0d90eb17ab224b21b7a594324d408c7e [id=89f338f09b7f4d1ab26013be97bf2f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.750705 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62203e2f19644c218c36768336af4b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.750977 9771 raft_consensus.cc:3060] T 9fa39697360c46138ef4cd21e8c7a950 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:13.754289 10029 tablet_bootstrap.cc:492] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:13.754419 10029 ts_tablet_manager.cc:1403] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.114s user 0.006s sys 0.001s
I20251027 21:08:13.755014 10032 tablet_bootstrap.cc:492] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:13.755115 10032 ts_tablet_manager.cc:1403] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20251027 21:08:13.755066 10029 raft_consensus.cc:359] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.755221 10029 raft_consensus.cc:385] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.755270 10029 raft_consensus.cc:740] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.755815 10032 raft_consensus.cc:359] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.755946 10032 raft_consensus.cc:385] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.755986 10032 raft_consensus.cc:740] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.756148 10032 consensus_queue.cc:260] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.756224 10029 consensus_queue.cc:260] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.756569 10032 ts_tablet_manager.cc:1434] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:13.756788 10032 tablet_bootstrap.cc:492] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:13.759326 10032 tablet_bootstrap.cc:654] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.764694 10029 ts_tablet_manager.cc:1434] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20251027 21:08:13.770399 10041 ts_tablet_manager.cc:1929] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:13.770498 10041 log.cc:1199] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/46185d382f0a4dea803aad692ed3c89a
I20251027 21:08:13.770853 10041 ts_tablet_manager.cc:1950] T 46185d382f0a4dea803aad692ed3c89a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:13.771957 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 46185d382f0a4dea803aad692ed3c89a (table default.Table_c6f57d0c3113494cb54754d67cfefa71 [id=068e2fce2c4a4915acd15991f2fdcc43]) successfully deleted
I20251027 21:08:13.772173 9893 tablet_service.cc:1505] Processing CreateTablet for tablet d2b3852873ac4153aa76890c0a807872 (DEFAULT_TABLE table=default.Table_196251158be7437dada70c9ff3938aa5 [id=e35ba40cc5d54b338d9ac898cd55d52f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.772505 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2b3852873ac4153aa76890c0a807872. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.772722 11514 raft_consensus.cc:493] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.772838 11514 raft_consensus.cc:515] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.772912 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 8d9e3c3a96774fe48fd6f9ff800b7c9d (DEFAULT_TABLE table=default.Table_a440aeae7b724bfc9703e0524b2eb72f [id=df2c1033dfd5442da9d4bc5e109f12c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.774540 9613 tablet_service.cc:1505] Processing CreateTablet for tablet db38df991dad431ea4bb064b17968d9f (DEFAULT_TABLE table=default.Table_b29fc23df2164435ae09aebea289defe [id=e7ecf1a6a97f4d0dac441261639289de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.774860 9892 tablet_service.cc:1505] Processing CreateTablet for tablet db38df991dad431ea4bb064b17968d9f (DEFAULT_TABLE table=default.Table_b29fc23df2164435ae09aebea289defe [id=e7ecf1a6a97f4d0dac441261639289de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.775301 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db38df991dad431ea4bb064b17968d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.777031 10032 tablet_bootstrap.cc:492] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:13.777137 10032 ts_tablet_manager.cc:1403] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20251027 21:08:13.777820 10032 raft_consensus.cc:359] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.777952 10032 raft_consensus.cc:385] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.777992 10032 raft_consensus.cc:740] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.778156 10032 consensus_queue.cc:260] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.778345 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 8d9e3c3a96774fe48fd6f9ff800b7c9d (DEFAULT_TABLE table=default.Table_a440aeae7b724bfc9703e0524b2eb72f [id=df2c1033dfd5442da9d4bc5e109f12c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.778599 10032 ts_tablet_manager.cc:1434] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 21:08:13.778631 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d9e3c3a96774fe48fd6f9ff800b7c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.778864 10032 tablet_bootstrap.cc:492] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:13.780313 10032 tablet_bootstrap.cc:654] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.785476 11411 consensus_queue.cc:1048] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:13.789250 11514 leader_election.cc:290] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:13.791718 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2d26784bbb24ae3893103802c41affb" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:13.792462 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 987edee73f6349fcade00493c054674a (DEFAULT_TABLE table=default.Table_546dee185ffb414b9876fbdda68cfd0a [id=e2bab22f4720453184b262fea391b326]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.792793 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 987edee73f6349fcade00493c054674a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.795018 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 57b8f69381904f96a0bfa96c985e8aa9
I20251027 21:08:13.795073 10032 tablet_bootstrap.cc:492] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:13.795174 10032 ts_tablet_manager.cc:1403] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.005s
I20251027 21:08:13.795661 9309 master_service.cc:954] ReplaceTablet: received request to replace tablet a12bf9c3d0b445dc92d54aa3059b834c from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:13.795907 10032 raft_consensus.cc:359] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:13.796072 10032 raft_consensus.cc:385] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.796116 10032 raft_consensus.cc:740] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.796270 10032 consensus_queue.cc:260] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:13.796706 10032 ts_tablet_manager.cc:1434] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:13.797194 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet 57b8f69381904f96a0bfa96c985e8aa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:13 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:13.797192 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet 57b8f69381904f96a0bfa96c985e8aa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:13 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:13.800207 10032 tablet_bootstrap.cc:492] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:13.800619 9752 tablet_service.cc:1505] Processing CreateTablet for tablet db38df991dad431ea4bb064b17968d9f (DEFAULT_TABLE table=default.Table_b29fc23df2164435ae09aebea289defe [id=e7ecf1a6a97f4d0dac441261639289de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.800931 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db38df991dad431ea4bb064b17968d9f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:13.804383 9711 leader_election.cc:343] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: a2d26784bbb24ae3893103802c41affb
I20251027 21:08:13.804488 9711 leader_election.cc:304] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:13.804977 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 57b8f69381904f96a0bfa96c985e8aa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:13 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:13.805503 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet a3c0d0c5263140728f3b8f0b11f46aab
I20251027 21:08:13.804980 11514 raft_consensus.cc:2749] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:13.805819 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 57b8f69381904f96a0bfa96c985e8aa9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57b8f69381904f96a0bfa96c985e8aa9 already in progress: creating tablet
I20251027 21:08:13.806092 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 8d9e3c3a96774fe48fd6f9ff800b7c9d (DEFAULT_TABLE table=default.Table_a440aeae7b724bfc9703e0524b2eb72f [id=df2c1033dfd5442da9d4bc5e109f12c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.806443 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d9e3c3a96774fe48fd6f9ff800b7c9d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:13.807185 9566 consensus_peers.cc:597] T 986bf148b3e9482fa2fe53bbac54750d P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:13.807463 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet a3c0d0c5263140728f3b8f0b11f46aab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:13 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:13.813205 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 57b8f69381904f96a0bfa96c985e8aa9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57b8f69381904f96a0bfa96c985e8aa9 already in progress: creating tablet
I20251027 21:08:13.814262 11373 consensus_queue.cc:1048] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:13.802536 10032 tablet_bootstrap.cc:654] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.806988 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet a3c0d0c5263140728f3b8f0b11f46aab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:13 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:13.816687 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet a3c0d0c5263140728f3b8f0b11f46aab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:13 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:13.817127 11521 tablet.cc:1722] T 7f7ad06a15c04efbb701634e14904351 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251027 21:08:13.817281 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet a3c0d0c5263140728f3b8f0b11f46aab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3c0d0c5263140728f3b8f0b11f46aab already in progress: creating tablet
I20251027 21:08:13.817341 11543 raft_consensus.cc:493] T e3654aa289154b3ea613b6ff94cbd57e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.817442 11543 raft_consensus.cc:515] T e3654aa289154b3ea613b6ff94cbd57e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.818106 11543 leader_election.cc:290] T e3654aa289154b3ea613b6ff94cbd57e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.818674 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3654aa289154b3ea613b6ff94cbd57e" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:13.821520 9859 leader_election.cc:343] T e3654aa289154b3ea613b6ff94cbd57e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:13.826509 9566 consensus_peers.cc:597] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:13.827433 11543 raft_consensus.cc:493] T dafbb806923c4cf595abde7b7daa1637 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.827581 11543 raft_consensus.cc:515] T dafbb806923c4cf595abde7b7daa1637 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.828167 11543 leader_election.cc:290] T dafbb806923c4cf595abde7b7daa1637 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.828388 11512 raft_consensus.cc:493] T 27bd3b2a011648b2b8f2f92944513751 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.828529 11512 raft_consensus.cc:515] T 27bd3b2a011648b2b8f2f92944513751 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.829227 11512 leader_election.cc:290] T 27bd3b2a011648b2b8f2f92944513751 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.829725 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27bd3b2a011648b2b8f2f92944513751" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:13.829912 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dafbb806923c4cf595abde7b7daa1637" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:13.830247 9566 leader_election.cc:343] T 27bd3b2a011648b2b8f2f92944513751 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:13.830505 9568 leader_election.cc:343] T 27bd3b2a011648b2b8f2f92944513751 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 27bd3b2a011648b2b8f2f92944513751
I20251027 21:08:13.830637 9568 leader_election.cc:304] T 27bd3b2a011648b2b8f2f92944513751 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:13.831384 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet a3c0d0c5263140728f3b8f0b11f46aab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3c0d0c5263140728f3b8f0b11f46aab already in progress: creating tablet
W20251027 21:08:13.831683 9859 leader_election.cc:343] T dafbb806923c4cf595abde7b7daa1637 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.831928 11512 raft_consensus.cc:2749] T 27bd3b2a011648b2b8f2f92944513751 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.835606 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca2deb7ad115495cb6dba8037a936b13. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:13.837981 9859 consensus_peers.cc:597] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:13.838379 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62203e2f19644c218c36768336af4b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.839083 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3654aa289154b3ea613b6ff94cbd57e" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:13.839511 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dafbb806923c4cf595abde7b7daa1637" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:13.840860 10032 tablet_bootstrap.cc:492] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:13.840970 10032 ts_tablet_manager.cc:1403] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.003s
I20251027 21:08:13.841706 10032 raft_consensus.cc:359] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.841869 10032 raft_consensus.cc:385] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.841918 10032 raft_consensus.cc:740] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.842343 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2b3852873ac4153aa76890c0a807872. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.842475 10029 tablet_bootstrap.cc:492] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
W20251027 21:08:13.843899 9860 leader_election.cc:343] T e3654aa289154b3ea613b6ff94cbd57e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.844002 10029 tablet_bootstrap.cc:654] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.844020 9860 leader_election.cc:304] T e3654aa289154b3ea613b6ff94cbd57e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.844949 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d9e3c3a96774fe48fd6f9ff800b7c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.846868 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db38df991dad431ea4bb064b17968d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.848080 11543 raft_consensus.cc:2749] T e3654aa289154b3ea613b6ff94cbd57e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:13.848511 9860 leader_election.cc:343] T dafbb806923c4cf595abde7b7daa1637 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.848623 9860 leader_election.cc:304] T dafbb806923c4cf595abde7b7daa1637 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.850270 11543 raft_consensus.cc:2749] T dafbb806923c4cf595abde7b7daa1637 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.850993 11514 raft_consensus.cc:493] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.851120 11514 raft_consensus.cc:515] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.851691 11514 leader_election.cc:290] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:13.852272 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "588e2f58f07b4a32a78dffd5b92ceb12" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:13.853444 9711 leader_election.cc:343] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 588e2f58f07b4a32a78dffd5b92ceb12
I20251027 21:08:13.853545 9711 leader_election.cc:304] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:13.853735 10031 tablet_bootstrap.cc:492] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:13.853873 10031 ts_tablet_manager.cc:1403] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.269s user 0.006s sys 0.038s
I20251027 21:08:13.854110 11514 raft_consensus.cc:2749] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.854720 10031 raft_consensus.cc:359] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.854912 10031 raft_consensus.cc:385] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.854969 10031 raft_consensus.cc:740] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.858687 11543 raft_consensus.cc:493] T d0c53b7b32c84a51a967e2a6a22258c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.858848 11543 raft_consensus.cc:515] T d0c53b7b32c84a51a967e2a6a22258c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.859414 11543 leader_election.cc:290] T d0c53b7b32c84a51a967e2a6a22258c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.860183 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0c53b7b32c84a51a967e2a6a22258c6" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:13.860661 9859 leader_election.cc:343] T d0c53b7b32c84a51a967e2a6a22258c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.862776 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0c53b7b32c84a51a967e2a6a22258c6" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:13.864137 10031 consensus_queue.cc:260] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.865875 10031 ts_tablet_manager.cc:1434] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251027 21:08:13.866673 10031 tablet_bootstrap.cc:492] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:08:13.866607 9860 leader_election.cc:343] T d0c53b7b32c84a51a967e2a6a22258c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.867527 9860 leader_election.cc:304] T d0c53b7b32c84a51a967e2a6a22258c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.869139 10031 tablet_bootstrap.cc:654] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.869939 11621 raft_consensus.cc:2749] T d0c53b7b32c84a51a967e2a6a22258c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.870440 10032 consensus_queue.cc:260] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.871533 10032 ts_tablet_manager.cc:1434] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.031s user 0.001s sys 0.001s
I20251027 21:08:13.872022 10032 tablet_bootstrap.cc:492] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:13.873802 9775 raft_consensus.cc:3060] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:13.878576 10029 tablet_bootstrap.cc:492] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:13.878859 10029 ts_tablet_manager.cc:1403] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.002s
I20251027 21:08:13.880503 10029 raft_consensus.cc:359] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.882637 10029 raft_consensus.cc:385] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.883046 10029 raft_consensus.cc:740] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.883744 10031 tablet_bootstrap.cc:492] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:13.883920 10031 ts_tablet_manager.cc:1403] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.002s
I20251027 21:08:13.883720 10029 consensus_queue.cc:260] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.884678 10032 tablet_bootstrap.cc:654] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.884752 10031 raft_consensus.cc:359] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.884922 10031 raft_consensus.cc:385] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.884970 10031 raft_consensus.cc:740] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.890931 10043 ts_tablet_manager.cc:1929] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:13.891160 10043 log.cc:1199] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/5c34c8f1433b40d491ae05e7c6e60ba1
I20251027 21:08:13.891644 10031 consensus_queue.cc:260] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.895012 10029 ts_tablet_manager.cc:1434] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20251027 21:08:13.897646 11521 tablet.cc:1722] T 9fa39697360c46138ef4cd21e8c7a950 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251027 21:08:13.899360 10032 tablet_bootstrap.cc:492] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:13.900050 10032 ts_tablet_manager.cc:1403] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.003s
I20251027 21:08:13.900235 10029 tablet_bootstrap.cc:492] T 6210d3d10fc44b9e9e2b0780883186ce P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:13.901806 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 987edee73f6349fcade00493c054674a (DEFAULT_TABLE table=default.Table_546dee185ffb414b9876fbdda68cfd0a [id=e2bab22f4720453184b262fea391b326]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.902043 10029 tablet_bootstrap.cc:654] T 6210d3d10fc44b9e9e2b0780883186ce P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.902168 11521 tablet.cc:1722] T 9fa39697360c46138ef4cd21e8c7a950 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251027 21:08:13.902558 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 987edee73f6349fcade00493c054674a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.906762 10032 raft_consensus.cc:359] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.907474 10032 raft_consensus.cc:385] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.907536 10032 raft_consensus.cc:740] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.908046 10032 consensus_queue.cc:260] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.909264 10031 ts_tablet_manager.cc:1434] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.025s user 0.001s sys 0.002s
I20251027 21:08:13.909658 10032 ts_tablet_manager.cc:1434] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251027 21:08:13.910135 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 987edee73f6349fcade00493c054674a (DEFAULT_TABLE table=default.Table_546dee185ffb414b9876fbdda68cfd0a [id=e2bab22f4720453184b262fea391b326]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.910157 10032 tablet_bootstrap.cc:492] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:13.910854 10031 tablet_bootstrap.cc:492] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:13.910846 11621 raft_consensus.cc:493] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.910984 11621 raft_consensus.cc:515] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.911181 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 987edee73f6349fcade00493c054674a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.911716 11621 leader_election.cc:290] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.912793 10032 tablet_bootstrap.cc:654] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.914093 11514 raft_consensus.cc:493] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51902b6b61ff4d5584ff21896e847e14)
I20251027 21:08:13.914269 11514 raft_consensus.cc:515] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } }
I20251027 21:08:13.914440 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1385beb1dc14ee0b42a2f4e1f66f532" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:13.915129 11514 leader_election.cc:290] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531)
I20251027 21:08:13.916685 9603 tablet_service.cc:1505] Processing CreateTablet for tablet d0477049bd5c46959e6e866402a90b85 (DEFAULT_TABLE table=default.Table_ad34618843dc4a708617166fb4909e93 [id=f100befd54a94c64ad46083fc60b1c4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.917035 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0477049bd5c46959e6e866402a90b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.917744 9752 tablet_service.cc:1505] Processing CreateTablet for tablet d0477049bd5c46959e6e866402a90b85 (DEFAULT_TABLE table=default.Table_ad34618843dc4a708617166fb4909e93 [id=f100befd54a94c64ad46083fc60b1c4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.918187 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1385beb1dc14ee0b42a2f4e1f66f532" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:13.920271 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0477049bd5c46959e6e866402a90b85. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:13.914984 9860 leader_election.cc:343] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:13.921970 9859 leader_election.cc:343] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.922426 9859 leader_election.cc:304] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.922780 11543 raft_consensus.cc:2749] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.922817 9922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b8156d9ab2e4dfba383ddc72137aab4" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:13.924093 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b8156d9ab2e4dfba383ddc72137aab4" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:13.924592 9710 leader_election.cc:304] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:13.927464 11514 raft_consensus.cc:2749] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251027 21:08:13.931595 9887 tablet_service.cc:1505] Processing CreateTablet for tablet d0477049bd5c46959e6e866402a90b85 (DEFAULT_TABLE table=default.Table_ad34618843dc4a708617166fb4909e93 [id=f100befd54a94c64ad46083fc60b1c4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:13.932077 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0477049bd5c46959e6e866402a90b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:13.932360 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet aaaf70c950044387a63820079db3c829
I20251027 21:08:13.932770 9304 catalog_manager.cc:6719] ReplaceTablet: tablet aaaf70c950044387a63820079db3c829 deleted and replaced by tablet 262b4fb71a2d40fc879da56f8c51ef83
I20251027 21:08:13.933272 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet dac3c4836e91423ba5baa58039dc0c55
I20251027 21:08:13.934749 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet aaaf70c950044387a63820079db3c829 with delete_type TABLET_DATA_DELETED (replaced by tablet 262b4fb71a2d40fc879da56f8c51ef83) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:13.934937 10041 tablet_replica.cc:333] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:13.935079 10041 raft_consensus.cc:2243] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.935196 10041 raft_consensus.cc:2272] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.935516 9306 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 72c1dc3b9eff47ee81df821deb789307 (table default.Table_ff5458dda9f042e189bd8cd69c06d025 [id=2a96776404334cf79ddb611a234a5844]). Expected version 2 got 0
I20251027 21:08:13.935658 10041 ts_tablet_manager.cc:1916] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.935879 9306 catalog_manager.cc:5649] T dd9e6656d8df43afbb7f70706bc14585 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:13.936219 9306 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet dd9e6656d8df43afbb7f70706bc14585 (table default.Table_f78d839900634631b6963b2d63f0d084 [id=3cdf432ec43d42e8bc2d253a92eac8d3]). Expected version 3 got 0
I20251027 21:08:13.936641 9306 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet e6b58b1f4f63461aa96e416cae879be8 (table default.Table_f3c2de263e8849caaef87b570fc69459 [id=ef50ef2238804c98ad19c5b1c0b881c4]). Expected version 1 got 0
I20251027 21:08:13.937314 9307 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_cc8a1775cbcb4e0eaeb8c90bf9d61c04 [id=bb676562b97f48088767941239cfbf33] alter complete (version 3)
W20251027 21:08:13.938539 9307 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.9.14.66:44229 (request call id 217) took 782 ms (client timeout 1000 ms). Trace:
W20251027 21:08:13.938608 9307 rpcz_store.cc:269] 1027 21:08:13.156388 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:13.187099 (+ 30711us) service_pool.cc:225] Handling call
1027 21:08:13.198411 (+ 11312us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x616000051154. stack: 00007f956e0a89f6 00007f956e0a87b0 00007f956d0843ad 0000000000983f04 0000000000983ecd 00007f957d3af6bd 00007f957d3af7b3 00007f9579f76441 00007f9581c309dd 00007f9581db7aa8 00007f95788caca6 00007f95788ca9d5 00007f957504727e 00007f957504615e 00007f957504c395 00007f957504ccbf
1027 21:08:13.938521 (+740110us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:08:13.317561 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:08:13.317863 (+ 302us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:08:13.317871 (+ 8us) write_op.cc:435] Acquired schema lock
1027 21:08:13.317874 (+ 3us) tablet.cc:660] Decoding operations
1027 21:08:13.317993 (+ 119us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:08:13.318010 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1027 21:08:13.318016 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1027 21:08:13.318085 (+ 69us) tablet.cc:699] Row locks acquired
1027 21:08:13.318089 (+ 4us) write_op.cc:260] PREPARE: finished
1027 21:08:13.318118 (+ 29us) write_op.cc:270] Start()
1027 21:08:13.318165 (+ 47us) write_op.cc:276] Timestamp: P: 1761599293318100 usec, L: 0
1027 21:08:13.318169 (+ 4us) op_driver.cc:348] REPLICATION: starting
1027 21:08:13.318334 (+ 165us) log.cc:844] Serialized 356 byte log entry
1027 21:08:13.322991 (+ 4657us) op_driver.cc:464] REPLICATION: finished
1027 21:08:13.323208 (+ 217us) write_op.cc:301] APPLY: starting
1027 21:08:13.323229 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
1027 21:08:13.323280 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
1027 21:08:13.323313 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:08:13.323375 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:08:13.323415 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:08:13.323427 (+ 12us) write_op.cc:312] APPLY: finished
1027 21:08:13.323513 (+ 86us) log.cc:844] Serialized 23 byte log entry
1027 21:08:13.323578 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:13.323601 (+ 23us) write_op.cc:454] Released schema lock
1027 21:08:13.323610 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1027 21:08:13.332856 (+ 9246us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:13.332862 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"spinlock_wait_cycles":25008896,"child_traces":[["op",{"apply.queue_time_us":159,"apply.run_cpu_time_us":496,"apply.run_wall_time_us":495,"num_ops":1,"prepare.queue_time_us":117107,"prepare.run_cpu_time_us":758,"prepare.run_wall_time_us":15287,"raft.queue_time_us":70,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":850,"replication_time_us":4787}]]}
I20251027 21:08:13.939177 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet fcd43479e12e4ababfb4f781aa1827e3 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:13.940444 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet aaaf70c950044387a63820079db3c829 with delete_type TABLET_DATA_DELETED (replaced by tablet 262b4fb71a2d40fc879da56f8c51ef83) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:13.940824 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet dac3c4836e91423ba5baa58039dc0c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:13 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:13.940999 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 9861a0efcba64141aae450663c5d83de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:13.941038 9745 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d64a13667194c12864fa388eb88a380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:13.941160 10040 tablet_replica.cc:333] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:13.941330 10040 raft_consensus.cc:2243] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.941469 10040 raft_consensus.cc:2272] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.941502 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c2d713d72fd4b5ca34cb3e828c75761 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:08:13.941881 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet aaaf70c950044387a63820079db3c829 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aaaf70c950044387a63820079db3c829 already in progress: creating tablet
I20251027 21:08:13.941921 10040 ts_tablet_manager.cc:1916] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:13.943109 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet dac3c4836e91423ba5baa58039dc0c55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dac3c4836e91423ba5baa58039dc0c55 already in progress: creating tablet
I20251027 21:08:13.943857 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet dac3c4836e91423ba5baa58039dc0c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:13 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:13.944320 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6666bff9087744179f5eb067a74af9f9" } new_table_name: "default.Table_858501c0f55d4579a3951e49a75d47b0" modify_external_catalogs: true
I20251027 21:08:13.945559 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_1833f7f718ae412da5598787977143c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:13.946066 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1833f7f718ae412da5598787977143c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:13.947203 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_32b6d93944194801af6407774d5afbec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:13.947592 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32b6d93944194801af6407774d5afbec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:13.947959 10040 ts_tablet_manager.cc:1929] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:13.948052 10040 log.cc:1199] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/9861a0efcba64141aae450663c5d83de
I20251027 21:08:13.948320 10040 ts_tablet_manager.cc:1950] T 9861a0efcba64141aae450663c5d83de P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:13.948658 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_deb3d240fb84407e810d7077b5bda502"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:13.948814 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 9861a0efcba64141aae450663c5d83de (table default.Table_a8906ade8f93405bb76d1ca9b7a0afa6 [id=a0c9cfaafd6c46dfb9a999f25f8977fa]) successfully deleted
W20251027 21:08:13.949011 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_deb3d240fb84407e810d7077b5bda502 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:13.949074 10040 tablet_replica.cc:333] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:13.949235 10040 raft_consensus.cc:2243] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.949385 10040 raft_consensus.cc:2272] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.950171 10040 ts_tablet_manager.cc:1916] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.952659 11624 raft_consensus.cc:493] T e6ed9d7a700446edafcd0daa02aa7ef6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.952793 11624 raft_consensus.cc:515] T e6ed9d7a700446edafcd0daa02aa7ef6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.953400 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 83f5fbf5757a49df93ab949b7657c61f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:04 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:13.953418 11624 leader_election.cc:290] T e6ed9d7a700446edafcd0daa02aa7ef6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.953882 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6ed9d7a700446edafcd0daa02aa7ef6" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:13.954334 9566 leader_election.cc:343] T e6ed9d7a700446edafcd0daa02aa7ef6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:13.954627 9568 leader_election.cc:343] T e6ed9d7a700446edafcd0daa02aa7ef6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: e6ed9d7a700446edafcd0daa02aa7ef6
I20251027 21:08:13.954720 9568 leader_election.cc:304] T e6ed9d7a700446edafcd0daa02aa7ef6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.955108 11624 raft_consensus.cc:2749] T e6ed9d7a700446edafcd0daa02aa7ef6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:13.959528 10040 ts_tablet_manager.cc:1929] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:13.960237 11621 raft_consensus.cc:493] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:13.960366 11621 raft_consensus.cc:515] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.960659 10041 ts_tablet_manager.cc:1929] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:13.960801 10041 log.cc:1199] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/aaaf70c950044387a63820079db3c829
I20251027 21:08:13.960911 11621 leader_election.cc:290] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:13.961115 10041 ts_tablet_manager.cc:1950] T aaaf70c950044387a63820079db3c829 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:13.961120 10040 log.cc:1199] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/1d64a13667194c12864fa388eb88a380
I20251027 21:08:13.961423 10040 ts_tablet_manager.cc:1950] T 1d64a13667194c12864fa388eb88a380 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:13.961485 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43611e33512b49d58485cf15de788be7" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:13.961799 10040 tablet_replica.cc:333] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:13.961958 10040 raft_consensus.cc:2243] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.962076 10040 raft_consensus.cc:2272] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.962234 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 1d64a13667194c12864fa388eb88a380 (table default.Table_d11118d727fb4579a6465bd6deebaa43 [id=9df9c59afc184f9bb7494b154852d943]) successfully deleted
I20251027 21:08:13.962414 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet aaaf70c950044387a63820079db3c829 (table default.Table_e08f0cbb71674a44a697686c1b6effc9 [id=471f86ddb9de40a7ae58ec202926a1fd]) successfully deleted
I20251027 21:08:13.962539 10040 ts_tablet_manager.cc:1916] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:13.962527 9859 leader_election.cc:343] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.963152 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43611e33512b49d58485cf15de788be7" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:13.963617 9860 leader_election.cc:343] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:13.963784 9860 leader_election.cc:304] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:13.963980 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet dac3c4836e91423ba5baa58039dc0c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:13 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:13.964015 11621 raft_consensus.cc:2749] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:13.965631 9566 consensus_peers.cc:597] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:13.965705 10029 tablet_bootstrap.cc:492] T 6210d3d10fc44b9e9e2b0780883186ce P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:13.965817 10029 ts_tablet_manager.cc:1403] T 6210d3d10fc44b9e9e2b0780883186ce P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.000s
I20251027 21:08:13.966548 10029 raft_consensus.cc:359] T 6210d3d10fc44b9e9e2b0780883186ce P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.966710 10029 raft_consensus.cc:385] T 6210d3d10fc44b9e9e2b0780883186ce P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.966760 10029 raft_consensus.cc:740] T 6210d3d10fc44b9e9e2b0780883186ce P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.966941 10029 consensus_queue.cc:260] T 6210d3d10fc44b9e9e2b0780883186ce P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.967411 10029 ts_tablet_manager.cc:1434] T 6210d3d10fc44b9e9e2b0780883186ce P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:13.967696 10029 tablet_bootstrap.cc:492] T 465be1b24d16457e834023a2b2194049 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:13.969308 10029 tablet_bootstrap.cc:654] T 465be1b24d16457e834023a2b2194049 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.974328 10043 ts_tablet_manager.cc:1950] T 5c34c8f1433b40d491ae05e7c6e60ba1 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:13.975258 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1092) took 5360 ms. Trace:
I20251027 21:08:13.975443 10043 rpcz_store.cc:276] 1027 21:08:08.615168 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.618227 (+ 3059us) service_pool.cc:225] Handling call
1027 21:08:13.702091 (+5083864us) ts_tablet_manager.cc:1151] Deleting tablet 5c34c8f1433b40d491ae05e7c6e60ba1
1027 21:08:13.702100 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:13.748341 (+ 46241us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.890794 (+142453us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:13.975237 (+ 84443us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5083676}
I20251027 21:08:13.976017 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 5c34c8f1433b40d491ae05e7c6e60ba1 (table default.Table_5877315061df40eb80ba9d966632471a [id=c70fb6448778479fa17e811b21830ea5]) successfully deleted
I20251027 21:08:13.976424 10043 tablet_replica.cc:333] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:13.976648 10043 raft_consensus.cc:2243] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:13.976858 10043 raft_consensus.cc:2272] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:13.977806 10043 ts_tablet_manager.cc:1916] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:13.979012 10031 tablet_bootstrap.cc:654] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.979992 10029 tablet_bootstrap.cc:492] T 465be1b24d16457e834023a2b2194049 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:13.980139 10029 ts_tablet_manager.cc:1403] T 465be1b24d16457e834023a2b2194049 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
W20251027 21:08:13.980825 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet dac3c4836e91423ba5baa58039dc0c55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dac3c4836e91423ba5baa58039dc0c55 already in progress: creating tablet
I20251027 21:08:13.981359 10029 raft_consensus.cc:359] T 465be1b24d16457e834023a2b2194049 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.981535 10029 raft_consensus.cc:385] T 465be1b24d16457e834023a2b2194049 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.981585 10029 raft_consensus.cc:740] T 465be1b24d16457e834023a2b2194049 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.981822 10029 consensus_queue.cc:260] T 465be1b24d16457e834023a2b2194049 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.982808 10029 ts_tablet_manager.cc:1434] T 465be1b24d16457e834023a2b2194049 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251027 21:08:13.983069 10029 tablet_bootstrap.cc:492] T d8691aa22fe84cf29893ed4e79d156cc P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:13.984911 10029 tablet_bootstrap.cc:654] T d8691aa22fe84cf29893ed4e79d156cc P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:13.985821 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet aaaf70c950044387a63820079db3c829 with delete_type TABLET_DATA_DELETED (replaced by tablet 262b4fb71a2d40fc879da56f8c51ef83) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:13.989460 10029 tablet_bootstrap.cc:492] T d8691aa22fe84cf29893ed4e79d156cc P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:13.989591 10029 ts_tablet_manager.cc:1403] T d8691aa22fe84cf29893ed4e79d156cc P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20251027 21:08:13.990289 10029 raft_consensus.cc:359] T d8691aa22fe84cf29893ed4e79d156cc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.990454 10029 raft_consensus.cc:385] T d8691aa22fe84cf29893ed4e79d156cc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:13.990517 10029 raft_consensus.cc:740] T d8691aa22fe84cf29893ed4e79d156cc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:13.990743 10029 consensus_queue.cc:260] T d8691aa22fe84cf29893ed4e79d156cc P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:13.991940 10029 ts_tablet_manager.cc:1434] T d8691aa22fe84cf29893ed4e79d156cc P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:13.992727 10029 tablet_bootstrap.cc:492] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:13.994226 10029 tablet_bootstrap.cc:654] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.000847 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 5fd596241ab9450b9c37068ff5bd8b01 (DEFAULT_TABLE table=default.Table_f2e5f26bedf64d45b5d74662abb032e8 [id=dae37121a23a4277ba79ea3994093dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.001255 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fd596241ab9450b9c37068ff5bd8b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.001505 10029 tablet_bootstrap.cc:492] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:14.001617 10029 ts_tablet_manager.cc:1403] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20251027 21:08:14.001770 9603 tablet_service.cc:1505] Processing CreateTablet for tablet 5fd596241ab9450b9c37068ff5bd8b01 (DEFAULT_TABLE table=default.Table_f2e5f26bedf64d45b5d74662abb032e8 [id=dae37121a23a4277ba79ea3994093dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.002089 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fd596241ab9450b9c37068ff5bd8b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.002373 11621 raft_consensus.cc:493] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.002525 11621 raft_consensus.cc:515] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.003219 11621 leader_election.cc:290] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.003795 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67994a7cacd94b2ebb16fe154832c8fd" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:14.004117 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67994a7cacd94b2ebb16fe154832c8fd" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:14.004611 9860 leader_election.cc:343] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:14.004981 9859 leader_election.cc:343] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.005069 9859 leader_election.cc:304] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.005513 11621 raft_consensus.cc:2749] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:14.005748 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce9bad29c7b24f4cb729c8ef2e163c72 consensus state has no leader
I20251027 21:08:14.005897 10029 raft_consensus.cc:359] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.006111 10029 raft_consensus.cc:385] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.006186 10029 raft_consensus.cc:740] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.006397 10029 consensus_queue.cc:260] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.006911 10032 tablet_bootstrap.cc:492] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:14.007031 10032 ts_tablet_manager.cc:1403] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.097s user 0.009s sys 0.005s
I20251027 21:08:14.007406 10029 ts_tablet_manager.cc:1434] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251027 21:08:14.007663 10029 tablet_bootstrap.cc:492] T e1e58f6228254e3eb25dbf627a505ca1 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:14.007697 10032 raft_consensus.cc:359] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.007843 10032 raft_consensus.cc:385] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.007890 10032 raft_consensus.cc:740] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.008004 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 5fd596241ab9450b9c37068ff5bd8b01 (DEFAULT_TABLE table=default.Table_f2e5f26bedf64d45b5d74662abb032e8 [id=dae37121a23a4277ba79ea3994093dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.008086 10032 consensus_queue.cc:260] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.008332 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fd596241ab9450b9c37068ff5bd8b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.009198 10029 tablet_bootstrap.cc:654] T e1e58f6228254e3eb25dbf627a505ca1 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.011615 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_676ac104a790481d9043a383d7c96caf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:14.012115 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_676ac104a790481d9043a383d7c96caf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:14.012637 10032 ts_tablet_manager.cc:1434] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251027 21:08:14.012876 10032 tablet_bootstrap.cc:492] T 720cfe66349f4a8896e433d79a2f25aa P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:14.015162 10032 tablet_bootstrap.cc:654] T 720cfe66349f4a8896e433d79a2f25aa P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.016440 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet a12bf9c3d0b445dc92d54aa3059b834c
I20251027 21:08:14.016837 9309 catalog_manager.cc:6719] ReplaceTablet: tablet a12bf9c3d0b445dc92d54aa3059b834c deleted and replaced by tablet 48ed3748472947e68690adce18053efa
I20251027 21:08:14.017035 9309 master_service.cc:954] ReplaceTablet: received request to replace tablet 054d78c606d64944b32618b93615befd from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:14.020468 9312 catalog_manager.cc:5649] T 823620e5486e4cf982fdf9105e8cc4f1 P 0fbd567f929a4f6d8e4d78feef7ed678 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67). New cstate: current_term: 1 leader_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:14.021184 9312 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 57ef07c764664deaa4b2e9bcae7e7973 (table default.Table_7d6b8d0ba5ad42fa9fdbcf13c7647d41 [id=a541c8d30a9540f5a7d4dd98a5d3739a]). Expected version 1 got 0
I20251027 21:08:14.021410 9312 catalog_manager.cc:5649] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67). New cstate: current_term: 1 leader_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:14.021701 9312 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 72c1dc3b9eff47ee81df821deb789307 (table default.Table_ff5458dda9f042e189bd8cd69c06d025 [id=2a96776404334cf79ddb611a234a5844]). Expected version 2 got 0
I20251027 21:08:14.022820 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet a12bf9c3d0b445dc92d54aa3059b834c with delete_type TABLET_DATA_DELETED (replaced by tablet 48ed3748472947e68690adce18053efa) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.023273 11621 raft_consensus.cc:493] T 54c2b10e79e149c2a637475e1f518907 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.023402 11621 raft_consensus.cc:515] T 54c2b10e79e149c2a637475e1f518907 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.023821 10029 tablet_bootstrap.cc:492] T e1e58f6228254e3eb25dbf627a505ca1 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:14.023972 11621 leader_election.cc:290] T 54c2b10e79e149c2a637475e1f518907 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.024547 10040 ts_tablet_manager.cc:1929] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:14.024570 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54c2b10e79e149c2a637475e1f518907" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:14.024573 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54c2b10e79e149c2a637475e1f518907" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:14.024665 10040 log.cc:1199] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/9c2d713d72fd4b5ca34cb3e828c75761
I20251027 21:08:14.025101 10040 ts_tablet_manager.cc:1950] T 9c2d713d72fd4b5ca34cb3e828c75761 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
W20251027 21:08:14.025091 9860 leader_election.cc:343] T 54c2b10e79e149c2a637475e1f518907 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:14.025453 9859 leader_election.cc:343] T 54c2b10e79e149c2a637475e1f518907 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.025559 9859 leader_election.cc:304] T 54c2b10e79e149c2a637475e1f518907 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.025913 11621 raft_consensus.cc:2749] T 54c2b10e79e149c2a637475e1f518907 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.026086 10029 ts_tablet_manager.cc:1403] T e1e58f6228254e3eb25dbf627a505ca1 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20251027 21:08:14.026821 10029 raft_consensus.cc:359] T e1e58f6228254e3eb25dbf627a505ca1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.026980 10029 raft_consensus.cc:385] T e1e58f6228254e3eb25dbf627a505ca1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.027031 10029 raft_consensus.cc:740] T e1e58f6228254e3eb25dbf627a505ca1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
W20251027 21:08:14.027124 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 49b6d7e547444187b7027b79e4ede9de consensus state has no leader
I20251027 21:08:14.027201 10029 consensus_queue.cc:260] T e1e58f6228254e3eb25dbf627a505ca1 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.027961 10032 tablet_bootstrap.cc:492] T 720cfe66349f4a8896e433d79a2f25aa P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:14.028074 10032 ts_tablet_manager.cc:1403] T 720cfe66349f4a8896e433d79a2f25aa P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20251027 21:08:14.028546 10029 ts_tablet_manager.cc:1434] T e1e58f6228254e3eb25dbf627a505ca1 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 21:08:14.028652 10032 raft_consensus.cc:359] T 720cfe66349f4a8896e433d79a2f25aa P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.028787 10032 raft_consensus.cc:385] T 720cfe66349f4a8896e433d79a2f25aa P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.028828 10032 raft_consensus.cc:740] T 720cfe66349f4a8896e433d79a2f25aa P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.029011 10032 consensus_queue.cc:260] T 720cfe66349f4a8896e433d79a2f25aa P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.029191 10029 tablet_bootstrap.cc:492] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:14.029675 10032 ts_tablet_manager.cc:1434] T 720cfe66349f4a8896e433d79a2f25aa P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251027 21:08:14.029990 10032 tablet_bootstrap.cc:492] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:14.031739 10032 tablet_bootstrap.cc:654] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.032150 10029 tablet_bootstrap.cc:654] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.032934 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet a12bf9c3d0b445dc92d54aa3059b834c with delete_type TABLET_DATA_DELETED (replaced by tablet 48ed3748472947e68690adce18053efa) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.033252 10040 tablet_replica.cc:333] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:14.033327 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 9c2d713d72fd4b5ca34cb3e828c75761 (table default.Table_294e3c4092274dc1a146d3f604bc2179 [id=bd8f38a55b9f42b99e372c133a148007]) successfully deleted
I20251027 21:08:14.033492 10040 raft_consensus.cc:2243] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:14.033658 10040 raft_consensus.cc:2272] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:14.033692 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet a12bf9c3d0b445dc92d54aa3059b834c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a12bf9c3d0b445dc92d54aa3059b834c already in progress: creating tablet
I20251027 21:08:14.034303 10040 ts_tablet_manager.cc:1916] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:14.035463 9773 raft_consensus.cc:3060] T 882c58a45d46402eabe96591a7c1de71 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.038563 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_1bd31d83bf85421ea90591d545184406"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:14.039268 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1bd31d83bf85421ea90591d545184406 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:14.040604 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_bff95c9aec0642ed94a4b1355a7a32af" } new_table_name: "default.Table_6b1655f44b7f422f888954117b126d5f" modify_external_catalogs: true
I20251027 21:08:14.041723 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b9dc7d4bed0d40c29eab4ab96fd00d20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:14.042157 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9dc7d4bed0d40c29eab4ab96fd00d20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:14.047233 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_fdf2451e0a2a48f9b902827e2a291e90" } new_table_name: "default.Table_09d01437f8784695815459e6001b1681" modify_external_catalogs: true
I20251027 21:08:14.055331 10040 ts_tablet_manager.cc:1929] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:14.055451 10040 log.cc:1199] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/83f5fbf5757a49df93ab949b7657c61f
I20251027 21:08:14.055755 10040 ts_tablet_manager.cc:1950] T 83f5fbf5757a49df93ab949b7657c61f P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:14.056813 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 83f5fbf5757a49df93ab949b7657c61f (table default.Table_50b757fa69504e57872e1ecaee13583c [id=2719caa93b0945c59e6ad4d6f1e9e717]) successfully deleted
I20251027 21:08:14.057178 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 260d2ca28d0d4cafa3b3ece9f143c314 (DEFAULT_TABLE table=default.Table_1d34cea4395c4a96b561f68c263f70cb [id=136bd6aa35de4abc83ebcbc53b285538]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.057672 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 260d2ca28d0d4cafa3b3ece9f143c314. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:14.059303 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet a12bf9c3d0b445dc92d54aa3059b834c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a12bf9c3d0b445dc92d54aa3059b834c already in progress: creating tablet
I20251027 21:08:14.065660 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 260d2ca28d0d4cafa3b3ece9f143c314 (DEFAULT_TABLE table=default.Table_1d34cea4395c4a96b561f68c263f70cb [id=136bd6aa35de4abc83ebcbc53b285538]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.066026 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 260d2ca28d0d4cafa3b3ece9f143c314. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.067260 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet a12bf9c3d0b445dc92d54aa3059b834c with delete_type TABLET_DATA_DELETED (replaced by tablet 48ed3748472947e68690adce18053efa) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:14.067597 10031 tablet_bootstrap.cc:492] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:14.067715 10031 ts_tablet_manager.cc:1403] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.158s user 0.020s sys 0.003s
I20251027 21:08:14.068394 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 260d2ca28d0d4cafa3b3ece9f143c314 (DEFAULT_TABLE table=default.Table_1d34cea4395c4a96b561f68c263f70cb [id=136bd6aa35de4abc83ebcbc53b285538]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.068400 10031 raft_consensus.cc:359] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.068543 10031 raft_consensus.cc:385] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.068593 10031 raft_consensus.cc:740] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.068739 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 260d2ca28d0d4cafa3b3ece9f143c314. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.068787 10031 consensus_queue.cc:260] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.070207 10031 ts_tablet_manager.cc:1434] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:14.071028 9775 raft_consensus.cc:3060] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.077250 10031 tablet_bootstrap.cc:492] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:14.079049 10031 tablet_bootstrap.cc:654] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.084102 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 54aab41ae940412c953da16346ce3657 (DEFAULT_TABLE table=default.Table_36902a42180149cbb6d361295f2a32cc [id=1fa27d1112304611af4b440ace89982c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.083596 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 54aab41ae940412c953da16346ce3657 (DEFAULT_TABLE table=default.Table_36902a42180149cbb6d361295f2a32cc [id=1fa27d1112304611af4b440ace89982c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.084493 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54aab41ae940412c953da16346ce3657. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.084808 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 54aab41ae940412c953da16346ce3657 (DEFAULT_TABLE table=default.Table_36902a42180149cbb6d361295f2a32cc [id=1fa27d1112304611af4b440ace89982c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.085201 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54aab41ae940412c953da16346ce3657. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.086750 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 090d1627d92f48dea238390c683a585d (DEFAULT_TABLE table=default.Table_2689f2aa6d3e4d30b155fa18f7f6b903 [id=84cd03ae95a642858dcb7548ecf4d1b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.087162 9613 tablet_service.cc:1505] Processing CreateTablet for tablet 090d1627d92f48dea238390c683a585d (DEFAULT_TABLE table=default.Table_2689f2aa6d3e4d30b155fa18f7f6b903 [id=84cd03ae95a642858dcb7548ecf4d1b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.087530 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 090d1627d92f48dea238390c683a585d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.087882 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 090d1627d92f48dea238390c683a585d (DEFAULT_TABLE table=default.Table_2689f2aa6d3e4d30b155fa18f7f6b903 [id=84cd03ae95a642858dcb7548ecf4d1b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.088229 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 090d1627d92f48dea238390c683a585d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.088367 9611 tablet_service.cc:1505] Processing CreateTablet for tablet f5bf7b066583425593b89b1d3eb2b82d (DEFAULT_TABLE table=default.Table_5dc88891b75b4f1ca7b3ffba0d2b1fda [id=b46f1e4f1e6d4bd8b4a3edd1d6787612]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.088680 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5bf7b066583425593b89b1d3eb2b82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.090385 9736 tablet_service.cc:1505] Processing CreateTablet for tablet bbe3975e2cda4d8799ffa42de9927b2d (DEFAULT_TABLE table=default.Table_e8e08989f9c4416fa4fb7ba0895ee310 [id=1041aedb0b254c9e90ba63d379c0cf4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.090915 9893 tablet_service.cc:1505] Processing CreateTablet for tablet bbe3975e2cda4d8799ffa42de9927b2d (DEFAULT_TABLE table=default.Table_e8e08989f9c4416fa4fb7ba0895ee310 [id=1041aedb0b254c9e90ba63d379c0cf4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.091238 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbe3975e2cda4d8799ffa42de9927b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.094902 11621 consensus_queue.cc:1048] T 72c1dc3b9eff47ee81df821deb789307 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:14.096333 11621 raft_consensus.cc:493] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.096467 11621 raft_consensus.cc:515] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.097000 11621 leader_election.cc:290] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.097116 9771 raft_consensus.cc:3060] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.097865 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9558d53008640739ac6fb45f1352fc0" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:14.098305 9859 leader_election.cc:343] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.100524 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9558d53008640739ac6fb45f1352fc0" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:14.100947 9860 leader_election.cc:343] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.101045 9860 leader_election.cc:304] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.101409 11621 raft_consensus.cc:2749] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.111058 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54aab41ae940412c953da16346ce3657. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.111416 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbe3975e2cda4d8799ffa42de9927b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.111585 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 090d1627d92f48dea238390c683a585d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.114799 9607 tablet_service.cc:1505] Processing CreateTablet for tablet bbe3975e2cda4d8799ffa42de9927b2d (DEFAULT_TABLE table=default.Table_e8e08989f9c4416fa4fb7ba0895ee310 [id=1041aedb0b254c9e90ba63d379c0cf4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.123878 9773 raft_consensus.cc:3060] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.128722 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet fcd43479e12e4ababfb4f781aa1827e3
I20251027 21:08:14.129113 9307 catalog_manager.cc:6719] ReplaceTablet: tablet fcd43479e12e4ababfb4f781aa1827e3 deleted and replaced by tablet d86d1a605afe4c29b1f2a139adf0259f
I20251027 21:08:14.131631 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet fcd43479e12e4ababfb4f781aa1827e3 with delete_type TABLET_DATA_DELETED (replaced by tablet d86d1a605afe4c29b1f2a139adf0259f) from {username='slave'} at 127.0.0.1:34644
W20251027 21:08:14.132192 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet fcd43479e12e4ababfb4f781aa1827e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fcd43479e12e4ababfb4f781aa1827e3 already in progress: creating tablet
I20251027 21:08:14.132529 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet fcd43479e12e4ababfb4f781aa1827e3 with delete_type TABLET_DATA_DELETED (replaced by tablet d86d1a605afe4c29b1f2a139adf0259f) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.134296 10031 tablet_bootstrap.cc:492] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:14.134454 10031 ts_tablet_manager.cc:1403] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.000s
I20251027 21:08:14.136647 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_86b3f1d7a96f4c788e7ab443853fd5cd" } new_table_name: "default.Table_bf97262f310649dfa463bad6a1ba5633" modify_external_catalogs: true
W20251027 21:08:14.138931 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet fcd43479e12e4ababfb4f781aa1827e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fcd43479e12e4ababfb4f781aa1827e3 already in progress: creating tablet
I20251027 21:08:14.133021 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet fcd43479e12e4ababfb4f781aa1827e3 with delete_type TABLET_DATA_DELETED (replaced by tablet d86d1a605afe4c29b1f2a139adf0259f) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:14.142292 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbe3975e2cda4d8799ffa42de9927b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.143074 11411 raft_consensus.cc:493] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.149145 11411 raft_consensus.cc:515] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.149704 11411 leader_election.cc:290] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.143884 11512 raft_consensus.cc:493] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.149844 11512 raft_consensus.cc:515] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.150372 11512 leader_election.cc:290] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.151093 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cccd253ae2a46328dc9f767b8e9ee97" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:14.151095 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "520f7142f8af4599ac4880b0692e3151" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:14.151661 9566 leader_election.cc:343] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.151679 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cccd253ae2a46328dc9f767b8e9ee97" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:14.151841 9925 raft_consensus.cc:2468] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
W20251027 21:08:14.151839 9566 leader_election.cc:343] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:14.152107 9568 leader_election.cc:343] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 520f7142f8af4599ac4880b0692e3151
I20251027 21:08:14.152215 9568 leader_election.cc:304] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.152662 9568 leader_election.cc:304] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.153076 11512 raft_consensus.cc:2749] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.153228 11512 raft_consensus.cc:2804] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:14.153275 11512 raft_consensus.cc:493] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:14.153322 11512 raft_consensus.cc:3060] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.156011 11512 raft_consensus.cc:515] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.156492 11512 leader_election.cc:290] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.156843 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cccd253ae2a46328dc9f767b8e9ee97" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:14.157177 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cccd253ae2a46328dc9f767b8e9ee97" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
W20251027 21:08:14.157272 9566 leader_election.cc:343] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.157423 9925 raft_consensus.cc:3060] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.161095 9925 raft_consensus.cc:2468] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:14.161998 9568 leader_election.cc:304] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.162482 11411 raft_consensus.cc:2804] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:14.162580 11411 raft_consensus.cc:697] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:14.162791 11411 consensus_queue.cc:237] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.178619 10031 raft_consensus.cc:359] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.178808 10031 raft_consensus.cc:385] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.178866 10031 raft_consensus.cc:740] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.179052 10031 consensus_queue.cc:260] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.179517 10031 ts_tablet_manager.cc:1434] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.045s user 0.002s sys 0.000s
I20251027 21:08:14.179765 10031 tablet_bootstrap.cc:492] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:14.181545 10031 tablet_bootstrap.cc:654] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:14.192360 9566 consensus_peers.cc:597] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d64a13667194c12864fa388eb88a380. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:14.193271 10032 tablet_bootstrap.cc:492] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:14.193426 10032 ts_tablet_manager.cc:1403] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.163s user 0.034s sys 0.022s
I20251027 21:08:14.194211 10032 raft_consensus.cc:359] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.194617 10032 raft_consensus.cc:385] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.194705 10032 raft_consensus.cc:740] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.194980 10032 consensus_queue.cc:260] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.196533 10029 tablet_bootstrap.cc:492] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:14.196997 10029 ts_tablet_manager.cc:1403] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.168s user 0.023s sys 0.036s
I20251027 21:08:14.197779 10029 raft_consensus.cc:359] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.197978 10029 raft_consensus.cc:385] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.198062 10029 raft_consensus.cc:740] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.198315 10029 consensus_queue.cc:260] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.198983 10029 ts_tablet_manager.cc:1434] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:14.201509 10031 tablet_bootstrap.cc:492] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:14.201644 10031 ts_tablet_manager.cc:1403] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.001s
I20251027 21:08:14.201790 10032 ts_tablet_manager.cc:1434] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251027 21:08:14.202396 10031 raft_consensus.cc:359] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.202544 10031 raft_consensus.cc:385] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.202603 10031 raft_consensus.cc:740] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.202802 10031 consensus_queue.cc:260] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.204046 10031 ts_tablet_manager.cc:1434] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:14.205142 10031 tablet_bootstrap.cc:492] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:14.206408 10032 tablet_bootstrap.cc:492] T 674e624257f84c47acff5d61f1fd5b54 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:14.207830 10032 tablet_bootstrap.cc:654] T 674e624257f84c47acff5d61f1fd5b54 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.207947 10031 tablet_bootstrap.cc:654] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.209642 11514 raft_consensus.cc:493] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.209748 11514 raft_consensus.cc:515] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.210227 11514 leader_election.cc:290] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:14.210762 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07a9c3b7ef1548ec9d5ae9f20545f263" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:14.211628 9711 leader_election.cc:343] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 07a9c3b7ef1548ec9d5ae9f20545f263
I20251027 21:08:14.211719 9711 leader_election.cc:304] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:14.211889 11514 raft_consensus.cc:2749] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.223065 10032 tablet_bootstrap.cc:492] T 674e624257f84c47acff5d61f1fd5b54 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:14.223191 10032 ts_tablet_manager.cc:1403] T 674e624257f84c47acff5d61f1fd5b54 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20251027 21:08:14.224073 10032 raft_consensus.cc:359] T 674e624257f84c47acff5d61f1fd5b54 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.224268 10032 raft_consensus.cc:385] T 674e624257f84c47acff5d61f1fd5b54 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.224321 10032 raft_consensus.cc:740] T 674e624257f84c47acff5d61f1fd5b54 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.225909 10031 tablet_bootstrap.cc:492] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:14.226063 10031 ts_tablet_manager.cc:1403] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20251027 21:08:14.226388 9307 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 334c580862bd4bf09be3f1710aa6a092 (table default.Table_06c15ff076cd4c24bc49b16b01bff965 [id=cf9029e184114ac0a5b641a496701122]). Expected version 1 got 0
I20251027 21:08:14.226732 10031 raft_consensus.cc:359] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.226897 10031 raft_consensus.cc:385] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.226943 10031 raft_consensus.cc:740] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.227123 10031 consensus_queue.cc:260] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.227387 9307 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet dd9e6656d8df43afbb7f70706bc14585 (table default.Table_f78d839900634631b6963b2d63f0d084 [id=3cdf432ec43d42e8bc2d253a92eac8d3]). Expected version 3 got 0
I20251027 21:08:14.227775 9307 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 72c1dc3b9eff47ee81df821deb789307 (table default.Table_ff5458dda9f042e189bd8cd69c06d025 [id=2a96776404334cf79ddb611a234a5844]). Expected version 2 got 0
I20251027 21:08:14.228299 10031 ts_tablet_manager.cc:1434] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 21:08:14.229177 10031 tablet_bootstrap.cc:492] T 30ce703650c94c06b93bce976cb5d1fd P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:14.230701 10031 tablet_bootstrap.cc:654] T 30ce703650c94c06b93bce976cb5d1fd P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.231096 9307 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet e6b58b1f4f63461aa96e416cae879be8 (table default.Table_f3c2de263e8849caaef87b570fc69459 [id=ef50ef2238804c98ad19c5b1c0b881c4]). Expected version 1 got 0
I20251027 21:08:14.231304 9307 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 76bee322d0b94abe9166170733608853 (table default.Table_4b0bc60a7c5e4888b79210296b7dfa20 [id=c9e7aa435d5b48e096c2b67b0e678eeb]). Expected version 1 got 0
I20251027 21:08:14.231559 9307 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 8cccd253ae2a46328dc9f767b8e9ee97 (table default.Table_4ba26495af344e3cb082a4b781f1b208 [id=501b230a824d4a17bb1903c8409407b7]). Expected version 2 got 0
I20251027 21:08:14.231899 9306 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_6327cacf4ed54d56b522291ad9594784 [id=3eb54d245ec64ee48022c541c2e49b17] alter complete (version 2)
I20251027 21:08:14.241477 10031 tablet_bootstrap.cc:492] T 30ce703650c94c06b93bce976cb5d1fd P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:14.241603 10031 ts_tablet_manager.cc:1403] T 30ce703650c94c06b93bce976cb5d1fd P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.001s
I20251027 21:08:14.242275 10031 raft_consensus.cc:359] T 30ce703650c94c06b93bce976cb5d1fd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.242419 10031 raft_consensus.cc:385] T 30ce703650c94c06b93bce976cb5d1fd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.242470 10031 raft_consensus.cc:740] T 30ce703650c94c06b93bce976cb5d1fd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.242666 10031 consensus_queue.cc:260] T 30ce703650c94c06b93bce976cb5d1fd P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.243569 10031 ts_tablet_manager.cc:1434] T 30ce703650c94c06b93bce976cb5d1fd P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:14.243932 10031 tablet_bootstrap.cc:492] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:14.248162 10032 consensus_queue.cc:260] T 674e624257f84c47acff5d61f1fd5b54 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.249886 10032 ts_tablet_manager.cc:1434] T 674e624257f84c47acff5d61f1fd5b54 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20251027 21:08:14.252707 10031 tablet_bootstrap.cc:654] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.259023 10032 tablet_bootstrap.cc:492] T 9ace4188d3494a05a5678381f8d8fd17 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:08:14.261992 9568 consensus_peers.cc:597] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aaaf70c950044387a63820079db3c829. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:14.265966 10031 tablet_bootstrap.cc:492] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:14.266089 10031 ts_tablet_manager.cc:1403] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.002s
I20251027 21:08:14.266978 10031 raft_consensus.cc:359] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.267127 10031 raft_consensus.cc:385] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.267168 10031 raft_consensus.cc:740] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.267354 11514 raft_consensus.cc:493] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.267452 11514 raft_consensus.cc:515] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.267921 11514 leader_election.cc:290] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:14.268198 10043 ts_tablet_manager.cc:1929] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:14.268323 10043 log.cc:1199] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/96a680e8f16343b1908829a67e960b60
I20251027 21:08:14.268424 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3abc5f78c4e41e29128e2d790319c75" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:14.268626 9634 raft_consensus.cc:2393] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77b0513f17c0472088901a1ce8b08218 in current term 1: Already voted for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in this term.
I20251027 21:08:14.268666 10043 ts_tablet_manager.cc:1950] T 96a680e8f16343b1908829a67e960b60 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
W20251027 21:08:14.268772 9711 leader_election.cc:343] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: b3abc5f78c4e41e29128e2d790319c75
I20251027 21:08:14.268934 10031 consensus_queue.cc:260] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.269306 9710 leader_election.cc:304] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:14.269914 11514 raft_consensus.cc:3060] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.271929 10031 ts_tablet_manager.cc:1434] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20251027 21:08:14.272153 10031 tablet_bootstrap.cc:492] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:14.272449 11514 raft_consensus.cc:2749] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.272557 10032 tablet_bootstrap.cc:654] T 9ace4188d3494a05a5678381f8d8fd17 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.274264 10031 tablet_bootstrap.cc:654] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.278620 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 054d78c606d64944b32618b93615befd
I20251027 21:08:14.279022 9309 catalog_manager.cc:6719] ReplaceTablet: tablet 054d78c606d64944b32618b93615befd deleted and replaced by tablet e070a791682e4924970c0d0f5d1ddf83
I20251027 21:08:14.282707 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 054d78c606d64944b32618b93615befd with delete_type TABLET_DATA_DELETED (replaced by tablet e070a791682e4924970c0d0f5d1ddf83) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.282907 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 054d78c606d64944b32618b93615befd with delete_type TABLET_DATA_DELETED (replaced by tablet e070a791682e4924970c0d0f5d1ddf83) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.283133 11543 raft_consensus.cc:493] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.283361 11543 raft_consensus.cc:515] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:14.283473 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 054d78c606d64944b32618b93615befd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 054d78c606d64944b32618b93615befd already in progress: creating tablet
W20251027 21:08:14.284194 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 054d78c606d64944b32618b93615befd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 054d78c606d64944b32618b93615befd already in progress: creating tablet
I20251027 21:08:14.284279 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe7b00e9cf984d12817c9ea860391199" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:14.284487 11543 leader_election.cc:290] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.284948 9893 tablet_service.cc:1505] Processing CreateTablet for tablet f5bf7b066583425593b89b1d3eb2b82d (DEFAULT_TABLE table=default.Table_5dc88891b75b4f1ca7b3ffba0d2b1fda [id=b46f1e4f1e6d4bd8b4a3edd1d6787612]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.285291 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5bf7b066583425593b89b1d3eb2b82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.285228 9752 tablet_service.cc:1505] Processing CreateTablet for tablet f5bf7b066583425593b89b1d3eb2b82d (DEFAULT_TABLE table=default.Table_5dc88891b75b4f1ca7b3ffba0d2b1fda [id=b46f1e4f1e6d4bd8b4a3edd1d6787612]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.285461 10032 tablet_bootstrap.cc:492] T 9ace4188d3494a05a5678381f8d8fd17 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:14.285542 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5bf7b066583425593b89b1d3eb2b82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.285559 10032 ts_tablet_manager.cc:1403] T 9ace4188d3494a05a5678381f8d8fd17 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20251027 21:08:14.286226 10032 raft_consensus.cc:359] T 9ace4188d3494a05a5678381f8d8fd17 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:14.286368 10032 raft_consensus.cc:385] T 9ace4188d3494a05a5678381f8d8fd17 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.286412 10032 raft_consensus.cc:740] T 9ace4188d3494a05a5678381f8d8fd17 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.286562 10032 consensus_queue.cc:260] T 9ace4188d3494a05a5678381f8d8fd17 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:08:14.289187 9860 leader_election.cc:343] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.290310 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 56afea36f94045dc99d8dd3f5a0d7d9c (DEFAULT_TABLE table=default.Table_f9fbc28451dc4d219623c15566ea0827 [id=b4e4278fa14e4f9481ab436dc5161105]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.290664 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56afea36f94045dc99d8dd3f5a0d7d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.290853 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 79f5d19489df4689b8781e2bd0f1f121 (DEFAULT_TABLE table=default.Table_4ecc813240f9453ab4ec46a231f2b0d9 [id=8b4304d218b64b64b3ada83e5e22a05d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.291126 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79f5d19489df4689b8781e2bd0f1f121. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.292703 10032 ts_tablet_manager.cc:1434] T 9ace4188d3494a05a5678381f8d8fd17 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251027 21:08:14.294355 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 56cea2d6c38b447ba070e777d0525810 (DEFAULT_TABLE table=default.Table_5030d0bdc78e4eb9979dbfc63446e63b [id=f500277871dc4f78ab54c95ad34ef1c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.294378 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 56afea36f94045dc99d8dd3f5a0d7d9c (DEFAULT_TABLE table=default.Table_f9fbc28451dc4d219623c15566ea0827 [id=b4e4278fa14e4f9481ab436dc5161105]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.294615 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56cea2d6c38b447ba070e777d0525810. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.294755 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56afea36f94045dc99d8dd3f5a0d7d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.297849 10032 tablet_bootstrap.cc:492] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:14.298799 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 79f5d19489df4689b8781e2bd0f1f121 (DEFAULT_TABLE table=default.Table_4ecc813240f9453ab4ec46a231f2b0d9 [id=8b4304d218b64b64b3ada83e5e22a05d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.299111 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79f5d19489df4689b8781e2bd0f1f121. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.299455 10032 tablet_bootstrap.cc:654] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.300995 11512 raft_consensus.cc:493] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.301118 11512 raft_consensus.cc:515] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.301591 11512 leader_election.cc:290] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.302345 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 56cea2d6c38b447ba070e777d0525810 (DEFAULT_TABLE table=default.Table_5030d0bdc78e4eb9979dbfc63446e63b [id=f500277871dc4f78ab54c95ad34ef1c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.302655 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56cea2d6c38b447ba070e777d0525810. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.302665 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb95618d9fd54a4f8121b43e717d5616" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:14.303181 9566 leader_election.cc:343] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.314818 10032 tablet_bootstrap.cc:492] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:14.314939 10032 ts_tablet_manager.cc:1403] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20251027 21:08:14.315580 10032 raft_consensus.cc:359] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:14.315747 10032 raft_consensus.cc:385] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.315821 10032 raft_consensus.cc:740] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.316027 10032 consensus_queue.cc:260] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:14.317348 10032 ts_tablet_manager.cc:1434] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:14.318467 10032 tablet_bootstrap.cc:492] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:14.320047 10032 tablet_bootstrap.cc:654] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.325977 9309 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet dd9e6656d8df43afbb7f70706bc14585 (table default.Table_f78d839900634631b6963b2d63f0d084 [id=3cdf432ec43d42e8bc2d253a92eac8d3]). Expected version 3 got 0
I20251027 21:08:14.329039 9309 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 72c1dc3b9eff47ee81df821deb789307 (table default.Table_ff5458dda9f042e189bd8cd69c06d025 [id=2a96776404334cf79ddb611a234a5844]). Expected version 2 got 0
I20251027 21:08:14.329581 9309 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet e3654aa289154b3ea613b6ff94cbd57e (table default.Table_64cbe911866949e285d945765603b0c1 [id=4c67737c2c734ad0ad33349fa69668bc]). Expected version 1 got 0
I20251027 21:08:14.329869 9309 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 54c2b10e79e149c2a637475e1f518907 (table default.Table_ae51ca0900c0448a8852598394e0e912 [id=9f1e545ab3174e8a893b9183cf95768d]). Expected version 1 got 0
I20251027 21:08:14.330610 9309 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 57ef07c764664deaa4b2e9bcae7e7973 (table default.Table_7d6b8d0ba5ad42fa9fdbcf13c7647d41 [id=a541c8d30a9540f5a7d4dd98a5d3739a]). Expected version 1 got 0
I20251027 21:08:14.330732 10031 tablet_bootstrap.cc:492] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:14.330893 10031 ts_tablet_manager.cc:1403] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.003s
I20251027 21:08:14.334803 11514 raft_consensus.cc:493] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.334908 11514 raft_consensus.cc:515] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.335367 11514 leader_election.cc:290] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:14.336190 9711 leader_election.cc:343] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 7292bc6c22ca4ab5b8f53a69520833ba
I20251027 21:08:14.336772 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7292bc6c22ca4ab5b8f53a69520833ba" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:14.331524 10031 raft_consensus.cc:359] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.337113 10031 raft_consensus.cc:385] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.337177 10031 raft_consensus.cc:740] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.337393 9710 leader_election.cc:304] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:14.337433 10031 consensus_queue.cc:260] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.337663 11626 raft_consensus.cc:2749] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.340658 10031 ts_tablet_manager.cc:1434] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251027 21:08:14.341037 10031 tablet_bootstrap.cc:492] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:14.342747 11543 raft_consensus.cc:493] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.342876 11543 raft_consensus.cc:515] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.343431 11543 leader_election.cc:290] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.344136 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c6c2292ad0e4066958897fff01370af" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:14.344249 10031 tablet_bootstrap.cc:654] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:14.344623 9860 leader_election.cc:343] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.344648 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe7b00e9cf984d12817c9ea860391199" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:14.345113 9859 leader_election.cc:343] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.345203 9859 leader_election.cc:304] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.345677 11543 raft_consensus.cc:2749] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.349256 11543 raft_consensus.cc:493] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.349376 11543 raft_consensus.cc:515] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.349829 11543 leader_election.cc:290] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.350312 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9be6b996df8145b9b0df1a2d2dc0ef03" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:14.351029 9860 leader_election.cc:343] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.351030 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9be6b996df8145b9b0df1a2d2dc0ef03" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:14.351444 9859 leader_election.cc:343] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.353873 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c6c2292ad0e4066958897fff01370af" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:14.370637 10029 tablet_bootstrap.cc:492] T 42371496a7494e77a7bf1169de5b083a P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:14.372249 10029 tablet_bootstrap.cc:654] T 42371496a7494e77a7bf1169de5b083a P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.373536 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1093) took 5758 ms. Trace:
I20251027 21:08:14.373626 10043 rpcz_store.cc:276] 1027 21:08:08.615246 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.618451 (+ 3205us) service_pool.cc:225] Handling call
1027 21:08:13.976367 (+5357916us) ts_tablet_manager.cc:1151] Deleting tablet 96a680e8f16343b1908829a67e960b60
1027 21:08:13.976387 (+ 20us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:14.264872 (+288485us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:14.268102 (+ 3230us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:14.373512 (+105410us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5357768}
I20251027 21:08:14.373984 10043 tablet_replica.cc:333] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:14.374179 10043 raft_consensus.cc:2243] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:14.374349 10043 raft_consensus.cc:2272] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:14.374889 10043 ts_tablet_manager.cc:1916] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:14.381592 10043 ts_tablet_manager.cc:1929] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251027 21:08:14.383180 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 52991b0eefbb4d1297266a95c17a4662 consensus state has no leader
I20251027 21:08:14.384137 10043 log.cc:1199] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/631ff72613734913818a3cad24f20dbc
I20251027 21:08:14.384449 10043 ts_tablet_manager.cc:1950] T 631ff72613734913818a3cad24f20dbc P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:14.385562 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1094) took 5770 ms. Trace:
I20251027 21:08:14.388310 10031 tablet_bootstrap.cc:492] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:14.388458 10031 ts_tablet_manager.cc:1403] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.047s user 0.015s sys 0.019s
I20251027 21:08:14.388643 10043 rpcz_store.cc:276] 1027 21:08:08.615324 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:08.618629 (+ 3305us) service_pool.cc:225] Handling call
1027 21:08:14.373943 (+5755314us) ts_tablet_manager.cc:1151] Deleting tablet 631ff72613734913818a3cad24f20dbc
1027 21:08:14.373952 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:14.378664 (+ 4712us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:14.381514 (+ 2850us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:14.385545 (+ 4031us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5755172}
I20251027 21:08:14.389247 10043 tablet_replica.cc:333] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:14.389174 10031 raft_consensus.cc:359] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.389328 10031 raft_consensus.cc:385] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.389387 10031 raft_consensus.cc:740] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.389637 10031 consensus_queue.cc:260] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.392951 10031 ts_tablet_manager.cc:1434] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251027 21:08:14.393224 10031 tablet_bootstrap.cc:492] T dd9e6656d8df43afbb7f70706bc14585 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:14.399559 10031 tablet_bootstrap.cc:654] T dd9e6656d8df43afbb7f70706bc14585 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.410785 11658 raft_consensus.cc:493] T e1e58f6228254e3eb25dbf627a505ca1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.410974 11658 raft_consensus.cc:515] T e1e58f6228254e3eb25dbf627a505ca1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.412536 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1e58f6228254e3eb25dbf627a505ca1" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:14.412961 9566 leader_election.cc:343] T e1e58f6228254e3eb25dbf627a505ca1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.413151 11658 leader_election.cc:290] T e1e58f6228254e3eb25dbf627a505ca1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.413734 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 054d78c606d64944b32618b93615befd with delete_type TABLET_DATA_DELETED (replaced by tablet e070a791682e4924970c0d0f5d1ddf83) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:14.414130 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 56afea36f94045dc99d8dd3f5a0d7d9c (DEFAULT_TABLE table=default.Table_f9fbc28451dc4d219623c15566ea0827 [id=b4e4278fa14e4f9481ab436dc5161105]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.414515 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56afea36f94045dc99d8dd3f5a0d7d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.414613 10029 tablet_bootstrap.cc:492] T 42371496a7494e77a7bf1169de5b083a P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:14.414719 10029 ts_tablet_manager.cc:1403] T 42371496a7494e77a7bf1169de5b083a P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.005s
I20251027 21:08:14.415347 10029 raft_consensus.cc:359] T 42371496a7494e77a7bf1169de5b083a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.415470 10029 raft_consensus.cc:385] T 42371496a7494e77a7bf1169de5b083a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.415513 10029 raft_consensus.cc:740] T 42371496a7494e77a7bf1169de5b083a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.415730 10029 consensus_queue.cc:260] T 42371496a7494e77a7bf1169de5b083a P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.416078 10029 ts_tablet_manager.cc:1434] T 42371496a7494e77a7bf1169de5b083a P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
W20251027 21:08:14.416090 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f838fb6e148c4b16b062e831c57cd454 consensus state has no leader
I20251027 21:08:14.416297 10029 tablet_bootstrap.cc:492] T 00c90ff84d3f4a7f92fda235416eed1d P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:14.416493 9603 tablet_service.cc:1505] Processing CreateTablet for tablet 79f5d19489df4689b8781e2bd0f1f121 (DEFAULT_TABLE table=default.Table_4ecc813240f9453ab4ec46a231f2b0d9 [id=8b4304d218b64b64b3ada83e5e22a05d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:14.416643 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 915f250f9fd046cfa9c6590bd8e4f33c consensus state has no leader
I20251027 21:08:14.416793 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79f5d19489df4689b8781e2bd0f1f121. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.418004 10029 tablet_bootstrap.cc:654] T 00c90ff84d3f4a7f92fda235416eed1d P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.419097 9312 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_1b066b58b2c649a2887ed91461f15d7c [id=887ea33c2fbe4573a1d7169dfd226567] alter complete (version 2)
W20251027 21:08:14.421527 9312 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.9.14.67:53829 (request call id 235) took 1267 ms (client timeout 1000 ms). Trace:
W20251027 21:08:14.424120 9306 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.9.14.65:32849 (request call id 206) took 1310 ms (client timeout 1000 ms). Trace:
I20251027 21:08:14.424283 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 56cea2d6c38b447ba070e777d0525810 (DEFAULT_TABLE table=default.Table_5030d0bdc78e4eb9979dbfc63446e63b [id=f500277871dc4f78ab54c95ad34ef1c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.424577 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56cea2d6c38b447ba070e777d0525810. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:14.424191 9306 rpcz_store.cc:269] 1027 21:08:13.113890 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:13.147829 (+ 33939us) service_pool.cc:225] Handling call
1027 21:08:14.424101 (+1276272us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:08:13.956091 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:08:13.956174 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:08:13.956179 (+ 5us) write_op.cc:435] Acquired schema lock
1027 21:08:13.956181 (+ 2us) tablet.cc:660] Decoding operations
1027 21:08:13.956251 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:08:13.956259 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1027 21:08:13.956264 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1027 21:08:13.956305 (+ 41us) tablet.cc:699] Row locks acquired
1027 21:08:13.956307 (+ 2us) write_op.cc:260] PREPARE: finished
1027 21:08:13.956329 (+ 22us) write_op.cc:270] Start()
1027 21:08:13.956351 (+ 22us) write_op.cc:276] Timestamp: P: 1761599293956307 usec, L: 0
1027 21:08:13.956354 (+ 3us) op_driver.cc:348] REPLICATION: starting
1027 21:08:13.956491 (+ 137us) log.cc:844] Serialized 527 byte log entry
1027 21:08:14.015260 (+ 58769us) op_driver.cc:464] REPLICATION: finished
1027 21:08:14.015989 (+ 729us) write_op.cc:301] APPLY: starting
1027 21:08:14.015998 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1027 21:08:14.016025 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
1027 21:08:14.016045 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:08:14.016076 (+ 31us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:08:14.016099 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:08:14.016106 (+ 7us) write_op.cc:312] APPLY: finished
1027 21:08:14.016153 (+ 47us) log.cc:844] Serialized 23 byte log entry
1027 21:08:14.016188 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:14.016198 (+ 10us) write_op.cc:454] Released schema lock
1027 21:08:14.016203 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1027 21:08:14.140820 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:08:14.140973 (+ 153us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:08:14.140980 (+ 7us) write_op.cc:435] Acquired schema lock
1027 21:08:14.140985 (+ 5us) tablet.cc:660] Decoding operations
1027 21:08:14.141116 (+ 131us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:08:14.141135 (+ 19us) write_op.cc:641] Partition lock acquired for write op
1027 21:08:14.141143 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
1027 21:08:14.141226 (+ 83us) tablet.cc:699] Row locks acquired
1027 21:08:14.141230 (+ 4us) write_op.cc:260] PREPARE: finished
1027 21:08:14.141267 (+ 37us) write_op.cc:270] Start()
1027 21:08:14.141306 (+ 39us) write_op.cc:276] Timestamp: P: 1761599294141232 usec, L: 0
1027 21:08:14.141311 (+ 5us) op_driver.cc:348] REPLICATION: starting
1027 21:08:14.141532 (+ 221us) log.cc:844] Serialized 356 byte log entry
1027 21:08:14.217435 (+ 75903us) op_driver.cc:464] REPLICATION: finished
1027 21:08:14.217895 (+ 460us) write_op.cc:301] APPLY: starting
1027 21:08:14.217911 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1027 21:08:14.217954 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
1027 21:08:14.217979 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:08:14.218032 (+ 53us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:08:14.218062 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:08:14.218069 (+ 7us) write_op.cc:312] APPLY: finished
1027 21:08:14.218193 (+ 124us) log.cc:844] Serialized 23 byte log entry
1027 21:08:14.218253 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:14.218271 (+ 18us) write_op.cc:454] Released schema lock
1027 21:08:14.218278 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1027 21:08:14.218349 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:14.218353 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":15,"spinlock_wait_cycles":4992,"child_traces":[["op",{"apply.queue_time_us":692,"apply.run_cpu_time_us":259,"apply.run_wall_time_us":259,"num_ops":1,"prepare.queue_time_us":18838,"prepare.run_cpu_time_us":500,"prepare.run_wall_time_us":810,"raft.queue_time_us":28,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":58833}],["op",{"apply.queue_time_us":414,"apply.run_cpu_time_us":427,"apply.run_wall_time_us":443,"num_ops":1,"prepare.queue_time_us":122743,"prepare.run_cpu_time_us":939,"prepare.run_wall_time_us":41077,"raft.queue_time_us":1588,"raft.run_cpu_time_us":354,"raft.run_wall_time_us":793,"replication_time_us":76084}]]}
I20251027 21:08:14.427557 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 618f1657a10947fd98976820564c20b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.427748 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet e419e588b31d48c8bc19d7948081cc81 with delete_type TABLET_DATA_DELETED (replaced by tablet cb5ac0fff70b491e8c8842f7a331005e) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.427872 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet 815e12edcd3847ff8485ba768291a824 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.428017 10041 tablet_replica.cc:333] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:14.428174 10041 raft_consensus.cc:2243] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:14.428303 10041 raft_consensus.cc:2272] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:14.421622 9312 rpcz_store.cc:269] 1027 21:08:13.154333 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:13.167502 (+ 13169us) service_pool.cc:225] Handling call
1027 21:08:13.177257 (+ 9755us) spinlock_profiling.cc:243] Waited 8.86 ms on lock 0x616000051154. stack: 00007f956e0a89f6 00007f956e0a87b0 00007f956d0843ad 0000000000983f04 0000000000983ecd 00007f957d3af6bd 00007f957d3af7b3 00007f9579f76441 00007f9581c309dd 00007f9581db7aa8 00007f95788caca6 00007f95788ca9d5 00007f957504727e 00007f957504615e 00007f957504c395 00007f957504ccbf
1027 21:08:14.421508 (+1244251us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:08:14.181965 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:08:14.182169 (+ 204us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:08:14.182179 (+ 10us) write_op.cc:435] Acquired schema lock
1027 21:08:14.182202 (+ 23us) tablet.cc:660] Decoding operations
1027 21:08:14.182364 (+ 162us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:08:14.182380 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1027 21:08:14.182388 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:08:14.182548 (+ 160us) tablet.cc:699] Row locks acquired
1027 21:08:14.182553 (+ 5us) write_op.cc:260] PREPARE: finished
1027 21:08:14.182608 (+ 55us) write_op.cc:270] Start()
1027 21:08:14.182655 (+ 47us) write_op.cc:276] Timestamp: P: 1761599294182561 usec, L: 0
1027 21:08:14.182661 (+ 6us) op_driver.cc:348] REPLICATION: starting
1027 21:08:14.182901 (+ 240us) log.cc:844] Serialized 901 byte log entry
1027 21:08:14.217829 (+ 34928us) op_driver.cc:464] REPLICATION: finished
1027 21:08:14.218425 (+ 596us) write_op.cc:301] APPLY: starting
1027 21:08:14.218437 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
1027 21:08:14.218469 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
1027 21:08:14.218488 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:08:14.218542 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:08:14.218567 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1027 21:08:14.218579 (+ 12us) write_op.cc:312] APPLY: finished
1027 21:08:14.218639 (+ 60us) log.cc:844] Serialized 29 byte log entry
1027 21:08:14.218679 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:14.218694 (+ 15us) write_op.cc:454] Released schema lock
1027 21:08:14.218699 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1027 21:08:14.258053 (+ 39354us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:14.258061 (+ 8us) write_op.cc:454] Released schema lock
Related trace 'op':
1027 21:08:14.262593 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:08:14.262675 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:08:14.262679 (+ 4us) write_op.cc:435] Acquired schema lock
1027 21:08:14.262681 (+ 2us) tablet.cc:660] Decoding operations
1027 21:08:14.262751 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:08:14.262758 (+ 7us) write_op.cc:641] Partition lock acquired for write op
1027 21:08:14.262763 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1027 21:08:14.262829 (+ 66us) tablet.cc:699] Row locks acquired
1027 21:08:14.262833 (+ 4us) write_op.cc:260] PREPARE: finished
1027 21:08:14.262864 (+ 31us) write_op.cc:270] Start()
1027 21:08:14.262893 (+ 29us) write_op.cc:276] Timestamp: P: 1761599294262844 usec, L: 0
1027 21:08:14.262899 (+ 6us) op_driver.cc:348] REPLICATION: starting
1027 21:08:14.263055 (+ 156us) log.cc:844] Serialized 356 byte log entry
1027 21:08:14.418534 (+155479us) op_driver.cc:464] REPLICATION: finished
1027 21:08:14.418618 (+ 84us) write_op.cc:301] APPLY: starting
1027 21:08:14.418628 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1027 21:08:14.418656 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
1027 21:08:14.418677 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:08:14.418714 (+ 37us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:08:14.418737 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:08:14.418744 (+ 7us) write_op.cc:312] APPLY: finished
1027 21:08:14.418801 (+ 57us) log.cc:844] Serialized 23 byte log entry
1027 21:08:14.418848 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:14.418861 (+ 13us) write_op.cc:454] Released schema lock
1027 21:08:14.418866 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":16,"spinlock_wait_cycles":20381568,"child_traces":[["op",{"apply.queue_time_us":559,"apply.run_cpu_time_us":324,"apply.run_wall_time_us":324,"num_ops":2,"prepare.queue_time_us":156084,"prepare.run_cpu_time_us":1210,"prepare.run_wall_time_us":76088,"raft.queue_time_us":4548,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":5824,"replication_time_us":35144}],["op",{"apply.queue_time_us":33,"num_ops":1,"prepare.queue_time_us":42438,"prepare.run_cpu_time_us":546,"prepare.run_wall_time_us":545,"raft.queue_time_us":227,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":155610}]]}
I20251027 21:08:14.428795 10041 ts_tablet_manager.cc:1916] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:14.430245 9303 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_df13ae1f688a44f8be79d5db36fbd976 [id=2bc39b5bc02645ef830eee14e92707db] alter complete (version 2)
I20251027 21:08:14.430960 9307 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_c93714f3039c415d80e39b7ed977450f [id=3a0c8b19df934d1580349b061fe0e8f5] alter complete (version 2)
I20251027 21:08:14.431951 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2db01113aba5414ba6dad68dc4f2bb56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:14.432583 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2db01113aba5414ba6dad68dc4f2bb56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:14.434099 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d5b5337ae438439d984f34a4e0af6c93" } modify_external_catalogs: true
I20251027 21:08:14.434248 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_d5b5337ae438439d984f34a4e0af6c93" } modify_external_catalogs: true
I20251027 21:08:14.434332 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 43611e33512b49d58485cf15de788be7 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e372257733148199d2dbe70d0887cbb) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.427438 10032 tablet_bootstrap.cc:492] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:14.434527 10032 ts_tablet_manager.cc:1403] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.116s user 0.004s sys 0.005s
I20251027 21:08:14.435124 10032 raft_consensus.cc:359] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.435261 10032 raft_consensus.cc:385] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.435310 10032 raft_consensus.cc:740] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.435456 10032 consensus_queue.cc:260] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:14.435683 9299 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet f838fb6e148c4b16b062e831c57cd454 (table default.Table_09d01437f8784695815459e6001b1681 [id=6eae55ab35bf40bca718266926b1187f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f838fb6e148c4b16b062e831c57cd454 consensus state has no leader
I20251027 21:08:14.436199 9305 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8fac6e33215e40128ceb1dfe8a622da6" } modify_external_catalogs: true
I20251027 21:08:14.436319 9305 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8fac6e33215e40128ceb1dfe8a622da6" } modify_external_catalogs: true
I20251027 21:08:14.436808 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_1f3761445da945bbb906e156fe5be872"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:14.437323 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f3761445da945bbb906e156fe5be872 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:14.438448 10032 ts_tablet_manager.cc:1434] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251027 21:08:14.438705 10032 tablet_bootstrap.cc:492] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:14.438812 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_cc8a1775cbcb4e0eaeb8c90bf9d61c04" } modify_external_catalogs: true
I20251027 21:08:14.438920 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_cc8a1775cbcb4e0eaeb8c90bf9d61c04" } modify_external_catalogs: true
I20251027 21:08:14.439118 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet 218d244dc8a549408863110fefbc7b20 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:14.439909 10032 tablet_bootstrap.cc:654] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.440793 10041 ts_tablet_manager.cc:1929] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:14.440868 10041 log.cc:1199] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/618f1657a10947fd98976820564c20b3
I20251027 21:08:14.441130 10041 ts_tablet_manager.cc:1950] T 618f1657a10947fd98976820564c20b3 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:14.441891 10041 tablet_replica.cc:333] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:14.442030 10041 raft_consensus.cc:2243] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:14.442129 10041 raft_consensus.cc:2272] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:14.442535 10041 ts_tablet_manager.cc:1916] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:14.445904 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 618f1657a10947fd98976820564c20b3 (table default.Table_7929c591af15477aa34ef5952495dfc6 [id=822f0869aaa946c69c92b5b27d715621]) successfully deleted
I20251027 21:08:14.449122 10031 tablet_bootstrap.cc:492] T dd9e6656d8df43afbb7f70706bc14585 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:14.449257 10031 ts_tablet_manager.cc:1403] T dd9e6656d8df43afbb7f70706bc14585 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.056s user 0.009s sys 0.004s
I20251027 21:08:14.449800 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8e9ad7e433ba4538be8f764418dd70a2" } modify_external_catalogs: true
I20251027 21:08:14.449927 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_8e9ad7e433ba4538be8f764418dd70a2" } modify_external_catalogs: true
I20251027 21:08:14.450012 10031 raft_consensus.cc:359] T dd9e6656d8df43afbb7f70706bc14585 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.450188 10031 raft_consensus.cc:385] T dd9e6656d8df43afbb7f70706bc14585 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.450256 10031 raft_consensus.cc:740] T dd9e6656d8df43afbb7f70706bc14585 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.451098 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_7cc32a1529164529b4a58e2ea9df9058"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:14.451489 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7cc32a1529164529b4a58e2ea9df9058 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:14.456651 10029 tablet_bootstrap.cc:492] T 00c90ff84d3f4a7f92fda235416eed1d P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:14.456787 10029 ts_tablet_manager.cc:1403] T 00c90ff84d3f4a7f92fda235416eed1d P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.003s
I20251027 21:08:14.456835 10031 consensus_queue.cc:260] T dd9e6656d8df43afbb7f70706bc14585 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.457360 10031 ts_tablet_manager.cc:1434] T dd9e6656d8df43afbb7f70706bc14585 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251027 21:08:14.457417 10029 raft_consensus.cc:359] T 00c90ff84d3f4a7f92fda235416eed1d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.457580 10029 raft_consensus.cc:385] T 00c90ff84d3f4a7f92fda235416eed1d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.457509 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet e477ca4b69c84327b04618d90406e19a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.457638 10029 raft_consensus.cc:740] T 00c90ff84d3f4a7f92fda235416eed1d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.459167 10031 tablet_bootstrap.cc:492] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:14.463277 10041 ts_tablet_manager.cc:1929] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:14.463369 10041 log.cc:1199] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/e419e588b31d48c8bc19d7948081cc81
I20251027 21:08:14.463640 10041 ts_tablet_manager.cc:1950] T e419e588b31d48c8bc19d7948081cc81 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:14.464474 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet e419e588b31d48c8bc19d7948081cc81 (table default.Table_c305a5155cce4754bcbc792f3900b690 [id=82b0b6e6d2e145a7870a13ed5b407b47]) successfully deleted
I20251027 21:08:14.466354 10029 consensus_queue.cc:260] T 00c90ff84d3f4a7f92fda235416eed1d P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.466825 10029 ts_tablet_manager.cc:1434] T 00c90ff84d3f4a7f92fda235416eed1d P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251027 21:08:14.467065 10029 tablet_bootstrap.cc:492] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:14.468118 10032 tablet_bootstrap.cc:492] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:14.468266 10032 ts_tablet_manager.cc:1403] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.001s
I20251027 21:08:14.468381 10029 tablet_bootstrap.cc:654] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.468456 10031 tablet_bootstrap.cc:654] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.469275 10032 raft_consensus.cc:359] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.469475 10032 raft_consensus.cc:385] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.469568 10032 raft_consensus.cc:740] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.469843 10032 consensus_queue.cc:260] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.470767 10032 ts_tablet_manager.cc:1434] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:14.477686 10031 tablet_bootstrap.cc:492] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:14.477726 11543 raft_consensus.cc:493] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.477860 10031 ts_tablet_manager.cc:1403] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20251027 21:08:14.477838 11543 raft_consensus.cc:515] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:14.478312 11543 leader_election.cc:290] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.478844 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11627a84ff9b478c87c323617a09570b" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:14.479314 9859 leader_election.cc:304] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:14.479316 9860 leader_election.cc:343] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.479885 11543 raft_consensus.cc:2749] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:14.480333 9859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 128896us
I20251027 21:08:14.481475 10031 raft_consensus.cc:359] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.481647 10031 raft_consensus.cc:385] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.481710 10031 raft_consensus.cc:740] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.481894 10031 consensus_queue.cc:260] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.482293 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11627a84ff9b478c87c323617a09570b" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:14.484596 10031 ts_tablet_manager.cc:1434] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251027 21:08:14.485172 11543 raft_consensus.cc:493] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.485299 11543 raft_consensus.cc:515] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.485935 11543 leader_election.cc:290] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:14.486613 9859 leader_election.cc:343] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.486691 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23673513189f47638dbbc737da0991d1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:14.486757 9859 leader_election.cc:304] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.486953 11543 raft_consensus.cc:2749] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.487170 10031 tablet_bootstrap.cc:492] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:08:14.487154 9860 leader_election.cc:343] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:14.487651 9859 leader_election.cc:343] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.487785 9859 leader_election.cc:304] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.487998 11543 raft_consensus.cc:2749] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.488215 10041 tablet_replica.cc:333] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:14.488392 10041 raft_consensus.cc:2243] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:14.488530 10041 raft_consensus.cc:2272] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:14.488703 10031 tablet_bootstrap.cc:654] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.489005 10041 ts_tablet_manager.cc:1916] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:14.489511 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23673513189f47638dbbc737da0991d1" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:14.490299 9859 leader_election.cc:343] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.490442 9859 leader_election.cc:304] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.490645 11543 raft_consensus.cc:2749] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.493342 10032 tablet_bootstrap.cc:492] T ed66f9eeeb2b4b4bb77062c241d2140f P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:14.494494 10032 tablet_bootstrap.cc:654] T ed66f9eeeb2b4b4bb77062c241d2140f P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.499887 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 7b8156d9ab2e4dfba383ddc72137aab4
I20251027 21:08:14.500523 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_64efdeb42d0f4db68c0e13066cd8d38f" } modify_external_catalogs: true
I20251027 21:08:14.500669 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_64efdeb42d0f4db68c0e13066cd8d38f" } modify_external_catalogs: true
I20251027 21:08:14.502979 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_508fcc20bab34856b1fb148b6fd6bf3c" } new_table_name: "default.Table_d0a25d8f21e4472c99ec237228bb69c9" modify_external_catalogs: true
I20251027 21:08:14.505903 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet f6789dc2f7cb4a3c826ac507d5c9a239 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.506114 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet b3abc5f78c4e41e29128e2d790319c75 with delete_type TABLET_DATA_DELETED (replaced by tablet ccd419ff40a94f9587b024ec87a2517e) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.506268 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.506352 9745 tablet_service.cc:1552] Processing DeleteTablet for tablet 588e2f58f07b4a32a78dffd5b92ceb12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.506290 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d94bcf0455894a20baa4a30010869f62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:14.506462 10040 tablet_replica.cc:333] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:14.506567 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b8156d9ab2e4dfba383ddc72137aab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.506688 10040 raft_consensus.cc:2243] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:14.506721 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet a2d26784bbb24ae3893103802c41affb with delete_type TABLET_DATA_DELETED (replaced by tablet 9ace4188d3494a05a5678381f8d8fd17) from {username='slave'} at 127.0.0.1:40732
W20251027 21:08:14.506793 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d94bcf0455894a20baa4a30010869f62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:14.506855 10040 raft_consensus.cc:2272] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:14.507308 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 07a9c3b7ef1548ec9d5ae9f20545f263 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.515475 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b8156d9ab2e4dfba383ddc72137aab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.517433 11543 raft_consensus.cc:493] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.517573 11543 raft_consensus.cc:515] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.518126 11543 leader_election.cc:290] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.518719 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a48067b40324c79b1ee4eb431ce4a46" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:14.519192 9860 leader_election.cc:343] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.519745 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a48067b40324c79b1ee4eb431ce4a46" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:14.519898 10040 ts_tablet_manager.cc:1916] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:14.520241 9859 leader_election.cc:343] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.520401 9859 leader_election.cc:304] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.520671 11543 raft_consensus.cc:2749] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.522459 11514 raft_consensus.cc:493] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.522589 11514 raft_consensus.cc:515] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.523056 11514 leader_election.cc:290] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:14.523988 9711 leader_election.cc:343] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: f4187b2f66f24055b1df170f1062cc88
I20251027 21:08:14.525326 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4187b2f66f24055b1df170f1062cc88" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:14.525489 9634 raft_consensus.cc:2468] T f4187b2f66f24055b1df170f1062cc88 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:08:14.525950 9710 leader_election.cc:304] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:14.526401 11514 raft_consensus.cc:2804] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:14.526494 11514 raft_consensus.cc:493] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:14.526582 11514 raft_consensus.cc:3060] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.526827 9773 raft_consensus.cc:3060] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.527252 10043 raft_consensus.cc:2243] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:14.527493 10043 raft_consensus.cc:2272] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:14.528651 10043 ts_tablet_manager.cc:1916] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:14.529162 11514 raft_consensus.cc:515] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:14.530596 9711 leader_election.cc:343] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: f4187b2f66f24055b1df170f1062cc88
I20251027 21:08:14.531350 11514 leader_election.cc:290] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:14.531901 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4187b2f66f24055b1df170f1062cc88" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:14.532034 9634 raft_consensus.cc:3060] T f4187b2f66f24055b1df170f1062cc88 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.534829 9634 raft_consensus.cc:2468] T f4187b2f66f24055b1df170f1062cc88 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:14.535198 9771 raft_consensus.cc:3060] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.535410 9710 leader_election.cc:304] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:14.538784 11626 raft_consensus.cc:2804] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:14.549028 10040 ts_tablet_manager.cc:1929] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:14.549289 10040 log.cc:1199] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/f6789dc2f7cb4a3c826ac507d5c9a239
I20251027 21:08:14.549726 10040 ts_tablet_manager.cc:1950] T f6789dc2f7cb4a3c826ac507d5c9a239 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:14.549958 10043 ts_tablet_manager.cc:1929] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:14.550305 10040 tablet_replica.cc:333] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:14.550287 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet f6789dc2f7cb4a3c826ac507d5c9a239 (table default.Table_ab9ae94ee52942d588d846e58da2e533 [id=19f6e48bb56848afb9b5982c825b093e]) successfully deleted
I20251027 21:08:14.550493 10040 raft_consensus.cc:2243] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:14.550647 10040 raft_consensus.cc:2272] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:14.551077 10040 ts_tablet_manager.cc:1916] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:14.555119 10031 tablet_bootstrap.cc:492] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:14.555265 10031 ts_tablet_manager.cc:1403] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.000s
I20251027 21:08:14.556604 10043 log.cc:1199] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/a2d26784bbb24ae3893103802c41affb
I20251027 21:08:14.557153 10043 ts_tablet_manager.cc:1950] T a2d26784bbb24ae3893103802c41affb P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:14.558857 10031 raft_consensus.cc:359] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.559036 10031 raft_consensus.cc:385] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.559115 10031 raft_consensus.cc:740] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.559376 10031 consensus_queue.cc:260] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.559998 10031 ts_tablet_manager.cc:1434] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251027 21:08:14.560657 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1116) took 5424 ms. Trace:
I20251027 21:08:14.561105 10043 rpcz_store.cc:276] 1027 21:08:09.136129 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.136435 (+ 306us) service_pool.cc:225] Handling call
1027 21:08:14.389190 (+5252755us) ts_tablet_manager.cc:1151] Deleting tablet a2d26784bbb24ae3893103802c41affb
1027 21:08:14.389204 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:14.531935 (+142731us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:14.549855 (+ 17920us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:14.560640 (+ 10785us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12531,"tablet-delete.queue_time_us":5252551}
I20251027 21:08:14.562366 10043 tablet_replica.cc:333] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:14.564513 10031 tablet_bootstrap.cc:492] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:14.564874 10043 raft_consensus.cc:2243] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:14.567730 10043 raft_consensus.cc:2272] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:14.568252 10031 tablet_bootstrap.cc:654] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.569033 10043 ts_tablet_manager.cc:1916] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:14.570178 11543 raft_consensus.cc:493] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.570319 11543 raft_consensus.cc:515] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.570883 11543 leader_election.cc:290] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.571487 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae0ee7cd2c1b406f84285f25420588d0" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:14.572110 9859 leader_election.cc:343] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.572299 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae0ee7cd2c1b406f84285f25420588d0" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:14.575788 11514 raft_consensus.cc:493] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.575973 11514 raft_consensus.cc:515] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:14.576061 9860 leader_election.cc:343] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.576157 9860 leader_election.cc:304] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.576501 11543 raft_consensus.cc:2749] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.576671 11514 leader_election.cc:290] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:14.577391 9711 leader_election.cc:343] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 9ed4f9c61ad54e1292281b30bcd19533
I20251027 21:08:14.577441 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ed4f9c61ad54e1292281b30bcd19533" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:14.577658 9634 raft_consensus.cc:2468] T 9ed4f9c61ad54e1292281b30bcd19533 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:08:14.578253 9710 leader_election.cc:304] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:14.578841 11514 raft_consensus.cc:2804] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:14.578979 11514 raft_consensus.cc:493] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:14.579090 11514 raft_consensus.cc:3060] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.580889 11512 raft_consensus.cc:493] T 42371496a7494e77a7bf1169de5b083a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.581023 11512 raft_consensus.cc:515] T 42371496a7494e77a7bf1169de5b083a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.581604 11512 leader_election.cc:290] T 42371496a7494e77a7bf1169de5b083a P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.582680 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42371496a7494e77a7bf1169de5b083a" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:14.582883 11514 raft_consensus.cc:515] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.583256 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet 28a2e85021394f9d8a9ea524959b52a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:14.583355 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet c9c86253ce914916bee6584675a82067 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:14.583518 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 94f2819dd0394530ae93754bdd1e482f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:14.583551 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet cdbde6272af147829ecadd67d634f049 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:14.583667 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 465be1b24d16457e834023a2b2194049 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:14.583676 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet e6ed9d7a700446edafcd0daa02aa7ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:14.583278 9566 leader_election.cc:343] T 42371496a7494e77a7bf1169de5b083a P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.583974 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet d8691aa22fe84cf29893ed4e79d156cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:14.586074 9711 leader_election.cc:343] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 9ed4f9c61ad54e1292281b30bcd19533
I20251027 21:08:14.586838 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ed4f9c61ad54e1292281b30bcd19533" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:14.587037 9634 raft_consensus.cc:3060] T 9ed4f9c61ad54e1292281b30bcd19533 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.591555 9634 raft_consensus.cc:2468] T 9ed4f9c61ad54e1292281b30bcd19533 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:14.592128 9710 leader_election.cc:304] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:14.592710 11514 leader_election.cc:290] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:14.593132 11514 raft_consensus.cc:2804] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:14.593453 11514 raft_consensus.cc:697] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:14.593837 11514 consensus_queue.cc:237] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.594758 11543 raft_consensus.cc:493] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.594904 11543 raft_consensus.cc:515] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.595543 11543 leader_election.cc:290] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.596153 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8f0cd910b9d4daea287b3b192a81bd2" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:14.596311 10040 ts_tablet_manager.cc:1929] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:14.596292 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8f0cd910b9d4daea287b3b192a81bd2" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:14.596439 10040 log.cc:1199] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/b3abc5f78c4e41e29128e2d790319c75
W20251027 21:08:14.596639 9859 leader_election.cc:343] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:14.596741 9860 leader_election.cc:343] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.596848 9860 leader_election.cc:304] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.597219 11603 raft_consensus.cc:2749] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.602267 10041 ts_tablet_manager.cc:1929] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:14.602509 10041 log.cc:1199] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/815e12edcd3847ff8485ba768291a824
I20251027 21:08:14.602957 10041 ts_tablet_manager.cc:1950] T 815e12edcd3847ff8485ba768291a824 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:14.604235 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 815e12edcd3847ff8485ba768291a824 (table default.Table_2992b8e207024e06957ce7befb657acb [id=decffeb1481e4d798cffd8c9883be59b]) successfully deleted
I20251027 21:08:14.605361 10040 ts_tablet_manager.cc:1950] T b3abc5f78c4e41e29128e2d790319c75 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:14.611263 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet aaa8910d9fdb4146945621ec4ebf969e
I20251027 21:08:14.614524 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet aaa8910d9fdb4146945621ec4ebf969e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.615351 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 11627a84ff9b478c87c323617a09570b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.615957 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet d91fced778784b65be2486e24426d1b5
I20251027 21:08:14.616833 10040 tablet_replica.cc:333] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:14.617048 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet b3abc5f78c4e41e29128e2d790319c75 (table default.Table_3cb6bc9f33ea40aa9d58e82f709c64b7 [id=b9930c5b513749ac905895ebca813733]) successfully deleted
I20251027 21:08:14.617063 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet d91fced778784b65be2486e24426d1b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.617260 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet aaa8910d9fdb4146945621ec4ebf969e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.617585 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet d91fced778784b65be2486e24426d1b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.619002 10040 raft_consensus.cc:2243] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:14.619222 10040 raft_consensus.cc:2272] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:14.619474 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet ae0ee7cd2c1b406f84285f25420588d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.619680 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet cc4635c00e9b40c49ce04debc76a04bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.619829 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet c8f0cd910b9d4daea287b3b192a81bd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.620015 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet e1385beb1dc14ee0b42a2f4e1f66f532 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.620179 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet c548d2413a5940c8b114ced667b4dc41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:05 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.620483 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d2c40f983e743b0b53f987b6a7ba437 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.623829 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 23673513189f47638dbbc737da0991d1 with delete_type TABLET_DATA_DELETED (replaced by tablet 050e9442041d495496bfa7ce25ac6a88) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.624184 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet e419e588b31d48c8bc19d7948081cc81 with delete_type TABLET_DATA_DELETED (replaced by tablet cb5ac0fff70b491e8c8842f7a331005e) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.624388 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet 815e12edcd3847ff8485ba768291a824 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.624647 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c6c2292ad0e4066958897fff01370af with delete_type TABLET_DATA_DELETED (replaced by tablet aeae07c3906145149cea31e5a7c466fe) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.624862 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 618f1657a10947fd98976820564c20b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.629272 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e8fe63ea8aba4614900ac254189464de
I20251027 21:08:14.630291 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet e8fe63ea8aba4614900ac254189464de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.631728 10040 ts_tablet_manager.cc:1916] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:14.635356 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet e8fe63ea8aba4614900ac254189464de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.635718 11681 raft_consensus.cc:493] T dd9e6656d8df43afbb7f70706bc14585 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.636693 11621 raft_consensus.cc:493] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.636816 11621 raft_consensus.cc:515] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.636811 11681 raft_consensus.cc:515] T dd9e6656d8df43afbb7f70706bc14585 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.637445 11621 leader_election.cc:290] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.637981 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0820648719a34f379507efced9cc434b" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:14.638164 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd9e6656d8df43afbb7f70706bc14585" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:14.638492 9859 leader_election.cc:343] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.639125 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0820648719a34f379507efced9cc434b" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:14.639613 9860 leader_election.cc:343] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.639729 9860 leader_election.cc:304] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.640233 11621 raft_consensus.cc:2749] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:14.643184 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9d19e9f63acf4c8c894af38a02225fc6 consensus state has no leader
I20251027 21:08:14.646421 11681 leader_election.cc:290] T dd9e6656d8df43afbb7f70706bc14585 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:14.646777 9922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd9e6656d8df43afbb7f70706bc14585" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:14.647397 9711 leader_election.cc:304] T dd9e6656d8df43afbb7f70706bc14585 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:14.647660 11514 raft_consensus.cc:2749] T dd9e6656d8df43afbb7f70706bc14585 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.648975 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e4d35f229bc46e1859ded7c8196ab9d with delete_type TABLET_DATA_DELETED (replaced by tablet 280ee23d80d64587b18811ff06340eee) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:14.649381 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 6210d3d10fc44b9e9e2b0780883186ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:14.649744 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 27bd3b2a011648b2b8f2f92944513751 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:14.650085 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet d2203ee62f7e404c95c5f86b5ae85feb with delete_type TABLET_DATA_DELETED (replaced by tablet 00bfb0bdf2ca4b288eac0c1dbee5765a) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:14.650478 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet e1e58f6228254e3eb25dbf627a505ca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:14.650755 9568 leader_election.cc:343] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: cb95618d9fd54a4f8121b43e717d5616
I20251027 21:08:14.650916 9568 leader_election.cc:304] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.651217 11411 raft_consensus.cc:2749] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251027 21:08:14.652787 9568 leader_election.cc:343] T e1e58f6228254e3eb25dbf627a505ca1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: e1e58f6228254e3eb25dbf627a505ca1
I20251027 21:08:14.652957 9568 leader_election.cc:304] T e1e58f6228254e3eb25dbf627a505ca1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.653229 11411 raft_consensus.cc:2749] T e1e58f6228254e3eb25dbf627a505ca1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.659081 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_3c8bffe5fd42431fa97808af0d8c634d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:14.659940 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7cf03d793a0548ef80ca9ce245e0fc1f" } new_table_name: "default.Table_245ed7ff92b44dff8b51ce770aab5550" modify_external_catalogs: true
I20251027 21:08:14.661026 11626 raft_consensus.cc:697] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:14.661352 11626 consensus_queue.cc:237] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.662503 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_dc77fcc2c86242789d1775592ee6a417"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:14.666549 11621 raft_consensus.cc:493] T 674e624257f84c47acff5d61f1fd5b54 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.666672 11621 raft_consensus.cc:515] T 674e624257f84c47acff5d61f1fd5b54 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.667224 11621 leader_election.cc:290] T 674e624257f84c47acff5d61f1fd5b54 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.668061 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "674e624257f84c47acff5d61f1fd5b54" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:14.668599 9860 leader_election.cc:343] T 674e624257f84c47acff5d61f1fd5b54 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.668889 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c6a9ddd35e0549cc81fe330827797023"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:14.669620 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2f6ebdb6b30146d3af2e3911749f8c9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:14.669706 10040 ts_tablet_manager.cc:1929] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:14.669813 10040 log.cc:1199] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/5d18ff6d8ddb465fbf479fdc7ebe7a05
I20251027 21:08:14.670198 10040 ts_tablet_manager.cc:1950] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:14.671463 9310 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet dd9e6656d8df43afbb7f70706bc14585 (table default.Table_f78d839900634631b6963b2d63f0d084 [id=3cdf432ec43d42e8bc2d253a92eac8d3]). Expected version 3 got 0
I20251027 21:08:14.671716 9310 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 67cb71ce43244d8ab9e71ead6bb315c9 (table default.Table_6327cacf4ed54d56b522291ad9594784 [id=3eb54d245ec64ee48022c541c2e49b17]). Expected version 2 got 0
W20251027 21:08:14.674278 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c8bffe5fd42431fa97808af0d8c634d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:14.674763 10040 tablet_replica.cc:333] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:14.675014 10040 raft_consensus.cc:2243] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:14.675225 10040 raft_consensus.cc:2272] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:14.675586 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 (table default.Table_f99fba6d68834ef4b426a23a2ca11c9e [id=e6e865b72bbc4d01867f19308bdd343b]) successfully deleted
I20251027 21:08:14.677243 10040 ts_tablet_manager.cc:1916] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:14.682951 10032 tablet_bootstrap.cc:492] T ed66f9eeeb2b4b4bb77062c241d2140f P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:14.683081 10032 ts_tablet_manager.cc:1403] T ed66f9eeeb2b4b4bb77062c241d2140f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.190s user 0.003s sys 0.000s
I20251027 21:08:14.683794 10032 raft_consensus.cc:359] T ed66f9eeeb2b4b4bb77062c241d2140f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:14.683975 10032 raft_consensus.cc:385] T ed66f9eeeb2b4b4bb77062c241d2140f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.684026 10032 raft_consensus.cc:740] T ed66f9eeeb2b4b4bb77062c241d2140f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.684254 10032 consensus_queue.cc:260] T ed66f9eeeb2b4b4bb77062c241d2140f P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:14.689867 10032 ts_tablet_manager.cc:1434] T ed66f9eeeb2b4b4bb77062c241d2140f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251027 21:08:14.690968 10032 tablet_bootstrap.cc:492] T 3ee02a05b5ae46e7bf5cf9fdd3e9a80f P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:14.692502 10032 tablet_bootstrap.cc:654] T 3ee02a05b5ae46e7bf5cf9fdd3e9a80f P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.694187 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "674e624257f84c47acff5d61f1fd5b54" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:14.694661 9859 leader_election.cc:343] T 674e624257f84c47acff5d61f1fd5b54 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.694792 9859 leader_election.cc:304] T 674e624257f84c47acff5d61f1fd5b54 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:14.695183 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6a9ddd35e0549cc81fe330827797023 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:14.697417 11621 raft_consensus.cc:2749] T 674e624257f84c47acff5d61f1fd5b54 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.698329 10041 tablet_replica.cc:333] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:14.698529 10041 raft_consensus.cc:2243] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:14.698689 10041 raft_consensus.cc:2272] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:14.699129 10029 tablet_bootstrap.cc:492] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:14.699203 10041 ts_tablet_manager.cc:1916] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:14.699545 10029 ts_tablet_manager.cc:1403] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.233s user 0.015s sys 0.024s
I20251027 21:08:14.702185 10029 raft_consensus.cc:359] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.702713 11657 raft_consensus.cc:493] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.704712 11657 raft_consensus.cc:515] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.704936 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b8156d9ab2e4dfba383ddc72137aab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:14.705819 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet aaa8910d9fdb4146945621ec4ebf969e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:14.705797 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76bee322d0b94abe9166170733608853" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:14.706213 9566 leader_election.cc:343] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.706228 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet d91fced778784b65be2486e24426d1b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:14.706487 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c734e20b87ce4c62a2c1b8851783c8b1 consensus state has no leader
I20251027 21:08:14.706605 11657 leader_election.cc:290] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.706720 9312 master_service.cc:954] ReplaceTablet: received request to replace tablet 5e20fb964a4543ff9eda684acc8eb2bb from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:14.707201 9309 master_service.cc:954] ReplaceTablet: received request to replace tablet 4bbb69d62b9a41e3b88a5527f7549881 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:14.707964 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9a0390f5f62d4b9da2fbc3425e8781f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:14.708629 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet e8fe63ea8aba4614900ac254189464de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:14.703647 10029 raft_consensus.cc:385] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.711635 10029 raft_consensus.cc:740] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.712519 10029 consensus_queue.cc:260] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.714573 10029 ts_tablet_manager.cc:1434] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.015s user 0.002s sys 0.001s
I20251027 21:08:14.715999 10029 tablet_bootstrap.cc:492] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:14.716315 11626 raft_consensus.cc:493] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.716655 11626 raft_consensus.cc:515] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.717577 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 96a680e8f16343b1908829a67e960b60 (table default.Table_a46825cc888a4a0898d6db70b02e5ab6 [id=8579d3ff46d64c75b1da213fdee9cc1b]) successfully deleted
I20251027 21:08:14.718005 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67cb71ce43244d8ab9e71ead6bb315c9" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:14.718178 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 631ff72613734913818a3cad24f20dbc (table default.Table_ed23972576b74b5fbc7167cf7b541bfc [id=e7b7bdb2ea7f468c98bfe88f900a5be8]) successfully deleted
I20251027 21:08:14.720081 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67cb71ce43244d8ab9e71ead6bb315c9" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:14.720085 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_c572071929f9429f9d8f7bfa30fbf964"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:14.720562 9711 leader_election.cc:304] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:14.721596 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet a2d26784bbb24ae3893103802c41affb (table default.Table_9bf7b754a163446db873b9b81a700e4d [id=54ab773e30384ae98a0b0539284c85b4]) successfully deleted
I20251027 21:08:14.722257 11676 raft_consensus.cc:2749] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.723016 11626 leader_election.cc:290] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:14.723058 9298 consensus_peers.cc:597] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 -> Peer 18b15927ce5744ad8ba81dda52725c11 (127.9.14.125:46313): Couldn't send request to peer 18b15927ce5744ad8ba81dda52725c11. Status: Network error: Client connection negotiation failed: client connection to 127.9.14.125:46313: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251027 21:08:14.725716 10043 ts_tablet_manager.cc:1929] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251027 21:08:14.726715 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c572071929f9429f9d8f7bfa30fbf964 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:14.726773 10031 tablet_bootstrap.cc:492] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:14.726913 10031 ts_tablet_manager.cc:1403] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.167s user 0.039s sys 0.069s
I20251027 21:08:14.727596 10031 raft_consensus.cc:359] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:14.727711 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc77fcc2c86242789d1775592ee6a417 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:14.727831 10031 raft_consensus.cc:385] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.727901 10031 raft_consensus.cc:740] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.728096 10031 consensus_queue.cc:260] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.728552 10031 ts_tablet_manager.cc:1434] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20251027 21:08:14.728607 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a0390f5f62d4b9da2fbc3425e8781f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:14.728825 10031 tablet_bootstrap.cc:492] T 986bf148b3e9482fa2fe53bbac54750d P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:08:14.729463 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f6ebdb6b30146d3af2e3911749f8c9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:14.730297 10031 tablet_bootstrap.cc:654] T 986bf148b3e9482fa2fe53bbac54750d P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.730922 10043 log.cc:1199] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/5d18ff6d8ddb465fbf479fdc7ebe7a05
W20251027 21:08:14.730733 9568 leader_election.cc:343] T 42371496a7494e77a7bf1169de5b083a P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 42371496a7494e77a7bf1169de5b083a
I20251027 21:08:14.731789 9568 leader_election.cc:304] T 42371496a7494e77a7bf1169de5b083a P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.732174 11657 raft_consensus.cc:2749] T 42371496a7494e77a7bf1169de5b083a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.732664 10043 ts_tablet_manager.cc:1950] T 5d18ff6d8ddb465fbf479fdc7ebe7a05 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:14.736171 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet f6789dc2f7cb4a3c826ac507d5c9a239 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.736367 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 7292bc6c22ca4ab5b8f53a69520833ba with delete_type TABLET_DATA_DELETED (replaced by tablet 37ae3b6bf590473e8e09092fc5f7df1b) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.736457 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 90f96c21da9944f1ade05d35787d47e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.736618 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ed4f9c61ad54e1292281b30bcd19533 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.736688 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 7425b6b0184e4d0b871777b25d904ac4 with delete_type TABLET_DATA_DELETED (replaced by tablet 070167be7d8d4ddf8da561df14448360) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.736804 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet aaaf70c950044387a63820079db3c829 with delete_type TABLET_DATA_DELETED (replaced by tablet 262b4fb71a2d40fc879da56f8c51ef83) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.736904 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.744654 10040 ts_tablet_manager.cc:1929] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:14.745540 10029 tablet_bootstrap.cc:654] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.748649 10040 log.cc:1199] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/588e2f58f07b4a32a78dffd5b92ceb12
I20251027 21:08:14.749188 10040 ts_tablet_manager.cc:1950] T 588e2f58f07b4a32a78dffd5b92ceb12 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:14.749792 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 588e2f58f07b4a32a78dffd5b92ceb12 (table default.Table_7a3ebf1ecf35428495d2d7bb5540530a [id=c4838f30ee9f4aec96e6c73b4ce15c7b]) successfully deleted
I20251027 21:08:14.750602 10031 tablet_bootstrap.cc:492] T 986bf148b3e9482fa2fe53bbac54750d P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:14.750768 10031 ts_tablet_manager.cc:1403] T 986bf148b3e9482fa2fe53bbac54750d P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.006s
I20251027 21:08:14.751480 10031 raft_consensus.cc:359] T 986bf148b3e9482fa2fe53bbac54750d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.751691 10031 raft_consensus.cc:385] T 986bf148b3e9482fa2fe53bbac54750d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.751793 10031 raft_consensus.cc:740] T 986bf148b3e9482fa2fe53bbac54750d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.752048 10031 consensus_queue.cc:260] T 986bf148b3e9482fa2fe53bbac54750d P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.752193 11512 raft_consensus.cc:493] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.752336 11512 raft_consensus.cc:515] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.752967 11512 leader_election.cc:290] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.752969 11626 raft_consensus.cc:493] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.753094 11626 raft_consensus.cc:515] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.753608 11626 leader_election.cc:290] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:14.754974 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30ce703650c94c06b93bce976cb5d1fd" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:14.755142 9711 leader_election.cc:343] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 90f96c21da9944f1ade05d35787d47e2
I20251027 21:08:14.755162 9771 raft_consensus.cc:2468] T 30ce703650c94c06b93bce976cb5d1fd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:14.755621 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76bee322d0b94abe9166170733608853" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:14.755815 9909 raft_consensus.cc:2468] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:14.756194 10040 tablet_replica.cc:333] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:14.756219 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90f96c21da9944f1ade05d35787d47e2" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:14.757591 9566 leader_election.cc:304] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:14.757702 10040 raft_consensus.cc:2243] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:14.757318 9710 leader_election.cc:304] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:14.757866 10040 raft_consensus.cc:2272] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:14.758047 11621 raft_consensus.cc:493] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.758191 11621 raft_consensus.cc:515] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:14.758390 11626 raft_consensus.cc:2749] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.758832 11621 leader_election.cc:290] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.758877 10031 ts_tablet_manager.cc:1434] T 986bf148b3e9482fa2fe53bbac54750d P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20251027 21:08:14.759004 10040 ts_tablet_manager.cc:1916] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:14.759544 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5054bbc3e27243298b4df6b0a87af8be" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:14.759756 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5054bbc3e27243298b4df6b0a87af8be" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:14.760013 9860 leader_election.cc:343] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:14.760272 9859 leader_election.cc:343] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.760304 11657 raft_consensus.cc:2804] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:14.760375 11657 raft_consensus.cc:493] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:14.760372 9859 leader_election.cc:304] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.760430 11657 raft_consensus.cc:3060] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.760812 10031 tablet_bootstrap.cc:492] T 28a2e85021394f9d8a9ea524959b52a1 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:14.761072 11621 raft_consensus.cc:2749] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.761159 11512 raft_consensus.cc:493] T d8691aa22fe84cf29893ed4e79d156cc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.761250 11512 raft_consensus.cc:515] T d8691aa22fe84cf29893ed4e79d156cc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.761802 11512 leader_election.cc:290] T d8691aa22fe84cf29893ed4e79d156cc P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.762616 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1117) took 5626 ms. Trace:
I20251027 21:08:14.762693 10043 rpcz_store.cc:276] 1027 21:08:09.136217 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.136672 (+ 455us) service_pool.cc:225] Handling call
1027 21:08:14.562302 (+5425630us) ts_tablet_manager.cc:1151] Deleting tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05
1027 21:08:14.562314 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:14.591558 (+ 29244us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:14.725592 (+134034us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:14.762601 (+ 37009us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28967,"tablet-delete.queue_time_us":5425427}
I20251027 21:08:14.763018 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8691aa22fe84cf29893ed4e79d156cc" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:14.763340 10043 tablet_replica.cc:333] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
W20251027 21:08:14.763411 9566 leader_election.cc:343] T d8691aa22fe84cf29893ed4e79d156cc P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.763563 10043 raft_consensus.cc:2243] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:14.763886 10043 pending_rounds.cc:70] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14: Trying to abort 1 pending ops.
I20251027 21:08:14.763978 10043 pending_rounds.cc:77] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510688991195136 op_type: NO_OP noop_request { }
I20251027 21:08:14.764099 10043 raft_consensus.cc:2889] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:14.764217 10043 raft_consensus.cc:2272] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:14.764752 10031 tablet_bootstrap.cc:654] T 28a2e85021394f9d8a9ea524959b52a1 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.765102 10043 ts_tablet_manager.cc:1916] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:14.765286 11657 raft_consensus.cc:515] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.766206 11657 leader_election.cc:290] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.768571 10029 tablet_bootstrap.cc:492] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:14.769177 10029 ts_tablet_manager.cc:1403] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.002s
I20251027 21:08:14.770372 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30ce703650c94c06b93bce976cb5d1fd" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:14.770604 9773 raft_consensus.cc:3060] T 30ce703650c94c06b93bce976cb5d1fd P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.771780 9568 leader_election.cc:304] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.771997 11657 raft_consensus.cc:2804] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:14.772060 11657 raft_consensus.cc:493] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:14.772116 11657 raft_consensus.cc:3060] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.773303 9773 raft_consensus.cc:2468] T 30ce703650c94c06b93bce976cb5d1fd P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:14.773590 11626 raft_consensus.cc:493] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.773727 11626 raft_consensus.cc:515] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.773833 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c26460a4ac0742d997bcc2dc9d88ce5e" } new_table_name: "default.Table_fbdc5a062a6644b4a37c8b35ef66cda0" modify_external_catalogs: true
I20251027 21:08:14.774295 11626 leader_election.cc:290] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:14.774533 9566 leader_election.cc:304] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:14.774796 11411 raft_consensus.cc:2804] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:14.774878 11657 raft_consensus.cc:515] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.775195 9309 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_10376e17301944fe9c799d6a87ff1f34" } new_table_name: "default.Table_8faf246fd4704277852aa24e7304f564" modify_external_catalogs: true
I20251027 21:08:14.775281 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7425b6b0184e4d0b871777b25d904ac4" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:14.775419 9634 raft_consensus.cc:2468] T 7425b6b0184e4d0b871777b25d904ac4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:08:14.774890 11411 raft_consensus.cc:697] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:14.775857 9710 leader_election.cc:304] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:14.775864 11411 consensus_queue.cc:237] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.776163 11657 leader_election.cc:290] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.776297 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a6cf2819ed9241cea09738477423c84a" } new_table_name: "default.Table_3f494d4168ee473183b3d7d09f4ce3ba" modify_external_catalogs: true
I20251027 21:08:14.776937 11626 raft_consensus.cc:2804] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:14.777006 11626 raft_consensus.cc:493] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:14.777055 11626 raft_consensus.cc:3060] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.777349 10032 tablet_bootstrap.cc:492] T 3ee02a05b5ae46e7bf5cf9fdd3e9a80f P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:14.777473 10032 ts_tablet_manager.cc:1403] T 3ee02a05b5ae46e7bf5cf9fdd3e9a80f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.087s user 0.004s sys 0.000s
I20251027 21:08:14.778522 10032 raft_consensus.cc:359] T 3ee02a05b5ae46e7bf5cf9fdd3e9a80f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.778725 10032 raft_consensus.cc:385] T 3ee02a05b5ae46e7bf5cf9fdd3e9a80f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.778777 10032 raft_consensus.cc:740] T 3ee02a05b5ae46e7bf5cf9fdd3e9a80f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.779263 10032 consensus_queue.cc:260] T 3ee02a05b5ae46e7bf5cf9fdd3e9a80f P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.779718 10032 ts_tablet_manager.cc:1434] T 3ee02a05b5ae46e7bf5cf9fdd3e9a80f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:14.779654 11626 raft_consensus.cc:515] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:14.780073 9568 leader_election.cc:343] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 30ce703650c94c06b93bce976cb5d1fd
I20251027 21:08:14.780637 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7425b6b0184e4d0b871777b25d904ac4" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:14.780778 9634 raft_consensus.cc:3060] T 7425b6b0184e4d0b871777b25d904ac4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.783234 9634 raft_consensus.cc:2468] T 7425b6b0184e4d0b871777b25d904ac4 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:14.783252 9303 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_368703da645247fb87af3db268932e92" } modify_external_catalogs: true
I20251027 21:08:14.783389 9303 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_368703da645247fb87af3db268932e92" } modify_external_catalogs: true
I20251027 21:08:14.784071 11626 leader_election.cc:290] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:14.784276 9710 leader_election.cc:304] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:14.785125 11676 raft_consensus.cc:2804] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:14.785300 11676 raft_consensus.cc:697] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:14.785641 11676 consensus_queue.cc:237] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:14.787019 9711 leader_election.cc:343] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 7425b6b0184e4d0b871777b25d904ac4
I20251027 21:08:14.787775 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76bee322d0b94abe9166170733608853" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:14.788262 9566 leader_election.cc:343] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:14.789060 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5bc30731304648d8bc702cf7793b8d22 consensus state has no leader
I20251027 21:08:14.789434 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f1f34db7d6324d4c8cb9f5a8922e27b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:14.790990 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1f34db7d6324d4c8cb9f5a8922e27b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:14.792889 9310 master_service.cc:954] ReplaceTablet: received request to replace tablet cf1751af4c774d078a4aede9d2bb8108 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:14.793831 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet cb95618d9fd54a4f8121b43e717d5616 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.794035 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 30ce703650c94c06b93bce976cb5d1fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.794044 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 (table default.Table_f99fba6d68834ef4b426a23a2ca11c9e [id=e6e865b72bbc4d01867f19308bdd343b]) successfully deleted
I20251027 21:08:14.794240 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet f4187b2f66f24055b1df170f1062cc88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.795670 11512 raft_consensus.cc:493] T 6210d3d10fc44b9e9e2b0780883186ce P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.795797 11512 raft_consensus.cc:515] T 6210d3d10fc44b9e9e2b0780883186ce P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.796322 11512 leader_election.cc:290] T 6210d3d10fc44b9e9e2b0780883186ce P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:14.799044 9309 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 538c1a8045da4864ab2f4c8b0c672f29 consensus state has no leader
I20251027 21:08:14.799252 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6210d3d10fc44b9e9e2b0780883186ce" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:14.799688 9566 leader_election.cc:343] T 6210d3d10fc44b9e9e2b0780883186ce P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.801717 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76bee322d0b94abe9166170733608853" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:14.801910 9909 raft_consensus.cc:3060] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:14.802554 9568 leader_election.cc:343] T d8691aa22fe84cf29893ed4e79d156cc P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: d8691aa22fe84cf29893ed4e79d156cc
I20251027 21:08:14.802662 9568 leader_election.cc:304] T d8691aa22fe84cf29893ed4e79d156cc P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.802897 11512 raft_consensus.cc:2749] T d8691aa22fe84cf29893ed4e79d156cc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.803237 10029 raft_consensus.cc:359] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.803370 10029 raft_consensus.cc:385] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.803411 10029 raft_consensus.cc:740] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.803544 10029 consensus_queue.cc:260] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.803938 10029 ts_tablet_manager.cc:1434] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.034s user 0.000s sys 0.000s
I20251027 21:08:14.805248 11657 raft_consensus.cc:493] T 00c90ff84d3f4a7f92fda235416eed1d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.805315 9909 raft_consensus.cc:2468] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:14.805367 11657 raft_consensus.cc:515] T 00c90ff84d3f4a7f92fda235416eed1d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.805771 11621 raft_consensus.cc:493] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.805896 11657 leader_election.cc:290] T 00c90ff84d3f4a7f92fda235416eed1d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.805900 11621 raft_consensus.cc:515] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.806502 11621 leader_election.cc:290] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:14.808396 9568 leader_election.cc:343] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 30ce703650c94c06b93bce976cb5d1fd
I20251027 21:08:14.808660 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c548d2413a5940c8b114ced667b4dc41" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:14.808955 9568 leader_election.cc:343] T 6210d3d10fc44b9e9e2b0780883186ce P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 6210d3d10fc44b9e9e2b0780883186ce
I20251027 21:08:14.809038 9568 leader_election.cc:304] T 6210d3d10fc44b9e9e2b0780883186ce P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:14.809098 9859 leader_election.cc:343] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.809202 11512 raft_consensus.cc:2749] T 6210d3d10fc44b9e9e2b0780883186ce P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:14.809456 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1dc45f9d5219461b9fca93a7719d3f6c consensus state has no leader
I20251027 21:08:14.809593 9568 leader_election.cc:304] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.809798 11512 raft_consensus.cc:2804] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:14.809911 11512 raft_consensus.cc:697] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:14.810092 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00c90ff84d3f4a7f92fda235416eed1d" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:14.810292 11512 consensus_queue.cc:237] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:14.810500 9566 leader_election.cc:343] T 00c90ff84d3f4a7f92fda235416eed1d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:14.814713 9711 leader_election.cc:343] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 7425b6b0184e4d0b871777b25d904ac4
I20251027 21:08:14.815253 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_152a2bed589f47858b8a99ab7df9984c" } modify_external_catalogs: true
I20251027 21:08:14.815430 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_152a2bed589f47858b8a99ab7df9984c" } modify_external_catalogs: true
I20251027 21:08:14.815848 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f63a59e9a2bb49b29a8477d90871afde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:14.816300 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f63a59e9a2bb49b29a8477d90871afde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:14.824718 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_55c1322435df4656993fef22102a4f62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:14.825290 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55c1322435df4656993fef22102a4f62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:14.826753 10032 tablet_bootstrap.cc:492] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:14.828141 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c548d2413a5940c8b114ced667b4dc41" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:14.828676 9860 leader_election.cc:343] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.828773 9860 leader_election.cc:304] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.829067 11543 raft_consensus.cc:2749] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.831226 10029 tablet_bootstrap.cc:492] T 4ba13a89a33549af921d3b55023faf15 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:14.832574 10029 tablet_bootstrap.cc:654] T 4ba13a89a33549af921d3b55023faf15 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.833564 10041 ts_tablet_manager.cc:1929] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:14.833835 11512 raft_consensus.cc:493] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.834018 11512 raft_consensus.cc:515] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.834751 11512 leader_election.cc:290] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.835446 10040 ts_tablet_manager.cc:1929] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251027 21:08:14.837179 9568 leader_election.cc:343] T 00c90ff84d3f4a7f92fda235416eed1d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 00c90ff84d3f4a7f92fda235416eed1d
I20251027 21:08:14.837384 9568 leader_election.cc:304] T 00c90ff84d3f4a7f92fda235416eed1d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.837630 11512 raft_consensus.cc:2749] T 00c90ff84d3f4a7f92fda235416eed1d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.835681 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "334c580862bd4bf09be3f1710aa6a092" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:14.841331 9909 raft_consensus.cc:2468] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:14.841763 9568 leader_election.cc:304] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:14.836639 10032 tablet_bootstrap.cc:654] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.843003 10041 log.cc:1199] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/43611e33512b49d58485cf15de788be7
I20251027 21:08:14.845568 11512 raft_consensus.cc:2804] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:14.845763 11512 raft_consensus.cc:493] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:14.845917 11512 raft_consensus.cc:3060] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.850311 11512 raft_consensus.cc:515] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.851199 11512 leader_election.cc:290] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.851789 9304 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 371051fb3491426e8cf7797317baa3db (table default.Table_d481a540123e4620874b629ce6db65a7 [id=3b0044836ccc4b4082775de6eb111e49]). Expected version 1 got 0
I20251027 21:08:14.852121 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "334c580862bd4bf09be3f1710aa6a092" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:14.852015 9304 catalog_manager.cc:5649] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:14.852279 9909 raft_consensus.cc:3060] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.852370 9304 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 8cccd253ae2a46328dc9f767b8e9ee97 (table default.Table_4ba26495af344e3cb082a4b781f1b208 [id=501b230a824d4a17bb1903c8409407b7]). Expected version 2 got 0
I20251027 21:08:14.853684 9311 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 72c1dc3b9eff47ee81df821deb789307 (table default.Table_ff5458dda9f042e189bd8cd69c06d025 [id=2a96776404334cf79ddb611a234a5844]). Expected version 2 got 0
I20251027 21:08:14.853895 9311 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 9ace4188d3494a05a5678381f8d8fd17 (table default.Table_9bf7b754a163446db873b9b81a700e4d [id=54ab773e30384ae98a0b0539284c85b4]). Expected version 2 got 0
I20251027 21:08:14.857158 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet 0820648719a34f379507efced9cc434b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.857208 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet 5054bbc3e27243298b4df6b0a87af8be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.857357 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet fe7b00e9cf984d12817c9ea860391199 with delete_type TABLET_DATA_DELETED (replaced by tablet 023eb8eb693d446caef49e0eaecd147c) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.857414 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet 479d2cc8b8ce4c26841f50eea7c0e09a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.856770 9909 raft_consensus.cc:2468] T 334c580862bd4bf09be3f1710aa6a092 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:14.858601 10041 ts_tablet_manager.cc:1950] T 43611e33512b49d58485cf15de788be7 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:14.858827 9568 leader_election.cc:304] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:14.859124 11512 raft_consensus.cc:2804] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:14.859301 11512 raft_consensus.cc:697] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:14.860570 11512 consensus_queue.cc:237] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.862882 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 43611e33512b49d58485cf15de788be7 (table default.Table_02a5b78b84f1409b80d09d02e5069b9c [id=b3524d24371a43038db124bb4f0c5793]) successfully deleted
I20251027 21:08:14.862983 10041 tablet_replica.cc:333] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:14.863085 10032 tablet_bootstrap.cc:492] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:14.865526 10032 ts_tablet_manager.cc:1403] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.001s
I20251027 21:08:14.866197 10032 raft_consensus.cc:359] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:14.867259 9566 consensus_peers.cc:597] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:14.868163 10043 ts_tablet_manager.cc:1929] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:14.868623 10032 raft_consensus.cc:385] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.868752 10032 raft_consensus.cc:740] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.863161 10041 raft_consensus.cc:2243] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:14.868990 10032 consensus_queue.cc:260] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.869122 10041 raft_consensus.cc:2272] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:14.870149 10032 ts_tablet_manager.cc:1434] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251027 21:08:14.870462 10041 ts_tablet_manager.cc:1916] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:14.870571 9311 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 3ee02a05b5ae46e7bf5cf9fdd3e9a80f (table default.Table_fbcff2da42d54b38a090a042559bad38 [id=8c32f511b11d422fa461408a4e9d4e0f]). Expected version 2 got 0
I20251027 21:08:14.872200 10032 tablet_bootstrap.cc:492] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:14.872455 10043 log.cc:1199] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/07a9c3b7ef1548ec9d5ae9f20545f263
I20251027 21:08:14.874527 10043 ts_tablet_manager.cc:1950] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:14.874533 11543 raft_consensus.cc:493] T 3ee02a05b5ae46e7bf5cf9fdd3e9a80f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.874637 11543 raft_consensus.cc:515] T 3ee02a05b5ae46e7bf5cf9fdd3e9a80f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.875166 11543 leader_election.cc:290] T 3ee02a05b5ae46e7bf5cf9fdd3e9a80f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.877132 10032 tablet_bootstrap.cc:654] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.876264 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ee02a05b5ae46e7bf5cf9fdd3e9a80f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:14.877898 9860 leader_election.cc:343] T 3ee02a05b5ae46e7bf5cf9fdd3e9a80f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.878888 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1118) took 5742 ms. Trace:
I20251027 21:08:14.879580 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_685b53b410ef4403890d1494d6d77042" } modify_external_catalogs: true
I20251027 21:08:14.879726 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_685b53b410ef4403890d1494d6d77042" } modify_external_catalogs: true
I20251027 21:08:14.882341 10043 rpcz_store.cc:276] 1027 21:08:09.136299 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.136905 (+ 606us) service_pool.cc:225] Handling call
1027 21:08:14.763300 (+5626395us) ts_tablet_manager.cc:1151] Deleting tablet 07a9c3b7ef1548ec9d5ae9f20545f263
1027 21:08:14.763309 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:14.843272 (+ 79963us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:14.868063 (+ 24791us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:14.878869 (+ 10806us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5626245}
I20251027 21:08:14.883010 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 07a9c3b7ef1548ec9d5ae9f20545f263 (table default.Table_a7632420526d409290924efb41972ee1 [id=4a103f97fc964eaba06cd48d4ba854a6]) successfully deleted
I20251027 21:08:14.836334 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "334c580862bd4bf09be3f1710aa6a092" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:14.885478 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet c3f4e692c08244afa2c79e0a075bbcdb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.837071 11676 raft_consensus.cc:493] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.851539 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "334c580862bd4bf09be3f1710aa6a092" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:14.861069 10031 tablet_bootstrap.cc:492] T 28a2e85021394f9d8a9ea524959b52a1 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:14.885685 11676 raft_consensus.cc:515] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.885794 10031 ts_tablet_manager.cc:1403] T 28a2e85021394f9d8a9ea524959b52a1 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.127s user 0.010s sys 0.013s
I20251027 21:08:14.886092 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1119) took 5747 ms. Trace:
I20251027 21:08:14.861974 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet 952583fd7fde4b4baccdb98c01bdf9e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.862612 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 842c1a1298234d9ea2719bab29c54ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.872232 9736 tablet_service.cc:1505] Processing CreateTablet for tablet f32a2c6ab909477997b0bbc71e7414a9 (DEFAULT_TABLE table=default.Table_676ac104a790481d9043a383d7c96caf [id=75babbf4c35348a889479b2bd4531d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.886524 10031 raft_consensus.cc:359] T 28a2e85021394f9d8a9ea524959b52a1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.886345 10043 rpcz_store.cc:276] 1027 21:08:09.136382 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.137076 (+ 694us) service_pool.cc:225] Handling call
1027 21:08:14.883990 (+5746914us) ts_tablet_manager.cc:1151] Deleting tablet 8b30052ff19744a9bf4bfaaa1bc77b36
1027 21:08:14.884001 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:14.884069 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5746780}
I20251027 21:08:14.886685 10031 raft_consensus.cc:385] T 28a2e85021394f9d8a9ea524959b52a1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.886734 10031 raft_consensus.cc:740] T 28a2e85021394f9d8a9ea524959b52a1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.886806 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f32a2c6ab909477997b0bbc71e7414a9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:14.886919 9298 catalog_manager.cc:4952] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 8b30052ff19744a9bf4bfaaa1bc77b36 because the tablet was not found. No further retry: Not found: Tablet not found: 8b30052ff19744a9bf4bfaaa1bc77b36
I20251027 21:08:14.886895 10031 consensus_queue.cc:260] T 28a2e85021394f9d8a9ea524959b52a1 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.887310 10031 ts_tablet_manager.cc:1434] T 28a2e85021394f9d8a9ea524959b52a1 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 21:08:14.887583 10031 tablet_bootstrap.cc:492] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:14.888507 10043 tablet_replica.cc:333] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:14.888680 10043 raft_consensus.cc:2243] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:14.888836 10031 tablet_bootstrap.cc:654] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.888978 10043 raft_consensus.cc:2272] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:14.875735 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ee02a05b5ae46e7bf5cf9fdd3e9a80f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:14.890493 9859 leader_election.cc:343] T 3ee02a05b5ae46e7bf5cf9fdd3e9a80f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.890614 9859 leader_election.cc:304] T 3ee02a05b5ae46e7bf5cf9fdd3e9a80f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:14.890638 9566 leader_election.cc:343] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:14.890898 9566 leader_election.cc:343] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.890959 11543 raft_consensus.cc:2749] T 3ee02a05b5ae46e7bf5cf9fdd3e9a80f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.891423 9775 raft_consensus.cc:3060] T dd9e6656d8df43afbb7f70706bc14585 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.891431 10031 tablet_bootstrap.cc:492] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:14.891637 10031 ts_tablet_manager.cc:1403] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251027 21:08:14.892321 10031 raft_consensus.cc:359] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.892520 10031 raft_consensus.cc:385] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.892594 10031 raft_consensus.cc:740] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.892777 10031 consensus_queue.cc:260] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.893309 10031 ts_tablet_manager.cc:1434] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:14.893563 10031 tablet_bootstrap.cc:492] T 8cccd253ae2a46328dc9f767b8e9ee97 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:14.894786 10031 tablet_bootstrap.cc:654] T 8cccd253ae2a46328dc9f767b8e9ee97 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.894816 11676 leader_election.cc:290] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:14.895787 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaaf70c950044387a63820079db3c829" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:14.896354 9711 leader_election.cc:343] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: aaaf70c950044387a63820079db3c829
I20251027 21:08:14.896454 9711 leader_election.cc:304] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:14.896839 11676 raft_consensus.cc:2749] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.897470 11658 raft_consensus.cc:493] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.897583 11658 raft_consensus.cc:515] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.898105 11658 leader_election.cc:290] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.898785 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "371051fb3491426e8cf7797317baa3db" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:14.893893 10043 ts_tablet_manager.cc:1916] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:14.898953 9909 raft_consensus.cc:2468] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:14.899818 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "371051fb3491426e8cf7797317baa3db" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:14.900171 9566 leader_election.cc:343] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.901014 9568 leader_election.cc:304] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.901233 11658 raft_consensus.cc:2804] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:14.901301 11658 raft_consensus.cc:493] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:14.901360 11658 raft_consensus.cc:3060] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.901995 10040 log.cc:1199] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/7b8156d9ab2e4dfba383ddc72137aab4
I20251027 21:08:14.902395 10040 ts_tablet_manager.cc:1950] T 7b8156d9ab2e4dfba383ddc72137aab4 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:14.902774 10029 tablet_bootstrap.cc:492] T 4ba13a89a33549af921d3b55023faf15 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:14.902882 10029 ts_tablet_manager.cc:1403] T 4ba13a89a33549af921d3b55023faf15 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.072s user 0.000s sys 0.002s
I20251027 21:08:14.903671 10029 raft_consensus.cc:359] T 4ba13a89a33549af921d3b55023faf15 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.903810 10029 raft_consensus.cc:385] T 4ba13a89a33549af921d3b55023faf15 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.903856 10029 raft_consensus.cc:740] T 4ba13a89a33549af921d3b55023faf15 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.904031 10029 consensus_queue.cc:260] T 4ba13a89a33549af921d3b55023faf15 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.904500 10029 ts_tablet_manager.cc:1434] T 4ba13a89a33549af921d3b55023faf15 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 21:08:14.904438 9887 tablet_service.cc:1505] Processing CreateTablet for tablet f32a2c6ab909477997b0bbc71e7414a9 (DEFAULT_TABLE table=default.Table_676ac104a790481d9043a383d7c96caf [id=75babbf4c35348a889479b2bd4531d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.904749 10029 tablet_bootstrap.cc:492] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:14.904820 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f32a2c6ab909477997b0bbc71e7414a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.904896 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 7b8156d9ab2e4dfba383ddc72137aab4 (table default.Table_d5b5337ae438439d984f34a4e0af6c93 [id=8932a6e05aae478ab0a22e213896851f]) successfully deleted
I20251027 21:08:14.906067 10029 tablet_bootstrap.cc:654] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.907837 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 0c88e7aecf3f4e28987158f725e093da (DEFAULT_TABLE table=default.Table_deb3d240fb84407e810d7077b5bda502 [id=031d596510444fbabfb5cf06ea40e339]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.908160 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c88e7aecf3f4e28987158f725e093da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.914652 11658 raft_consensus.cc:515] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.915316 11658 leader_election.cc:290] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.916217 10031 tablet_bootstrap.cc:492] T 8cccd253ae2a46328dc9f767b8e9ee97 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:14.916364 10031 ts_tablet_manager.cc:1403] T 8cccd253ae2a46328dc9f767b8e9ee97 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20251027 21:08:14.916985 10031 raft_consensus.cc:359] T 8cccd253ae2a46328dc9f767b8e9ee97 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.917140 10031 raft_consensus.cc:385] T 8cccd253ae2a46328dc9f767b8e9ee97 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.917202 10031 raft_consensus.cc:740] T 8cccd253ae2a46328dc9f767b8e9ee97 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.917464 10031 consensus_queue.cc:260] T 8cccd253ae2a46328dc9f767b8e9ee97 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.918022 10031 ts_tablet_manager.cc:1434] T 8cccd253ae2a46328dc9f767b8e9ee97 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:14.918602 10031 tablet_bootstrap.cc:492] T 72c1dc3b9eff47ee81df821deb789307 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:14.920703 9602 tablet_service.cc:1505] Processing CreateTablet for tablet f32a2c6ab909477997b0bbc71e7414a9 (DEFAULT_TABLE table=default.Table_676ac104a790481d9043a383d7c96caf [id=75babbf4c35348a889479b2bd4531d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.921263 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "371051fb3491426e8cf7797317baa3db" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:14.921422 9909 raft_consensus.cc:3060] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.922344 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f32a2c6ab909477997b0bbc71e7414a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.924276 9909 raft_consensus.cc:2468] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:14.928566 10031 tablet_bootstrap.cc:654] T 72c1dc3b9eff47ee81df821deb789307 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.928545 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "371051fb3491426e8cf7797317baa3db" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:14.929049 9566 leader_election.cc:343] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.929252 9568 leader_election.cc:304] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.929709 11658 raft_consensus.cc:2804] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:14.929812 11658 raft_consensus.cc:697] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:14.929991 10041 ts_tablet_manager.cc:1929] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:14.930075 10041 log.cc:1199] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/e477ca4b69c84327b04618d90406e19a
I20251027 21:08:14.930008 11658 consensus_queue.cc:237] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.930341 10041 ts_tablet_manager.cc:1950] T e477ca4b69c84327b04618d90406e19a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:14.931159 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet e477ca4b69c84327b04618d90406e19a (table default.Table_7392db44fc434fd1a795922799aaf639 [id=2b5a90a29b994f1bad6491e84865172d]) successfully deleted
I20251027 21:08:14.935369 10032 tablet_bootstrap.cc:492] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:14.935778 10032 ts_tablet_manager.cc:1403] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.064s user 0.009s sys 0.000s
I20251027 21:08:14.937000 10032 raft_consensus.cc:359] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.937733 10032 raft_consensus.cc:385] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.937911 10032 raft_consensus.cc:740] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.938306 10032 consensus_queue.cc:260] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.940218 10032 ts_tablet_manager.cc:1434] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251027 21:08:14.935619 10041 tablet_replica.cc:333] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:14.944485 10029 tablet_bootstrap.cc:492] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:14.944617 10029 ts_tablet_manager.cc:1403] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.000s
I20251027 21:08:14.945183 10029 raft_consensus.cc:359] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.945334 10029 raft_consensus.cc:385] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.945390 10029 raft_consensus.cc:740] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.945559 10029 consensus_queue.cc:260] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.946026 10029 ts_tablet_manager.cc:1434] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:14.946316 10029 tablet_bootstrap.cc:492] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:14.946357 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 107c608f9d904d86b74e2ad682e1bdc7
I20251027 21:08:14.947134 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a8719fbbe9b6463bb1dd10f1b8f92cb8" } modify_external_catalogs: true
I20251027 21:08:14.947285 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_a8719fbbe9b6463bb1dd10f1b8f92cb8" } modify_external_catalogs: true
I20251027 21:08:14.947798 10029 tablet_bootstrap.cc:654] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.948191 10041 raft_consensus.cc:2243] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:14.948346 10041 raft_consensus.cc:2272] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:14.948871 10041 ts_tablet_manager.cc:1916] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:14.949424 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet 107c608f9d904d86b74e2ad682e1bdc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.949719 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_33a9f7934bc7496eaf4da402364c5179" } new_table_name: "default.Table_dfd6bbb8e92f4c1391279836a0a2e4bd" modify_external_catalogs: true
I20251027 21:08:14.952215 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 107c608f9d904d86b74e2ad682e1bdc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:14.941648 10040 tablet_replica.cc:333] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:14.948956 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 107c608f9d904d86b74e2ad682e1bdc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.951097 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet 28a2e85021394f9d8a9ea524959b52a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.954111 10040 raft_consensus.cc:2243] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:14.955139 10040 raft_consensus.cc:2272] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:14.955304 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 0c88e7aecf3f4e28987158f725e093da (DEFAULT_TABLE table=default.Table_deb3d240fb84407e810d7077b5bda502 [id=031d596510444fbabfb5cf06ea40e339]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.955682 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c88e7aecf3f4e28987158f725e093da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.956432 10040 ts_tablet_manager.cc:1916] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:14.956542 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 0c88e7aecf3f4e28987158f725e093da (DEFAULT_TABLE table=default.Table_deb3d240fb84407e810d7077b5bda502 [id=031d596510444fbabfb5cf06ea40e339]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.956792 11543 raft_consensus.cc:493] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.956910 11543 raft_consensus.cc:515] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.957504 11543 leader_election.cc:290] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.957965 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe94124b239e442dbf644e4a483615fc" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:14.958065 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c88e7aecf3f4e28987158f725e093da. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:14.958443 9859 leader_election.cc:343] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.959012 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe94124b239e442dbf644e4a483615fc" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:14.959543 9860 leader_election.cc:343] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:14.959654 9860 leader_election.cc:304] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.960119 11543 raft_consensus.cc:2749] T fe94124b239e442dbf644e4a483615fc P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.960160 9305 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 2044c73d56bf4434879dd65f35c5612b
I20251027 21:08:14.961726 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 2044c73d56bf4434879dd65f35c5612b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:14.961829 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet 2044c73d56bf4434879dd65f35c5612b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:14.963586 11657 raft_consensus.cc:493] T 465be1b24d16457e834023a2b2194049 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.963735 11657 raft_consensus.cc:515] T 465be1b24d16457e834023a2b2194049 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.964248 11657 leader_election.cc:290] T 465be1b24d16457e834023a2b2194049 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:14.964897 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "465be1b24d16457e834023a2b2194049" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:14.965328 9566 leader_election.cc:343] T 465be1b24d16457e834023a2b2194049 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:14.967705 9568 leader_election.cc:343] T 465be1b24d16457e834023a2b2194049 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 465be1b24d16457e834023a2b2194049
I20251027 21:08:14.967861 9568 leader_election.cc:304] T 465be1b24d16457e834023a2b2194049 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:14.968246 11411 raft_consensus.cc:2749] T 465be1b24d16457e834023a2b2194049 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.968323 10032 tablet_bootstrap.cc:492] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:14.968919 11676 raft_consensus.cc:493] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:14.969039 11676 raft_consensus.cc:515] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.969596 11676 leader_election.cc:290] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:14.970439 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb95618d9fd54a4f8121b43e717d5616" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:14.970490 9711 leader_election.cc:343] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: cb95618d9fd54a4f8121b43e717d5616
I20251027 21:08:14.970675 9634 raft_consensus.cc:2393] T cb95618d9fd54a4f8121b43e717d5616 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77b0513f17c0472088901a1ce8b08218 in current term 1: Already voted for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in this term.
I20251027 21:08:14.971195 9710 leader_election.cc:304] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:14.971546 10032 tablet_bootstrap.cc:654] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.971771 11676 raft_consensus.cc:3060] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:14.974803 10029 tablet_bootstrap.cc:492] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:14.974920 10029 ts_tablet_manager.cc:1403] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20251027 21:08:14.974926 11676 raft_consensus.cc:2749] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:14.975505 10029 raft_consensus.cc:359] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.975659 10029 raft_consensus.cc:385] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.975730 10029 raft_consensus.cc:740] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.977622 10043 ts_tablet_manager.cc:1929] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:14.977787 10043 log.cc:1199] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/f6789dc2f7cb4a3c826ac507d5c9a239
I20251027 21:08:14.978278 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 2044c73d56bf4434879dd65f35c5612b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:14.979327 10043 ts_tablet_manager.cc:1950] T f6789dc2f7cb4a3c826ac507d5c9a239 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:14.979801 10029 consensus_queue.cc:260] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.980306 10029 ts_tablet_manager.cc:1434] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251027 21:08:14.980548 10029 tablet_bootstrap.cc:492] T 0d2d977e360b4ed29c9ed63a309783d6 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:14.980681 10031 tablet_bootstrap.cc:492] T 72c1dc3b9eff47ee81df821deb789307 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:14.981734 10029 tablet_bootstrap.cc:654] T 0d2d977e360b4ed29c9ed63a309783d6 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:14.981981 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1121) took 5825 ms. Trace:
I20251027 21:08:14.983634 10031 ts_tablet_manager.cc:1403] T 72c1dc3b9eff47ee81df821deb789307 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.000s
I20251027 21:08:14.983790 10043 rpcz_store.cc:276] 1027 21:08:09.156298 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.158766 (+ 2468us) service_pool.cc:225] Handling call
1027 21:08:14.888463 (+5729697us) ts_tablet_manager.cc:1151] Deleting tablet f6789dc2f7cb4a3c826ac507d5c9a239
1027 21:08:14.888473 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:14.935733 (+ 47260us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:14.975786 (+ 40053us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:14.981965 (+ 6179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5729539}
I20251027 21:08:14.984232 10031 raft_consensus.cc:359] T 72c1dc3b9eff47ee81df821deb789307 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.984288 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet f6789dc2f7cb4a3c826ac507d5c9a239 (table default.Table_ab9ae94ee52942d588d846e58da2e533 [id=19f6e48bb56848afb9b5982c825b093e]) successfully deleted
I20251027 21:08:14.984393 10031 raft_consensus.cc:385] T 72c1dc3b9eff47ee81df821deb789307 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:14.984445 10031 raft_consensus.cc:740] T 72c1dc3b9eff47ee81df821deb789307 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:14.984656 10031 consensus_queue.cc:260] T 72c1dc3b9eff47ee81df821deb789307 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:14.984766 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1123) took 5828 ms. Trace:
I20251027 21:08:14.984932 10043 rpcz_store.cc:276] 1027 21:08:09.156481 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.159218 (+ 2737us) service_pool.cc:225] Handling call
1027 21:08:14.984655 (+5825437us) ts_tablet_manager.cc:1151] Deleting tablet 229ea4caa6af493cb5b3d593f22aa113
1027 21:08:14.984664 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:14.984748 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5825280}
W20251027 21:08:14.985426 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 229ea4caa6af493cb5b3d593f22aa113 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 229ea4caa6af493cb5b3d593f22aa113 already in progress: creating tablet
I20251027 21:08:14.988790 10041 ts_tablet_manager.cc:1929] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:14.988976 10043 tablet_replica.cc:333] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:14.989542 10043 raft_consensus.cc:2243] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:14.990340 10031 ts_tablet_manager.cc:1434] T 72c1dc3b9eff47ee81df821deb789307 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251027 21:08:14.993062 10043 raft_consensus.cc:2272] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:14.993638 10043 ts_tablet_manager.cc:1916] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:14.995843 10031 tablet_bootstrap.cc:492] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:14.996114 10040 ts_tablet_manager.cc:1929] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:14.996263 10040 log.cc:1199] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/a2d26784bbb24ae3893103802c41affb
I20251027 21:08:14.996681 10040 ts_tablet_manager.cc:1950] T a2d26784bbb24ae3893103802c41affb P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:14.996989 9607 tablet_service.cc:1505] Processing CreateTablet for tablet b227c57ac22649c58081e28982a0b01d (DEFAULT_TABLE table=default.Table_1833f7f718ae412da5598787977143c8 [id=063f77c778094b2f95b8d7819b69b181]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:14.997253 10040 tablet_replica.cc:333] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:14.997370 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b227c57ac22649c58081e28982a0b01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:14.997453 10040 raft_consensus.cc:2243] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:14.997597 10040 raft_consensus.cc:2272] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:14.999388 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet a2d26784bbb24ae3893103802c41affb (table default.Table_9bf7b754a163446db873b9b81a700e4d [id=54ab773e30384ae98a0b0539284c85b4]) successfully deleted
I20251027 21:08:15.000449 9736 tablet_service.cc:1505] Processing CreateTablet for tablet e99c50451f2e4c8eb1452c2a87a819fe (DEFAULT_TABLE table=default.Table_32b6d93944194801af6407774d5afbec [id=6ac3b5bf062d45f184d63c20b8ab025c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.000810 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e99c50451f2e4c8eb1452c2a87a819fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.001416 9602 tablet_service.cc:1505] Processing CreateTablet for tablet e99c50451f2e4c8eb1452c2a87a819fe (DEFAULT_TABLE table=default.Table_32b6d93944194801af6407774d5afbec [id=6ac3b5bf062d45f184d63c20b8ab025c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.001741 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e99c50451f2e4c8eb1452c2a87a819fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.002465 10029 tablet_bootstrap.cc:492] T 0d2d977e360b4ed29c9ed63a309783d6 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:15.002573 10029 ts_tablet_manager.cc:1403] T 0d2d977e360b4ed29c9ed63a309783d6 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20251027 21:08:15.003961 9603 tablet_service.cc:1505] Processing CreateTablet for tablet b82f077d623a446492c765436d3d70be (DEFAULT_TABLE table=default.Table_4a368b4b21ba46b083d78e0c1f3cceb5 [id=0a7a484b4439489ca8a20af97f631fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.004093 10029 raft_consensus.cc:359] T 0d2d977e360b4ed29c9ed63a309783d6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.004230 10029 raft_consensus.cc:385] T 0d2d977e360b4ed29c9ed63a309783d6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.004252 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b82f077d623a446492c765436d3d70be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.004280 10029 raft_consensus.cc:740] T 0d2d977e360b4ed29c9ed63a309783d6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.004230 9738 tablet_service.cc:1505] Processing CreateTablet for tablet b227c57ac22649c58081e28982a0b01d (DEFAULT_TABLE table=default.Table_1833f7f718ae412da5598787977143c8 [id=063f77c778094b2f95b8d7819b69b181]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.004536 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b227c57ac22649c58081e28982a0b01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.004442 10029 consensus_queue.cc:260] T 0d2d977e360b4ed29c9ed63a309783d6 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.004874 10029 ts_tablet_manager.cc:1434] T 0d2d977e360b4ed29c9ed63a309783d6 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:15.005133 10029 tablet_bootstrap.cc:492] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:15.009398 9752 tablet_service.cc:1505] Processing CreateTablet for tablet b82f077d623a446492c765436d3d70be (DEFAULT_TABLE table=default.Table_4a368b4b21ba46b083d78e0c1f3cceb5 [id=0a7a484b4439489ca8a20af97f631fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.009706 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b82f077d623a446492c765436d3d70be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.010192 10040 ts_tablet_manager.cc:1916] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.002823 9893 tablet_service.cc:1505] Processing CreateTablet for tablet b227c57ac22649c58081e28982a0b01d (DEFAULT_TABLE table=default.Table_1833f7f718ae412da5598787977143c8 [id=063f77c778094b2f95b8d7819b69b181]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.011134 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b227c57ac22649c58081e28982a0b01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.011626 10031 tablet_bootstrap.cc:654] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.003118 9891 tablet_service.cc:1505] Processing CreateTablet for tablet b82f077d623a446492c765436d3d70be (DEFAULT_TABLE table=default.Table_4a368b4b21ba46b083d78e0c1f3cceb5 [id=0a7a484b4439489ca8a20af97f631fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.014412 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b82f077d623a446492c765436d3d70be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.015187 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5d6d514c34fb46a6826cf5e730cddd8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:15.015735 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d6d514c34fb46a6826cf5e730cddd8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:15.003965 9887 tablet_service.cc:1505] Processing CreateTablet for tablet e99c50451f2e4c8eb1452c2a87a819fe (DEFAULT_TABLE table=default.Table_32b6d93944194801af6407774d5afbec [id=6ac3b5bf062d45f184d63c20b8ab025c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.018693 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e99c50451f2e4c8eb1452c2a87a819fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.018913 10031 tablet_bootstrap.cc:492] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:15.018913 10032 tablet_bootstrap.cc:492] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:15.019023 10032 ts_tablet_manager.cc:1403] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.003s
I20251027 21:08:15.019023 10031 ts_tablet_manager.cc:1403] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20251027 21:08:15.019582 10031 raft_consensus.cc:359] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.019718 10031 raft_consensus.cc:385] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.019762 10031 raft_consensus.cc:740] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.019958 10031 consensus_queue.cc:260] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.022387 10032 raft_consensus.cc:359] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:15.022707 10032 raft_consensus.cc:385] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.022838 10032 raft_consensus.cc:740] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.023065 10032 consensus_queue.cc:260] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:15.024113 10032 ts_tablet_manager.cc:1434] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:15.024382 10032 tablet_bootstrap.cc:492] T 4bfa70ded2f54e3fb2ca744c723395e2 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:15.025929 10032 tablet_bootstrap.cc:654] T 4bfa70ded2f54e3fb2ca744c723395e2 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.026374 10029 tablet_bootstrap.cc:654] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.024356 11543 raft_consensus.cc:493] T 720cfe66349f4a8896e433d79a2f25aa P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.027124 11543 raft_consensus.cc:515] T 720cfe66349f4a8896e433d79a2f25aa P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.027693 11543 leader_election.cc:290] T 720cfe66349f4a8896e433d79a2f25aa P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.031431 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "720cfe66349f4a8896e433d79a2f25aa" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:15.032241 9860 leader_election.cc:343] T 720cfe66349f4a8896e433d79a2f25aa P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.032697 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "720cfe66349f4a8896e433d79a2f25aa" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:15.033219 9859 leader_election.cc:343] T 720cfe66349f4a8896e433d79a2f25aa P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.033318 9859 leader_election.cc:304] T 720cfe66349f4a8896e433d79a2f25aa P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.033450 10041 log.cc:1199] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/7b8156d9ab2e4dfba383ddc72137aab4
I20251027 21:08:15.033689 11543 raft_consensus.cc:2749] T 720cfe66349f4a8896e433d79a2f25aa P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.033802 10041 ts_tablet_manager.cc:1950] T 7b8156d9ab2e4dfba383ddc72137aab4 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:15.035823 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 218d244dc8a549408863110fefbc7b20
I20251027 21:08:15.036224 9307 catalog_manager.cc:6719] ReplaceTablet: tablet 218d244dc8a549408863110fefbc7b20 deleted and replaced by tablet ab12fe6c9ad44f80ab610a0f03b611ec
I20251027 21:08:15.036309 9307 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:41710 (request call id 1583) took 1011 ms. Trace:
I20251027 21:08:15.036374 9307 rpcz_store.cc:276] 1027 21:08:14.024704 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:14.439086 (+414382us) service_pool.cc:225] Handling call
1027 21:08:15.036294 (+597208us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1027 21:08:14.967873 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1027 21:08:14.967990 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
1027 21:08:14.967995 (+ 5us) write_op.cc:435] Acquired schema lock
1027 21:08:14.967997 (+ 2us) tablet.cc:660] Decoding operations
1027 21:08:14.968109 (+ 112us) write_op.cc:620] Acquiring the partition lock for write op
1027 21:08:14.968122 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1027 21:08:14.968126 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1027 21:08:14.968201 (+ 75us) tablet.cc:699] Row locks acquired
1027 21:08:14.968204 (+ 3us) write_op.cc:260] PREPARE: finished
1027 21:08:14.968238 (+ 34us) write_op.cc:270] Start()
1027 21:08:14.968271 (+ 33us) write_op.cc:276] Timestamp: P: 1761599294968190 usec, L: 0
1027 21:08:14.968274 (+ 3us) op_driver.cc:348] REPLICATION: starting
1027 21:08:14.968471 (+ 197us) log.cc:844] Serialized 622 byte log entry
1027 21:08:15.034986 (+ 66515us) op_driver.cc:464] REPLICATION: finished
1027 21:08:15.035092 (+ 106us) write_op.cc:301] APPLY: starting
1027 21:08:15.035106 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1027 21:08:15.035149 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
1027 21:08:15.035171 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
1027 21:08:15.035274 (+ 103us) tablet.cc:1381] finished ApplyRowOperation cycle
1027 21:08:15.035317 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1027 21:08:15.035331 (+ 14us) write_op.cc:312] APPLY: finished
1027 21:08:15.035399 (+ 68us) log.cc:844] Serialized 29 byte log entry
1027 21:08:15.035452 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:15.035470 (+ 18us) write_op.cc:454] Released schema lock
1027 21:08:15.035476 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1027 21:08:15.035537 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
1027 21:08:15.035540 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":63,"apply.run_cpu_time_us":437,"apply.run_wall_time_us":436,"num_ops":2,"prepare.queue_time_us":119163,"prepare.run_cpu_time_us":747,"prepare.run_wall_time_us":29780,"raft.queue_time_us":1760,"raft.run_cpu_time_us":350,"raft.run_wall_time_us":20181,"replication_time_us":66679}]]}
I20251027 21:08:15.037388 10031 ts_tablet_manager.cc:1434] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20251027 21:08:15.038895 9745 tablet_service.cc:1552] Processing DeleteTablet for tablet 218d244dc8a549408863110fefbc7b20 with delete_type TABLET_DATA_DELETED (replaced by tablet ab12fe6c9ad44f80ab610a0f03b611ec) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:15.039613 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet 218d244dc8a549408863110fefbc7b20 with delete_type TABLET_DATA_DELETED (replaced by tablet ab12fe6c9ad44f80ab610a0f03b611ec) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:15.041244 10040 ts_tablet_manager.cc:1929] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:15.041496 10041 tablet_replica.cc:333] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:15.041729 10041 raft_consensus.cc:2243] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.041934 10041 raft_consensus.cc:2272] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.042497 10041 ts_tablet_manager.cc:1916] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.045390 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 218d244dc8a549408863110fefbc7b20 with delete_type TABLET_DATA_DELETED (replaced by tablet ab12fe6c9ad44f80ab610a0f03b611ec) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:15.045899 10043 ts_tablet_manager.cc:1929] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:15.046010 10043 log.cc:1199] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/1d64a13667194c12864fa388eb88a380
I20251027 21:08:15.046351 10043 ts_tablet_manager.cc:1950] T 1d64a13667194c12864fa388eb88a380 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:15.046877 10040 log.cc:1199] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/07a9c3b7ef1548ec9d5ae9f20545f263
I20251027 21:08:15.047245 10040 ts_tablet_manager.cc:1950] T 07a9c3b7ef1548ec9d5ae9f20545f263 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:15.047752 10040 tablet_replica.cc:333] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:15.047971 10040 raft_consensus.cc:2243] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.048127 10040 raft_consensus.cc:2272] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.048375 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 07a9c3b7ef1548ec9d5ae9f20545f263 (table default.Table_a7632420526d409290924efb41972ee1 [id=4a103f97fc964eaba06cd48d4ba854a6]) successfully deleted
I20251027 21:08:15.041299 9775 raft_consensus.cc:3060] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.048671 10040 ts_tablet_manager.cc:1916] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.041433 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 7b8156d9ab2e4dfba383ddc72137aab4 (table default.Table_d5b5337ae438439d984f34a4e0af6c93 [id=8932a6e05aae478ab0a22e213896851f]) successfully deleted
I20251027 21:08:15.055670 9745 tablet_service.cc:1505] Processing CreateTablet for tablet d4ace93620814b428867c8fd9ab461be (DEFAULT_TABLE table=default.Table_bd8533529e7a49a681bbc785aa61c597 [id=20f379773d2742408cae21549c1da0fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.056020 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4ace93620814b428867c8fd9ab461be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.055671 9742 tablet_service.cc:1505] Processing CreateTablet for tablet ea1ee9d5fd9543849ddcdcafea47b195 (DEFAULT_TABLE table=default.Table_305463d05bd84f69883cb6165ec89446 [id=fbff2651436946ceaf09dfbe8e571556]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.058570 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea1ee9d5fd9543849ddcdcafea47b195. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.066757 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 5e20fb964a4543ff9eda684acc8eb2bb
I20251027 21:08:15.067344 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet afff93eb06ab4bcca2990c8bc534b234
I20251027 21:08:15.068625 9891 tablet_service.cc:1505] Processing CreateTablet for tablet ea1ee9d5fd9543849ddcdcafea47b195 (DEFAULT_TABLE table=default.Table_305463d05bd84f69883cb6165ec89446 [id=fbff2651436946ceaf09dfbe8e571556]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.068982 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea1ee9d5fd9543849ddcdcafea47b195. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.074402 9887 tablet_service.cc:1505] Processing CreateTablet for tablet d4ace93620814b428867c8fd9ab461be (DEFAULT_TABLE table=default.Table_bd8533529e7a49a681bbc785aa61c597 [id=20f379773d2742408cae21549c1da0fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.074780 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4ace93620814b428867c8fd9ab461be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.068006 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e20fb964a4543ff9eda684acc8eb2bb with delete_type TABLET_DATA_DELETED (replaced by tablet 8a3985daaf24490d972e7e65f85d87d4) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:15.048446 10029 tablet_bootstrap.cc:492] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:15.069698 9602 tablet_service.cc:1505] Processing CreateTablet for tablet d4ace93620814b428867c8fd9ab461be (DEFAULT_TABLE table=default.Table_bd8533529e7a49a681bbc785aa61c597 [id=20f379773d2742408cae21549c1da0fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.081964 10029 ts_tablet_manager.cc:1403] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.077s user 0.005s sys 0.000s
I20251027 21:08:15.082118 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4ace93620814b428867c8fd9ab461be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.081353 11658 raft_consensus.cc:493] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.082299 11658 raft_consensus.cc:515] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.082549 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1122) took 5926 ms. Trace:
I20251027 21:08:15.082621 10029 raft_consensus.cc:359] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.082787 10029 raft_consensus.cc:385] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.082826 10029 raft_consensus.cc:740] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.082916 11658 leader_election.cc:290] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.083029 10029 consensus_queue.cc:260] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.083436 10029 ts_tablet_manager.cc:1434] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 21:08:15.083685 10029 tablet_bootstrap.cc:492] T 8dc358445526415d8cd4f1ac6779126d P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:15.083817 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9be6b996df8145b9b0df1a2d2dc0ef03" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:15.083976 9909 raft_consensus.cc:2468] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:15.084398 9568 leader_election.cc:304] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:15.082613 10043 rpcz_store.cc:276] 1027 21:08:09.156393 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.159406 (+ 3013us) service_pool.cc:225] Handling call
1027 21:08:14.987779 (+5828373us) ts_tablet_manager.cc:1151] Deleting tablet 1d64a13667194c12864fa388eb88a380
1027 21:08:14.988932 (+ 1153us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:15.043116 (+ 54184us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:15.045814 (+ 2698us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:15.082534 (+ 36720us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5828221}
I20251027 21:08:15.084678 11373 raft_consensus.cc:2804] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:15.084767 11373 raft_consensus.cc:493] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:15.084825 11373 raft_consensus.cc:3060] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.084905 10029 tablet_bootstrap.cc:654] T 8dc358445526415d8cd4f1ac6779126d P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.070003 9603 tablet_service.cc:1505] Processing CreateTablet for tablet ea1ee9d5fd9543849ddcdcafea47b195 (DEFAULT_TABLE table=default.Table_305463d05bd84f69883cb6165ec89446 [id=fbff2651436946ceaf09dfbe8e571556]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.085997 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea1ee9d5fd9543849ddcdcafea47b195. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.086012 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 1d64a13667194c12864fa388eb88a380 (table default.Table_d11118d727fb4579a6465bd6deebaa43 [id=9df9c59afc184f9bb7494b154852d943]) successfully deleted
I20251027 21:08:15.087414 11373 raft_consensus.cc:515] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.087913 11373 leader_election.cc:290] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.087898 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9be6b996df8145b9b0df1a2d2dc0ef03" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:15.088660 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9be6b996df8145b9b0df1a2d2dc0ef03" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:15.088804 9909 raft_consensus.cc:3060] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.090977 10043 tablet_replica.cc:333] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:15.091157 10043 raft_consensus.cc:2243] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.091289 10043 raft_consensus.cc:2272] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.091284 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9be6b996df8145b9b0df1a2d2dc0ef03" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:15.091606 9566 leader_election.cc:343] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.091763 10043 ts_tablet_manager.cc:1916] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.093160 9909 raft_consensus.cc:2468] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:15.093875 9568 leader_election.cc:304] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:15.094170 11658 raft_consensus.cc:2804] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:15.094307 11658 raft_consensus.cc:697] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:15.094591 11658 consensus_queue.cc:237] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:15.098028 9566 leader_election.cc:343] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.098335 10041 ts_tablet_manager.cc:1929] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:15.098464 10041 log.cc:1199] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/11627a84ff9b478c87c323617a09570b
I20251027 21:08:15.098744 10041 ts_tablet_manager.cc:1950] T 11627a84ff9b478c87c323617a09570b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:15.098769 10029 tablet_bootstrap.cc:492] T 8dc358445526415d8cd4f1ac6779126d P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:15.098867 10029 ts_tablet_manager.cc:1403] T 8dc358445526415d8cd4f1ac6779126d P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20251027 21:08:15.099395 10029 raft_consensus.cc:359] T 8dc358445526415d8cd4f1ac6779126d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.099519 10029 raft_consensus.cc:385] T 8dc358445526415d8cd4f1ac6779126d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.099531 10041 tablet_replica.cc:333] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:15.099565 10029 raft_consensus.cc:740] T 8dc358445526415d8cd4f1ac6779126d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.099709 10041 raft_consensus.cc:2243] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.099721 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 11627a84ff9b478c87c323617a09570b (table default.Table_7b970942803b42f29feb36d6d845167a [id=20a729dd21c440a882083323ddb2561e]) successfully deleted
I20251027 21:08:15.099731 10029 consensus_queue.cc:260] T 8dc358445526415d8cd4f1ac6779126d P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.099856 10041 raft_consensus.cc:2272] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.100126 10029 ts_tablet_manager.cc:1434] T 8dc358445526415d8cd4f1ac6779126d P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 21:08:15.100344 10029 tablet_bootstrap.cc:492] T e10ae34eaaf3476594813de8e96c7e77 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:15.101531 10029 tablet_bootstrap.cc:654] T e10ae34eaaf3476594813de8e96c7e77 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.102347 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet cf1751af4c774d078a4aede9d2bb8108
I20251027 21:08:15.042474 10031 tablet_bootstrap.cc:492] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:15.108929 10029 tablet_bootstrap.cc:492] T e10ae34eaaf3476594813de8e96c7e77 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:15.109046 10029 ts_tablet_manager.cc:1403] T e10ae34eaaf3476594813de8e96c7e77 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20251027 21:08:15.109615 10029 raft_consensus.cc:359] T e10ae34eaaf3476594813de8e96c7e77 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.109767 10029 raft_consensus.cc:385] T e10ae34eaaf3476594813de8e96c7e77 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.109812 10029 raft_consensus.cc:740] T e10ae34eaaf3476594813de8e96c7e77 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.110743 10031 tablet_bootstrap.cc:654] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.111519 10029 consensus_queue.cc:260] T e10ae34eaaf3476594813de8e96c7e77 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.114393 9771 raft_consensus.cc:3060] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.114429 10040 ts_tablet_manager.cc:1929] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251027 21:08:15.114527 10040 log.cc:1199] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/aaa8910d9fdb4146945621ec4ebf969e
I20251027 21:08:15.114837 10040 ts_tablet_manager.cc:1950] T aaa8910d9fdb4146945621ec4ebf969e P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:15.115206 10041 ts_tablet_manager.cc:1916] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.115962 10040 tablet_replica.cc:333] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:15.116142 10040 raft_consensus.cc:2243] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.116279 10040 raft_consensus.cc:2272] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.111994 10029 ts_tablet_manager.cc:1434] T e10ae34eaaf3476594813de8e96c7e77 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251027 21:08:15.116695 10029 tablet_bootstrap.cc:492] T 371aa8a57aee466ab01652d6061db4c2 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:15.118207 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet aaa8910d9fdb4146945621ec4ebf969e (table default.Table_cc8a1775cbcb4e0eaeb8c90bf9d61c04 [id=bb676562b97f48088767941239cfbf33]) successfully deleted
W20251027 21:08:15.118458 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet d91fced778784b65be2486e24426d1b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d91fced778784b65be2486e24426d1b5 already in progress: creating tablet
W20251027 21:08:15.118613 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet e8fe63ea8aba4614900ac254189464de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8fe63ea8aba4614900ac254189464de already in progress: creating tablet
I20251027 21:08:15.118845 10029 tablet_bootstrap.cc:654] T 371aa8a57aee466ab01652d6061db4c2 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.124370 10040 ts_tablet_manager.cc:1916] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.127031 9305 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 8cccd253ae2a46328dc9f767b8e9ee97 (table default.Table_4ba26495af344e3cb082a4b781f1b208 [id=501b230a824d4a17bb1903c8409407b7]). Expected version 2 got 0
I20251027 21:08:15.127207 9305 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet dd9e6656d8df43afbb7f70706bc14585 (table default.Table_f78d839900634631b6963b2d63f0d084 [id=3cdf432ec43d42e8bc2d253a92eac8d3]). Expected version 3 got 0
I20251027 21:08:15.128401 9305 master_service.cc:954] ReplaceTablet: received request to replace tablet db2735604dc14039810df4ecd330c671 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:15.128511 10043 ts_tablet_manager.cc:1929] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:15.128583 10043 log.cc:1199] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/9c2d713d72fd4b5ca34cb3e828c75761
I20251027 21:08:15.128839 10043 ts_tablet_manager.cc:1950] T 9c2d713d72fd4b5ca34cb3e828c75761 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:15.129550 10029 tablet_bootstrap.cc:492] T 371aa8a57aee466ab01652d6061db4c2 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:15.129675 10029 ts_tablet_manager.cc:1403] T 371aa8a57aee466ab01652d6061db4c2 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20251027 21:08:15.129710 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet afff93eb06ab4bcca2990c8bc534b234 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:15.118760 9297 catalog_manager.cc:4952] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet f6789dc2f7cb4a3c826ac507d5c9a239 because the tablet was not found. No further retry: Not found: Tablet not found: f6789dc2f7cb4a3c826ac507d5c9a239
I20251027 21:08:15.130000 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet cf1751af4c774d078a4aede9d2bb8108 with delete_type TABLET_DATA_DELETED (replaced by tablet 6443f2950db143bda9f098453ec4de25) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:15.130715 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet afff93eb06ab4bcca2990c8bc534b234 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:15.133023 9312 catalog_manager.cc:6719] ReplaceTablet: tablet 5e20fb964a4543ff9eda684acc8eb2bb deleted and replaced by tablet 8a3985daaf24490d972e7e65f85d87d4
I20251027 21:08:15.133486 9310 catalog_manager.cc:6719] ReplaceTablet: tablet cf1751af4c774d078a4aede9d2bb8108 deleted and replaced by tablet 6443f2950db143bda9f098453ec4de25
I20251027 21:08:15.134934 10029 raft_consensus.cc:359] T 371aa8a57aee466ab01652d6061db4c2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.135090 10029 raft_consensus.cc:385] T 371aa8a57aee466ab01652d6061db4c2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.135138 10029 raft_consensus.cc:740] T 371aa8a57aee466ab01652d6061db4c2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.136080 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet afff93eb06ab4bcca2990c8bc534b234 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:15.136284 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet cf1751af4c774d078a4aede9d2bb8108 with delete_type TABLET_DATA_DELETED (replaced by tablet 6443f2950db143bda9f098453ec4de25) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:15.136660 9891 tablet_service.cc:1505] Processing CreateTablet for tablet acf595946f7448ac8e085391521ed347 (DEFAULT_TABLE table=default.Table_8496081464c049219f4c7e4de97d11c7 [id=440025003276460f82abebbef2ca6785]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.137042 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acf595946f7448ac8e085391521ed347. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.137878 10029 consensus_queue.cc:260] T 371aa8a57aee466ab01652d6061db4c2 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.138221 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:15.138350 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet 42371496a7494e77a7bf1169de5b083a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:15.138407 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d64a13667194c12864fa388eb88a380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:15.138542 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet f6789dc2f7cb4a3c826ac507d5c9a239 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:52 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:15.138676 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e20fb964a4543ff9eda684acc8eb2bb with delete_type TABLET_DATA_DELETED (replaced by tablet 8a3985daaf24490d972e7e65f85d87d4) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:15.138743 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet a2d26784bbb24ae3893103802c41affb with delete_type TABLET_DATA_DELETED (replaced by tablet 9ace4188d3494a05a5678381f8d8fd17) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:15.138832 9607 tablet_service.cc:1505] Processing CreateTablet for tablet acf595946f7448ac8e085391521ed347 (DEFAULT_TABLE table=default.Table_8496081464c049219f4c7e4de97d11c7 [id=440025003276460f82abebbef2ca6785]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.138983 9602 tablet_service.cc:1505] Processing CreateTablet for tablet e37187ae2c52422eb8b3b02f1ae3c44d (DEFAULT_TABLE table=default.Table_45b899e2d3ab48ec90355341e1648881 [id=b044517162934cdcb6cd994cd2b0ea16]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.139148 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acf595946f7448ac8e085391521ed347. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.139254 10029 ts_tablet_manager.cc:1434] T 371aa8a57aee466ab01652d6061db4c2 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251027 21:08:15.139312 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e37187ae2c52422eb8b3b02f1ae3c44d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.139482 10029 tablet_bootstrap.cc:492] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:15.140929 10029 tablet_bootstrap.cc:654] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.142688 9893 tablet_service.cc:1552] Processing DeleteTablet for tablet 874d5974698c448ab79f19812cb0e886 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:15.143263 9904 tablet_service.cc:1505] Processing CreateTablet for tablet fb0b71c0633145888d4c2a325f7042f3 (DEFAULT_TABLE table=default.Table_c842d5beeb2a4a43b96a61a2cf97a8a0 [id=191361e27de84d9986de62a882afeefb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.143483 9613 tablet_service.cc:1505] Processing CreateTablet for tablet fb0b71c0633145888d4c2a325f7042f3 (DEFAULT_TABLE table=default.Table_c842d5beeb2a4a43b96a61a2cf97a8a0 [id=191361e27de84d9986de62a882afeefb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.143599 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb0b71c0633145888d4c2a325f7042f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.143805 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb0b71c0633145888d4c2a325f7042f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.146804 9893 tablet_service.cc:1505] Processing CreateTablet for tablet e37187ae2c52422eb8b3b02f1ae3c44d (DEFAULT_TABLE table=default.Table_45b899e2d3ab48ec90355341e1648881 [id=b044517162934cdcb6cd994cd2b0ea16]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.147126 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e37187ae2c52422eb8b3b02f1ae3c44d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.150260 11411 raft_consensus.cc:493] T 0d2d977e360b4ed29c9ed63a309783d6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.150370 11411 raft_consensus.cc:515] T 0d2d977e360b4ed29c9ed63a309783d6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.150877 11411 leader_election.cc:290] T 0d2d977e360b4ed29c9ed63a309783d6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.150821 9752 tablet_service.cc:1505] Processing CreateTablet for tablet e37187ae2c52422eb8b3b02f1ae3c44d (DEFAULT_TABLE table=default.Table_45b899e2d3ab48ec90355341e1648881 [id=b044517162934cdcb6cd994cd2b0ea16]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.151206 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e37187ae2c52422eb8b3b02f1ae3c44d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:15.151660 9568 leader_election.cc:343] T 0d2d977e360b4ed29c9ed63a309783d6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 0d2d977e360b4ed29c9ed63a309783d6
I20251027 21:08:15.153090 10040 ts_tablet_manager.cc:1929] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:15.153173 10040 log.cc:1199] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/7292bc6c22ca4ab5b8f53a69520833ba
I20251027 21:08:15.153496 10040 ts_tablet_manager.cc:1950] T 7292bc6c22ca4ab5b8f53a69520833ba P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:15.153455 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d2d977e360b4ed29c9ed63a309783d6" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:15.153846 9566 leader_election.cc:343] T 0d2d977e360b4ed29c9ed63a309783d6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.153939 9566 leader_election.cc:304] T 0d2d977e360b4ed29c9ed63a309783d6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.154277 11411 raft_consensus.cc:2749] T 0d2d977e360b4ed29c9ed63a309783d6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.154661 9736 tablet_service.cc:1505] Processing CreateTablet for tablet fb0b71c0633145888d4c2a325f7042f3 (DEFAULT_TABLE table=default.Table_c842d5beeb2a4a43b96a61a2cf97a8a0 [id=191361e27de84d9986de62a882afeefb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.154992 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb0b71c0633145888d4c2a325f7042f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.156224 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 520f7142f8af4599ac4880b0692e3151 with delete_type TABLET_DATA_DELETED (replaced by tablet 781a162e17dd495b888f7e389c062013) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:15.156421 9745 tablet_service.cc:1552] Processing DeleteTablet for tablet cf1751af4c774d078a4aede9d2bb8108 with delete_type TABLET_DATA_DELETED (replaced by tablet 6443f2950db143bda9f098453ec4de25) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:15.158401 9613 tablet_service.cc:1505] Processing CreateTablet for tablet 023eb8eb693d446caef49e0eaecd147c (DEFAULT_TABLE table=default.Table_4567c3392eb24748840827eb96835642 [id=9192a31ca6324aa2a919311093d54ae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.158779 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 023eb8eb693d446caef49e0eaecd147c (DEFAULT_TABLE table=default.Table_4567c3392eb24748840827eb96835642 [id=9192a31ca6324aa2a919311093d54ae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.159478 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 023eb8eb693d446caef49e0eaecd147c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.159780 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 023eb8eb693d446caef49e0eaecd147c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.160094 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 7292bc6c22ca4ab5b8f53a69520833ba (table default.Table_4658a756ae8149cf9e248c65a102bdb6 [id=0a31c72c314e4e5ab0655d9b853c941c]) successfully deleted
I20251027 21:08:15.161445 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e20fb964a4543ff9eda684acc8eb2bb with delete_type TABLET_DATA_DELETED (replaced by tablet 8a3985daaf24490d972e7e65f85d87d4) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:15.160359 10040 tablet_replica.cc:333] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:15.161227 9738 tablet_service.cc:1505] Processing CreateTablet for tablet acf595946f7448ac8e085391521ed347 (DEFAULT_TABLE table=default.Table_8496081464c049219f4c7e4de97d11c7 [id=440025003276460f82abebbef2ca6785]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.162055 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acf595946f7448ac8e085391521ed347. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.165917 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 023eb8eb693d446caef49e0eaecd147c (DEFAULT_TABLE table=default.Table_4567c3392eb24748840827eb96835642 [id=9192a31ca6324aa2a919311093d54ae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.166258 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 023eb8eb693d446caef49e0eaecd147c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.167404 10029 tablet_bootstrap.cc:492] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:15.167762 9602 tablet_service.cc:1505] Processing CreateTablet for tablet e5ba4b61dccf47d2b5c18d43a7a08e76 (DEFAULT_TABLE table=default.Table_8a73cb7d9a4849a4aae95965ddf1e738 [id=ed3b8d5919b34bb2ba190bd469ccd1ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.168052 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5ba4b61dccf47d2b5c18d43a7a08e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.168501 10040 raft_consensus.cc:2243] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.168646 10040 raft_consensus.cc:2272] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.169195 10040 ts_tablet_manager.cc:1916] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.170681 11411 raft_consensus.cc:493] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.170790 11411 raft_consensus.cc:515] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.171010 9603 tablet_service.cc:1505] Processing CreateTablet for tablet 48ed3748472947e68690adce18053efa (DEFAULT_TABLE table=default.Table_1a2deac4d0444b85b51757688f89b202 [id=71db9ec3df6a4f71aaaefb47e219929c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.171317 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48ed3748472947e68690adce18053efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.171376 11411 leader_election.cc:290] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.171941 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9558d53008640739ac6fb45f1352fc0" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:15.172403 9566 leader_election.cc:343] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.172986 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9558d53008640739ac6fb45f1352fc0" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:15.173200 9909 raft_consensus.cc:2468] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:15.173559 9568 leader_election.cc:304] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.174000 11411 raft_consensus.cc:2804] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:15.174065 11411 raft_consensus.cc:493] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:15.174115 11411 raft_consensus.cc:3060] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.176738 11411 raft_consensus.cc:515] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.177559 11411 leader_election.cc:290] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.178828 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9558d53008640739ac6fb45f1352fc0" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:15.179234 9566 leader_election.cc:343] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.179886 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9558d53008640739ac6fb45f1352fc0" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:15.180070 9909 raft_consensus.cc:3060] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.167515 10029 ts_tablet_manager.cc:1403] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.001s
I20251027 21:08:15.181557 10029 raft_consensus.cc:359] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.181702 10029 raft_consensus.cc:385] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.181757 10029 raft_consensus.cc:740] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.181946 10029 consensus_queue.cc:260] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.182534 10029 ts_tablet_manager.cc:1434] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 21:08:15.182781 10029 tablet_bootstrap.cc:492] T 5cb8ca3a50c148e6b602e949650353f8 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:15.183459 9909 raft_consensus.cc:2468] T a9558d53008640739ac6fb45f1352fc0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:15.184080 9568 leader_election.cc:304] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.184422 10029 tablet_bootstrap.cc:654] T 5cb8ca3a50c148e6b602e949650353f8 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.184454 11373 raft_consensus.cc:2804] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:15.184572 11373 raft_consensus.cc:697] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:15.184777 11373 consensus_queue.cc:237] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.186520 9904 tablet_service.cc:1505] Processing CreateTablet for tablet e5ba4b61dccf47d2b5c18d43a7a08e76 (DEFAULT_TABLE table=default.Table_8a73cb7d9a4849a4aae95965ddf1e738 [id=ed3b8d5919b34bb2ba190bd469ccd1ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.186983 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5ba4b61dccf47d2b5c18d43a7a08e76. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:15.190625 9566 consensus_peers.cc:597] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aaa8910d9fdb4146945621ec4ebf969e. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:15.191751 10040 ts_tablet_manager.cc:1929] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:15.192854 10040 log.cc:1199] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/90f96c21da9944f1ade05d35787d47e2
I20251027 21:08:15.193269 10040 ts_tablet_manager.cc:1950] T 90f96c21da9944f1ade05d35787d47e2 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:15.193897 9736 tablet_service.cc:1505] Processing CreateTablet for tablet e5ba4b61dccf47d2b5c18d43a7a08e76 (DEFAULT_TABLE table=default.Table_8a73cb7d9a4849a4aae95965ddf1e738 [id=ed3b8d5919b34bb2ba190bd469ccd1ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.194465 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 90f96c21da9944f1ade05d35787d47e2 (table default.Table_cdf514bc48a4478da802d1bbfd0f1c71 [id=ec9aa8523bf44f2f9486b14f4cf7639c]) successfully deleted
I20251027 21:08:15.194581 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5ba4b61dccf47d2b5c18d43a7a08e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.195612 10040 tablet_replica.cc:333] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:15.195784 10029 tablet_bootstrap.cc:492] T 5cb8ca3a50c148e6b602e949650353f8 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:15.195925 10029 ts_tablet_manager.cc:1403] T 5cb8ca3a50c148e6b602e949650353f8 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.003s
I20251027 21:08:15.196172 10040 raft_consensus.cc:2243] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:15.196604 10040 pending_rounds.cc:70] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
I20251027 21:08:15.196698 10029 raft_consensus.cc:359] T 5cb8ca3a50c148e6b602e949650353f8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:15.196890 10029 raft_consensus.cc:385] T 5cb8ca3a50c148e6b602e949650353f8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.196938 10029 raft_consensus.cc:740] T 5cb8ca3a50c148e6b602e949650353f8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.197050 10040 pending_rounds.cc:77] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510710660071424 op_type: NO_OP noop_request { }
I20251027 21:08:15.197130 10029 consensus_queue.cc:260] T 5cb8ca3a50c148e6b602e949650353f8 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:15.197412 11543 raft_consensus.cc:493] T ed66f9eeeb2b4b4bb77062c241d2140f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.197259 10040 raft_consensus.cc:2889] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:15.197521 10040 raft_consensus.cc:2272] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.197527 11543 raft_consensus.cc:515] T ed66f9eeeb2b4b4bb77062c241d2140f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:15.197695 10029 ts_tablet_manager.cc:1434] T 5cb8ca3a50c148e6b602e949650353f8 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:15.197971 10029 tablet_bootstrap.cc:492] T 43611e33512b49d58485cf15de788be7 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:15.198050 11543 leader_election.cc:290] T ed66f9eeeb2b4b4bb77062c241d2140f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.198662 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed66f9eeeb2b4b4bb77062c241d2140f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:15.199169 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed66f9eeeb2b4b4bb77062c241d2140f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:15.199477 10040 ts_tablet_manager.cc:1916] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:15.199878 9859 leader_election.cc:343] T ed66f9eeeb2b4b4bb77062c241d2140f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:15.200290 9860 leader_election.cc:343] T ed66f9eeeb2b4b4bb77062c241d2140f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.200379 9860 leader_election.cc:304] T ed66f9eeeb2b4b4bb77062c241d2140f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.200774 11543 raft_consensus.cc:2749] T ed66f9eeeb2b4b4bb77062c241d2140f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.202363 10029 tablet_bootstrap.cc:654] T 43611e33512b49d58485cf15de788be7 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.129468 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1120) took 5973 ms. Trace:
I20251027 21:08:15.205637 10043 rpcz_store.cc:276] 1027 21:08:09.155930 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.156039 (+ 109us) service_pool.cc:225] Handling call
1027 21:08:15.090915 (+5934876us) ts_tablet_manager.cc:1151] Deleting tablet 9c2d713d72fd4b5ca34cb3e828c75761
1027 21:08:15.090923 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:15.113973 (+ 23050us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:15.128437 (+ 14464us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:15.129450 (+ 1013us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5931226}
I20251027 21:08:15.207683 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 48ed3748472947e68690adce18053efa (DEFAULT_TABLE table=default.Table_1a2deac4d0444b85b51757688f89b202 [id=71db9ec3df6a4f71aaaefb47e219929c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.208133 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48ed3748472947e68690adce18053efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.208832 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 1b935a190e96421195d1fb8b2da3d00f (DEFAULT_TABLE table=default.Table_2a79f5c1ef84456fa6c96ec6a45a321d [id=2aa0b9adae9848509dafd19a663c017a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.209154 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b935a190e96421195d1fb8b2da3d00f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.212860 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 0a0334b492b94ccc8a42f2d6aa0302e0 (DEFAULT_TABLE table=default.Table_0655ada3126c49049155b9671bcbe409 [id=155318e67b2845fb83e07368aa5864de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.213263 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a0334b492b94ccc8a42f2d6aa0302e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.214255 11676 raft_consensus.cc:493] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.214370 11676 raft_consensus.cc:515] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.214468 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 262b4fb71a2d40fc879da56f8c51ef83 (DEFAULT_TABLE table=default.Table_e08f0cbb71674a44a697686c1b6effc9 [id=471f86ddb9de40a7ae58ec202926a1fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.214804 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 262b4fb71a2d40fc879da56f8c51ef83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.215003 11676 leader_election.cc:290] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:15.216130 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 262b4fb71a2d40fc879da56f8c51ef83 (DEFAULT_TABLE table=default.Table_e08f0cbb71674a44a697686c1b6effc9 [id=471f86ddb9de40a7ae58ec202926a1fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.216406 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 262b4fb71a2d40fc879da56f8c51ef83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.220206 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 48ed3748472947e68690adce18053efa (DEFAULT_TABLE table=default.Table_1a2deac4d0444b85b51757688f89b202 [id=71db9ec3df6a4f71aaaefb47e219929c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.220579 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48ed3748472947e68690adce18053efa. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:15.222059 9711 leader_election.cc:343] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 842c1a1298234d9ea2719bab29c54ae3
I20251027 21:08:15.222939 11543 raft_consensus.cc:493] T 9ace4188d3494a05a5678381f8d8fd17 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.223062 11543 raft_consensus.cc:515] T 9ace4188d3494a05a5678381f8d8fd17 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:15.223590 11543 leader_election.cc:290] T 9ace4188d3494a05a5678381f8d8fd17 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.224234 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ace4188d3494a05a5678381f8d8fd17" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:15.224756 9859 leader_election.cc:343] T 9ace4188d3494a05a5678381f8d8fd17 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.224958 11676 raft_consensus.cc:493] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.225072 11676 raft_consensus.cc:515] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.225502 11676 leader_election.cc:290] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:15.209324 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 1b935a190e96421195d1fb8b2da3d00f (DEFAULT_TABLE table=default.Table_2a79f5c1ef84456fa6c96ec6a45a321d [id=2aa0b9adae9848509dafd19a663c017a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.210536 9603 tablet_service.cc:1505] Processing CreateTablet for tablet 262b4fb71a2d40fc879da56f8c51ef83 (DEFAULT_TABLE table=default.Table_e08f0cbb71674a44a697686c1b6effc9 [id=471f86ddb9de40a7ae58ec202926a1fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.215803 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "842c1a1298234d9ea2719bab29c54ae3" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:15.227088 9634 raft_consensus.cc:2468] T 842c1a1298234d9ea2719bab29c54ae3 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:08:15.227185 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b935a190e96421195d1fb8b2da3d00f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.227357 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "952583fd7fde4b4baccdb98c01bdf9e1" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:15.227483 9635 raft_consensus.cc:2468] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
W20251027 21:08:15.227859 9711 leader_election.cc:343] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 952583fd7fde4b4baccdb98c01bdf9e1
I20251027 21:08:15.228062 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 262b4fb71a2d40fc879da56f8c51ef83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.226555 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ace4188d3494a05a5678381f8d8fd17" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:15.232530 11373 raft_consensus.cc:493] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251027 21:08:15.232699 9860 leader_election.cc:343] T 9ace4188d3494a05a5678381f8d8fd17 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.232807 9860 leader_election.cc:304] T 9ace4188d3494a05a5678381f8d8fd17 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.232707 11373 raft_consensus.cc:515] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.233016 11543 raft_consensus.cc:2749] T 9ace4188d3494a05a5678381f8d8fd17 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.233135 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1128) took 6076 ms. Trace:
I20251027 21:08:15.233214 10043 rpcz_store.cc:276] 1027 21:08:09.156955 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.162714 (+ 5759us) service_pool.cc:225] Handling call
1027 21:08:15.233034 (+6070320us) ts_tablet_manager.cc:1151] Deleting tablet b8c368698cfb40799571567e8bfdd8dc
1027 21:08:15.233043 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:15.233121 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6070190}
I20251027 21:08:15.233484 10043 tablet_replica.cc:333] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:15.233474 10040 ts_tablet_manager.cc:1929] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:15.233574 10040 log.cc:1199] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/9ed4f9c61ad54e1292281b30bcd19533
I20251027 21:08:15.233668 10043 raft_consensus.cc:2243] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:15.233983 10043 raft_consensus.cc:2272] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.233942 11373 leader_election.cc:290] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.234519 10043 ts_tablet_manager.cc:1916] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.234845 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 9c2d713d72fd4b5ca34cb3e828c75761 (table default.Table_294e3c4092274dc1a146d3f604bc2179 [id=bd8f38a55b9f42b99e372c133a148007]) successfully deleted
W20251027 21:08:15.235136 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet b8c368698cfb40799571567e8bfdd8dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8c368698cfb40799571567e8bfdd8dc already in progress: creating tablet
I20251027 21:08:15.235668 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e64d21abef342de86555e876f020787" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:15.235882 10040 ts_tablet_manager.cc:1950] T 9ed4f9c61ad54e1292281b30bcd19533 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:15.236047 9710 leader_election.cc:304] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:15.236299 11676 raft_consensus.cc:2804] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:15.236378 11676 raft_consensus.cc:493] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:15.236434 11676 raft_consensus.cc:3060] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.236516 9710 leader_election.cc:304] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:15.236723 11626 raft_consensus.cc:2804] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:15.236776 11626 raft_consensus.cc:493] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:15.236832 11626 raft_consensus.cc:3060] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.239008 11676 raft_consensus.cc:515] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.239461 11626 raft_consensus.cc:515] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.239972 11626 leader_election.cc:290] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:15.240062 11676 leader_election.cc:290] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:15.240604 9566 leader_election.cc:343] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:15.241021 9711 leader_election.cc:343] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 952583fd7fde4b4baccdb98c01bdf9e1
I20251027 21:08:15.241078 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "842c1a1298234d9ea2719bab29c54ae3" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:15.241205 9630 raft_consensus.cc:3060] T 842c1a1298234d9ea2719bab29c54ae3 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:15.241302 9568 leader_election.cc:343] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 0e64d21abef342de86555e876f020787
I20251027 21:08:15.241465 9568 leader_election.cc:304] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.242255 11512 raft_consensus.cc:2749] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.242277 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 1b935a190e96421195d1fb8b2da3d00f (DEFAULT_TABLE table=default.Table_2a79f5c1ef84456fa6c96ec6a45a321d [id=2aa0b9adae9848509dafd19a663c017a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.242622 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b935a190e96421195d1fb8b2da3d00f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.243520 10040 tablet_replica.cc:333] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:15.243705 10040 raft_consensus.cc:2243] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
W20251027 21:08:15.243911 9711 leader_election.cc:343] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 842c1a1298234d9ea2719bab29c54ae3
I20251027 21:08:15.243997 9630 raft_consensus.cc:2468] T 842c1a1298234d9ea2719bab29c54ae3 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:15.243927 10040 pending_rounds.cc:70] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
I20251027 21:08:15.244506 10040 pending_rounds.cc:77] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510711445061632 op_type: NO_OP noop_request { }
I20251027 21:08:15.244586 10040 raft_consensus.cc:2889] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:15.244652 10040 raft_consensus.cc:2272] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.248898 10029 tablet_bootstrap.cc:492] T 43611e33512b49d58485cf15de788be7 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:15.249027 10029 ts_tablet_manager.cc:1403] T 43611e33512b49d58485cf15de788be7 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.051s user 0.008s sys 0.004s
I20251027 21:08:15.249871 10029 raft_consensus.cc:359] T 43611e33512b49d58485cf15de788be7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.250029 10029 raft_consensus.cc:385] T 43611e33512b49d58485cf15de788be7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.250095 10029 raft_consensus.cc:740] T 43611e33512b49d58485cf15de788be7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.250521 10029 consensus_queue.cc:260] T 43611e33512b49d58485cf15de788be7 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.250983 10029 ts_tablet_manager.cc:1434] T 43611e33512b49d58485cf15de788be7 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:15.251238 10029 tablet_bootstrap.cc:492] T 133e474f9133487f89c44dbfa6266ca7 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:15.252851 10029 tablet_bootstrap.cc:654] T 133e474f9133487f89c44dbfa6266ca7 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.253360 10041 ts_tablet_manager.cc:1929] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251027 21:08:15.253470 10041 log.cc:1199] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/aaa8910d9fdb4146945621ec4ebf969e
I20251027 21:08:15.253793 10041 ts_tablet_manager.cc:1950] T aaa8910d9fdb4146945621ec4ebf969e P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:15.246127 10040 ts_tablet_manager.cc:1916] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.252625 11676 raft_consensus.cc:493] T 8cccd253ae2a46328dc9f767b8e9ee97 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.254264 11676 raft_consensus.cc:515] T 8cccd253ae2a46328dc9f767b8e9ee97 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.255198 11676 leader_election.cc:290] T 8cccd253ae2a46328dc9f767b8e9ee97 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:15.255491 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet aaa8910d9fdb4146945621ec4ebf969e (table default.Table_cc8a1775cbcb4e0eaeb8c90bf9d61c04 [id=bb676562b97f48088767941239cfbf33]) successfully deleted
I20251027 21:08:15.256150 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cccd253ae2a46328dc9f767b8e9ee97" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:15.256491 9909 raft_consensus.cc:2393] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77b0513f17c0472088901a1ce8b08218 in current term 1: Already voted for candidate 51902b6b61ff4d5584ff21896e847e14 in this term.
I20251027 21:08:15.256928 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "952583fd7fde4b4baccdb98c01bdf9e1" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
W20251027 21:08:15.257005 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet d91fced778784b65be2486e24426d1b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d91fced778784b65be2486e24426d1b5 already in progress: creating tablet
I20251027 21:08:15.257112 9630 raft_consensus.cc:3060] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.257479 10041 tablet_replica.cc:333] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:15.257714 10041 raft_consensus.cc:2243] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.257851 10041 raft_consensus.cc:2272] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.258425 10041 ts_tablet_manager.cc:1916] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.265575 11373 raft_consensus.cc:493] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.265815 11373 raft_consensus.cc:515] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.267096 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cccd253ae2a46328dc9f767b8e9ee97" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:15.267211 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93a3b22cdc1543aebcc7fe72f2f2c0ae" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:15.267417 9925 raft_consensus.cc:2468] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:15.267727 9710 leader_election.cc:304] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:15.267972 9630 raft_consensus.cc:2468] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:15.268530 9710 leader_election.cc:304] T 8cccd253ae2a46328dc9f767b8e9ee97 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:15.268010 9568 leader_election.cc:304] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:15.269166 11512 raft_consensus.cc:2804] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:15.269258 11512 raft_consensus.cc:493] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:15.269328 11512 raft_consensus.cc:3060] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.270637 11626 raft_consensus.cc:3060] T 8cccd253ae2a46328dc9f767b8e9ee97 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.271926 11512 raft_consensus.cc:515] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.272420 11512 leader_election.cc:290] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.273156 11626 raft_consensus.cc:2749] T 8cccd253ae2a46328dc9f767b8e9ee97 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.273753 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93a3b22cdc1543aebcc7fe72f2f2c0ae" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:15.274181 9710 leader_election.cc:304] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:15.274463 11626 raft_consensus.cc:2804] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
W20251027 21:08:15.278784 9566 leader_election.cc:343] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.280573 10043 ts_tablet_manager.cc:1929] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:15.268626 11676 raft_consensus.cc:2804] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:15.267773 10040 ts_tablet_manager.cc:1929] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:15.285286 10040 log.cc:1199] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/7425b6b0184e4d0b871777b25d904ac4
I20251027 21:08:15.285816 10040 ts_tablet_manager.cc:1950] T 7425b6b0184e4d0b871777b25d904ac4 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:15.286290 10043 log.cc:1199] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/7292bc6c22ca4ab5b8f53a69520833ba
I20251027 21:08:15.286893 11512 consensus_queue.cc:1048] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:15.287360 10040 tablet_replica.cc:333] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:15.287482 10029 tablet_bootstrap.cc:492] T 133e474f9133487f89c44dbfa6266ca7 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:15.287655 10029 ts_tablet_manager.cc:1403] T 133e474f9133487f89c44dbfa6266ca7 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.002s
I20251027 21:08:15.289621 10029 raft_consensus.cc:359] T 133e474f9133487f89c44dbfa6266ca7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.289788 10029 raft_consensus.cc:385] T 133e474f9133487f89c44dbfa6266ca7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.289875 10029 raft_consensus.cc:740] T 133e474f9133487f89c44dbfa6266ca7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.290762 10029 consensus_queue.cc:260] T 133e474f9133487f89c44dbfa6266ca7 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.291316 10029 ts_tablet_manager.cc:1434] T 133e474f9133487f89c44dbfa6266ca7 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251027 21:08:15.291659 11373 leader_election.cc:290] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.292068 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93a3b22cdc1543aebcc7fe72f2f2c0ae" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:15.292258 9909 raft_consensus.cc:3060] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.292932 10029 tablet_bootstrap.cc:492] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:15.295532 10029 tablet_bootstrap.cc:654] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.296118 9909 raft_consensus.cc:2468] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:15.296594 9568 leader_election.cc:304] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:15.297293 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93a3b22cdc1543aebcc7fe72f2f2c0ae" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:15.297925 9566 leader_election.cc:343] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.298688 11373 raft_consensus.cc:2804] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:15.299991 11373 raft_consensus.cc:697] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:15.300374 11373 consensus_queue.cc:237] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.303237 10029 tablet_bootstrap.cc:492] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:15.303371 10029 ts_tablet_manager.cc:1403] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20251027 21:08:15.305428 10029 raft_consensus.cc:359] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.305591 10029 raft_consensus.cc:385] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.305657 10029 raft_consensus.cc:740] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.306641 10029 consensus_queue.cc:260] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.310010 10029 ts_tablet_manager.cc:1434] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251027 21:08:15.310812 10040 raft_consensus.cc:2243] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.311057 10029 tablet_bootstrap.cc:492] T 95446c4333424f10a71829ce32d857e4 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:15.311872 10040 pending_rounds.cc:70] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
I20251027 21:08:15.311986 10040 pending_rounds.cc:77] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510704536158208 op_type: NO_OP noop_request { }
I20251027 21:08:15.312121 10040 raft_consensus.cc:2889] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:15.312242 10040 raft_consensus.cc:2272] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.312321 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6f6da8160dd64d67b34d80b00ab4029b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:15.312903 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f6da8160dd64d67b34d80b00ab4029b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:15.312938 10040 ts_tablet_manager.cc:1916] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.314090 10029 tablet_bootstrap.cc:654] T 95446c4333424f10a71829ce32d857e4 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.317665 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_bba03739a01b4ed09f57e8b629ae3690"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:15.318485 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_501dbc8bbb924002872a8eb8d1d7b517"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:15.319586 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_501dbc8bbb924002872a8eb8d1d7b517 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:15.319942 10040 ts_tablet_manager.cc:1929] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:15.320060 10040 log.cc:1199] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/aaaf70c950044387a63820079db3c829
I20251027 21:08:15.320488 10040 ts_tablet_manager.cc:1950] T aaaf70c950044387a63820079db3c829 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:15.318826 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e68bfc4835674033aaef4620bb648593"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:15.321101 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e68bfc4835674033aaef4620bb648593 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:15.321234 10029 tablet_bootstrap.cc:492] T 95446c4333424f10a71829ce32d857e4 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:15.321434 10029 ts_tablet_manager.cc:1403] T 95446c4333424f10a71829ce32d857e4 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20251027 21:08:15.322208 10029 raft_consensus.cc:359] T 95446c4333424f10a71829ce32d857e4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.322383 10029 raft_consensus.cc:385] T 95446c4333424f10a71829ce32d857e4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.319100 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_57de17bcd9ef4ca3b5153678366c719d" } new_table_name: "default.Table_40e2c2880c254ad898d7882c99b92fa5" modify_external_catalogs: true
W20251027 21:08:15.324450 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bba03739a01b4ed09f57e8b629ae3690 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:15.326298 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f639bf82a7f6412ea1aecca4f90193e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:15.326725 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f639bf82a7f6412ea1aecca4f90193e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:15.327993 10040 tablet_replica.cc:333] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:15.328329 10040 raft_consensus.cc:2243] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.328688 10040 raft_consensus.cc:2272] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.329494 10040 ts_tablet_manager.cc:1916] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.329928 11411 raft_consensus.cc:493] T 133e474f9133487f89c44dbfa6266ca7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.330128 11411 raft_consensus.cc:515] T 133e474f9133487f89c44dbfa6266ca7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.330767 11411 leader_election.cc:290] T 133e474f9133487f89c44dbfa6266ca7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.334049 11411 raft_consensus.cc:493] T e10ae34eaaf3476594813de8e96c7e77 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.334246 11411 raft_consensus.cc:515] T e10ae34eaaf3476594813de8e96c7e77 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.334512 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 3ef2fff1381b465a89eeb5f82ae4fd12
I20251027 21:08:15.335058 11411 leader_election.cc:290] T e10ae34eaaf3476594813de8e96c7e77 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.335311 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9c71cb1852864cb1ae2e8e650536ab77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:15.335781 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c71cb1852864cb1ae2e8e650536ab77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:15.337430 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ef2fff1381b465a89eeb5f82ae4fd12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:15.338629 9568 leader_election.cc:343] T e10ae34eaaf3476594813de8e96c7e77 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: e10ae34eaaf3476594813de8e96c7e77
I20251027 21:08:15.338853 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ef2fff1381b465a89eeb5f82ae4fd12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:15.339076 11713 tablet.cc:1722] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:15.339753 11411 raft_consensus.cc:493] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.339856 11411 raft_consensus.cc:515] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.340454 11411 leader_election.cc:290] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.340919 9911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a48067b40324c79b1ee4eb431ce4a46" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:15.341084 9911 raft_consensus.cc:2468] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:15.341411 9568 leader_election.cc:304] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:15.341650 11411 raft_consensus.cc:2804] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:15.341706 11411 raft_consensus.cc:493] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:15.341754 11411 raft_consensus.cc:3060] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.344312 11411 raft_consensus.cc:515] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.345132 11411 leader_election.cc:290] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.349121 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet d332d8e8f0c949589d1fd29b4bd7f076
I20251027 21:08:15.350353 10040 ts_tablet_manager.cc:1929] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:15.350453 10040 log.cc:1199] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/cb95618d9fd54a4f8121b43e717d5616
I20251027 21:08:15.350404 9911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a48067b40324c79b1ee4eb431ce4a46" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:15.350567 9911 raft_consensus.cc:3060] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.350759 10040 ts_tablet_manager.cc:1950] T cb95618d9fd54a4f8121b43e717d5616 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:15.351948 9891 tablet_service.cc:1552] Processing DeleteTablet for tablet d332d8e8f0c949589d1fd29b4bd7f076 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:15.352348 10040 tablet_replica.cc:333] T 30ce703650c94c06b93bce976cb5d1fd P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:15.352797 9603 tablet_service.cc:1552] Processing DeleteTablet for tablet d332d8e8f0c949589d1fd29b4bd7f076 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:15.353106 10040 raft_consensus.cc:2243] T 30ce703650c94c06b93bce976cb5d1fd P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.353385 10040 raft_consensus.cc:2272] T 30ce703650c94c06b93bce976cb5d1fd P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.353662 9911 raft_consensus.cc:2468] T 1a48067b40324c79b1ee4eb431ce4a46 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:15.354035 9568 leader_election.cc:304] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:15.354292 11411 raft_consensus.cc:2804] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:15.354393 11411 raft_consensus.cc:697] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:15.354607 11411 consensus_queue.cc:237] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.358335 9603 tablet_service.cc:1505] Processing CreateTablet for tablet 0a0334b492b94ccc8a42f2d6aa0302e0 (DEFAULT_TABLE table=default.Table_0655ada3126c49049155b9671bcbe409 [id=155318e67b2845fb83e07368aa5864de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.358726 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a0334b492b94ccc8a42f2d6aa0302e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.358979 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 0a0334b492b94ccc8a42f2d6aa0302e0 (DEFAULT_TABLE table=default.Table_0655ada3126c49049155b9671bcbe409 [id=155318e67b2845fb83e07368aa5864de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.359334 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a0334b492b94ccc8a42f2d6aa0302e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.359728 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 4778d342fa4a431fb55bf8460363c6f6 (DEFAULT_TABLE table=default.Table_ea0fba68a1e2447bb0379255e2aa61d0 [id=2ffea053312b41ac94db535f4ef6bb65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.360040 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4778d342fa4a431fb55bf8460363c6f6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:15.360013 9297 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 132527us
I20251027 21:08:15.360292 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 9ed4f9c61ad54e1292281b30bcd19533 (table default.Table_796614d1dc4148699dfa07316ea8febf [id=5f4093d9346248dba86e86cf2fbacd92]) successfully deleted
I20251027 21:08:15.360687 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 7425b6b0184e4d0b871777b25d904ac4 (table default.Table_92cfd00027b045f3b151deec3558ece6 [id=479548b373ef42b4ad8683a9dcfb3ddc]) successfully deleted
I20251027 21:08:15.360807 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet aaaf70c950044387a63820079db3c829 (table default.Table_e08f0cbb71674a44a697686c1b6effc9 [id=471f86ddb9de40a7ae58ec202926a1fd]) successfully deleted
W20251027 21:08:15.360936 9297 catalog_manager.cc:4952] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 5d18ff6d8ddb465fbf479fdc7ebe7a05 because the tablet was not found. No further retry: Not found: Tablet not found: 5d18ff6d8ddb465fbf479fdc7ebe7a05
I20251027 21:08:15.361086 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet cb95618d9fd54a4f8121b43e717d5616 (table default.Table_f2775123ffc8464eaf6d5e7d4ee692b0 [id=3298547a152045fdb51d360f4af22108]) successfully deleted
I20251027 21:08:15.362231 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 4778d342fa4a431fb55bf8460363c6f6 (DEFAULT_TABLE table=default.Table_ea0fba68a1e2447bb0379255e2aa61d0 [id=2ffea053312b41ac94db535f4ef6bb65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.362777 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4778d342fa4a431fb55bf8460363c6f6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:15.363277 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fad3a2aa57f04f40ba65bf1b2832b9a6 consensus state has no leader
I20251027 21:08:15.366477 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet d332d8e8f0c949589d1fd29b4bd7f076 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:15.366844 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ef2fff1381b465a89eeb5f82ae4fd12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:14 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:15.362330 10031 tablet_bootstrap.cc:492] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:15.367653 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 4778d342fa4a431fb55bf8460363c6f6 (DEFAULT_TABLE table=default.Table_ea0fba68a1e2447bb0379255e2aa61d0 [id=2ffea053312b41ac94db535f4ef6bb65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.367771 10031 ts_tablet_manager.cc:1403] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.325s user 0.002s sys 0.004s
I20251027 21:08:15.368057 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4778d342fa4a431fb55bf8460363c6f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.368537 10031 raft_consensus.cc:359] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.368702 10031 raft_consensus.cc:385] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.368784 10040 ts_tablet_manager.cc:1916] T 30ce703650c94c06b93bce976cb5d1fd P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.369333 11543 raft_consensus.cc:493] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.369443 11543 raft_consensus.cc:515] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.370023 11543 leader_election.cc:290] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.370462 9308 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 72c1dc3b9eff47ee81df821deb789307 (table default.Table_ff5458dda9f042e189bd8cd69c06d025 [id=2a96776404334cf79ddb611a234a5844]). Expected version 2 got 0
I20251027 21:08:15.370621 9308 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 67cb71ce43244d8ab9e71ead6bb315c9 (table default.Table_6327cacf4ed54d56b522291ad9594784 [id=3eb54d245ec64ee48022c541c2e49b17]). Expected version 2 got 0
I20251027 21:08:15.370647 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "479d2cc8b8ce4c26841f50eea7c0e09a" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:15.371263 9860 leader_election.cc:343] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.371716 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "479d2cc8b8ce4c26841f50eea7c0e09a" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:15.372141 9859 leader_election.cc:343] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.372231 9859 leader_election.cc:304] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.372331 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c78ef9ebe86b4da08443e37c83d4e80d" } new_table_name: "default.Table_3c5157c8728d40dfb6ffe4ecb2ab80d3" modify_external_catalogs: true
I20251027 21:08:15.372679 11543 raft_consensus.cc:2749] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.368791 10031 raft_consensus.cc:740] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.373119 10031 consensus_queue.cc:260] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.373626 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9797869b2b084612acc50e375cd116f8" } new_table_name: "default.Table_da011a34fc8b4689af1b44b83b3096bf" modify_external_catalogs: true
I20251027 21:08:15.375018 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_ea0fba68a1e2447bb0379255e2aa61d0" } new_table_name: "default.Table_161619c1b3374ded8b33e5443528d972" modify_external_catalogs: true
I20251027 21:08:15.378093 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet e6ed9d7a700446edafcd0daa02aa7ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:15.378228 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet aaaf70c950044387a63820079db3c829 with delete_type TABLET_DATA_DELETED (replaced by tablet 262b4fb71a2d40fc879da56f8c51ef83) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:15.382110 9312 master_service.cc:954] ReplaceTablet: received request to replace tablet 123e6e3748574ea581f67fff49037762 from {username='slave'} at 127.0.0.1:41710
W20251027 21:08:15.383651 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c442d337f5f945c097fa71b3833654f3 consensus state has no leader
I20251027 21:08:15.383807 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet 37ae3b6bf590473e8e09092fc5f7df1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:15.385457 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a656c74e16c24bde86338f71c0836028"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:15.386042 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a656c74e16c24bde86338f71c0836028 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:15.387809 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c47ddf1ee1b944d28fb5a3b4db111de5" } modify_external_catalogs: true
I20251027 21:08:15.387933 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c47ddf1ee1b944d28fb5a3b4db111de5" } modify_external_catalogs: true
I20251027 21:08:15.389369 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e6ea09cd17ac421eb0f6a9411d2532e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:15.389815 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6ea09cd17ac421eb0f6a9411d2532e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:15.391366 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0f0e665cd21f443b9c1e7f3fa32c66da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:15.391793 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f0e665cd21f443b9c1e7f3fa32c66da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:15.391942 9568 leader_election.cc:343] T 133e474f9133487f89c44dbfa6266ca7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 133e474f9133487f89c44dbfa6266ca7
I20251027 21:08:15.394423 10040 ts_tablet_manager.cc:1929] T 30ce703650c94c06b93bce976cb5d1fd P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:15.402928 10032 tablet_bootstrap.cc:492] T 4bfa70ded2f54e3fb2ca744c723395e2 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:15.403256 10032 ts_tablet_manager.cc:1403] T 4bfa70ded2f54e3fb2ca744c723395e2 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.379s user 0.032s sys 0.034s
I20251027 21:08:15.404119 10032 raft_consensus.cc:359] T 4bfa70ded2f54e3fb2ca744c723395e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.404385 10032 raft_consensus.cc:385] T 4bfa70ded2f54e3fb2ca744c723395e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.404493 10032 raft_consensus.cc:740] T 4bfa70ded2f54e3fb2ca744c723395e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.404757 10032 consensus_queue.cc:260] T 4bfa70ded2f54e3fb2ca744c723395e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.405447 10032 ts_tablet_manager.cc:1434] T 4bfa70ded2f54e3fb2ca744c723395e2 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:15.405758 10032 tablet_bootstrap.cc:492] T 95f9595784df434dbb0dd29d9a05511e P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:15.407760 10032 tablet_bootstrap.cc:654] T 95f9595784df434dbb0dd29d9a05511e P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.412197 10040 log.cc:1199] T 30ce703650c94c06b93bce976cb5d1fd P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/30ce703650c94c06b93bce976cb5d1fd
I20251027 21:08:15.413105 10040 ts_tablet_manager.cc:1950] T 30ce703650c94c06b93bce976cb5d1fd P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:15.414498 10040 tablet_replica.cc:333] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:15.415871 10040 raft_consensus.cc:2243] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:15.416628 10040 pending_rounds.cc:70] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
I20251027 21:08:15.417032 10041 ts_tablet_manager.cc:1929] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:15.417201 10041 log.cc:1199] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/ae0ee7cd2c1b406f84285f25420588d0
I20251027 21:08:15.417646 10041 ts_tablet_manager.cc:1950] T ae0ee7cd2c1b406f84285f25420588d0 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:15.417264 10040 pending_rounds.cc:77] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510710935597056 op_type: NO_OP noop_request { }
I20251027 21:08:15.418951 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 30ce703650c94c06b93bce976cb5d1fd (table default.Table_cfb36d57b28d49f88f50a05650a40b49 [id=b82c052176d94b1292aa47473d08a68c]) successfully deleted
I20251027 21:08:15.419164 10040 raft_consensus.cc:2889] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:15.419494 10040 raft_consensus.cc:2272] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.420672 10040 ts_tablet_manager.cc:1916] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.422799 11725 raft_consensus.cc:493] T 986bf148b3e9482fa2fe53bbac54750d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.428543 11725 raft_consensus.cc:515] T 986bf148b3e9482fa2fe53bbac54750d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.429725 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "986bf148b3e9482fa2fe53bbac54750d" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:15.430313 11725 leader_election.cc:290] T 986bf148b3e9482fa2fe53bbac54750d P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:15.425233 10031 ts_tablet_manager.cc:1434] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.057s user 0.006s sys 0.005s
I20251027 21:08:15.430699 10031 tablet_bootstrap.cc:492] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:15.431627 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_877147b58eff438190609ce0142c08f5" } modify_external_catalogs: true
I20251027 21:08:15.431797 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_877147b58eff438190609ce0142c08f5" } modify_external_catalogs: true
I20251027 21:08:15.431963 10031 tablet_bootstrap.cc:654] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.422904 11676 raft_consensus.cc:697] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:15.432700 11676 consensus_queue.cc:237] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:15.433111 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f6434c5ccd14bae97ebd000ae7b2fdb consensus state has no leader
I20251027 21:08:15.433404 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6a6205df84c04bf8a9fb02bef162312c" } modify_external_catalogs: true
I20251027 21:08:15.433557 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6a6205df84c04bf8a9fb02bef162312c" } modify_external_catalogs: true
I20251027 21:08:15.436312 9307 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 8cccd253ae2a46328dc9f767b8e9ee97 (table default.Table_4ba26495af344e3cb082a4b781f1b208 [id=501b230a824d4a17bb1903c8409407b7]). Expected version 2 got 0
I20251027 21:08:15.438369 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 958dafe1e6844f19902524e8fa27d69a
W20251027 21:08:15.440272 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4778d342fa4a431fb55bf8460363c6f6 consensus state has no leader
I20251027 21:08:15.442340 10031 tablet_bootstrap.cc:492] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:15.442509 10031 ts_tablet_manager.cc:1403] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20251027 21:08:15.442685 10032 tablet_bootstrap.cc:492] T 95f9595784df434dbb0dd29d9a05511e P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:15.442772 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 958dafe1e6844f19902524e8fa27d69a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:15.442862 10032 ts_tablet_manager.cc:1403] T 95f9595784df434dbb0dd29d9a05511e P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.001s
I20251027 21:08:15.443280 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet 958dafe1e6844f19902524e8fa27d69a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:15.443696 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 3f734b7cd8b9484fbcfe130dc60051ea (DEFAULT_TABLE table=default.Table_5c1f633e896747d6a15a4998a12a1ef9 [id=627d8d7d55e74342b15b0e801ed4974d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.444346 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f734b7cd8b9484fbcfe130dc60051ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.445832 9603 tablet_service.cc:1505] Processing CreateTablet for tablet d86d1a605afe4c29b1f2a139adf0259f (DEFAULT_TABLE table=default.Table_b81999a26dfe4176a646bd057c455f92 [id=5a7302b9fde24d2c9d289e34e0ef4ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.446314 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d86d1a605afe4c29b1f2a139adf0259f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.446411 9613 tablet_service.cc:1505] Processing CreateTablet for tablet f954566c02a341e7b445143489f609ff (DEFAULT_TABLE table=default.Table_a01b887ede5f492b9175988cbfec56c1 [id=fbb4b554eabd4888b68d71ae38932da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.446766 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f954566c02a341e7b445143489f609ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.450462 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 3f734b7cd8b9484fbcfe130dc60051ea (DEFAULT_TABLE table=default.Table_5c1f633e896747d6a15a4998a12a1ef9 [id=627d8d7d55e74342b15b0e801ed4974d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.450899 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f734b7cd8b9484fbcfe130dc60051ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.451243 9607 tablet_service.cc:1505] Processing CreateTablet for tablet e070a791682e4924970c0d0f5d1ddf83 (DEFAULT_TABLE table=default.Table_877147b58eff438190609ce0142c08f5 [id=05da7a8eb8ea441d90763f0bf6abd69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.451592 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e070a791682e4924970c0d0f5d1ddf83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.452440 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "133e474f9133487f89c44dbfa6266ca7" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:15.452843 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e10ae34eaaf3476594813de8e96c7e77" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:15.453210 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a48067b40324c79b1ee4eb431ce4a46" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:15.453601 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a48067b40324c79b1ee4eb431ce4a46" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:15.454335 9299 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet c442d337f5f945c097fa71b3833654f3 (table default.Table_40e2c2880c254ad898d7882c99b92fa5 [id=bdeb9e4bcc9d40d19992b3cc93aa0126], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c442d337f5f945c097fa71b3833654f3 consensus state has no leader
I20251027 21:08:15.454666 11626 raft_consensus.cc:697] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:15.454905 10029 raft_consensus.cc:740] T 95446c4333424f10a71829ce32d857e4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.455078 11626 consensus_queue.cc:237] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.455045 9305 catalog_manager.cc:5649] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:15.455452 9305 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 371051fb3491426e8cf7797317baa3db (table default.Table_d481a540123e4620874b629ce6db65a7 [id=3b0044836ccc4b4082775de6eb111e49]). Expected version 1 got 0
I20251027 21:08:15.455935 9305 catalog_manager.cc:5649] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:15.456364 9305 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 9be6b996df8145b9b0df1a2d2dc0ef03 (table default.Table_d09f1941d15f4517ac7561a66dd9b5e9 [id=b57564104d854b67a653c03de0650c09]). Expected version 1 got 0
I20251027 21:08:15.456602 10031 raft_consensus.cc:359] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.456789 10031 raft_consensus.cc:385] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.456604 9305 catalog_manager.cc:5649] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:15.456856 10031 raft_consensus.cc:740] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.457074 9305 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 334c580862bd4bf09be3f1710aa6a092 (table default.Table_06c15ff076cd4c24bc49b16b01bff965 [id=cf9029e184114ac0a5b641a496701122]). Expected version 1 got 0
I20251027 21:08:15.457494 11543 raft_consensus.cc:493] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.457571 9305 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet a9558d53008640739ac6fb45f1352fc0 (table default.Table_9821265dde764aee9571790dd10b4910 [id=c575fb3af44c46db8c198569241a425f]). Expected version 2 got 0
I20251027 21:08:15.457638 11543 raft_consensus.cc:515] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:15.457806 9305 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 59300e16b3e148a08a217fc24d1e572f (table default.Table_37d44f2ffd8b4816b4bff0b1960cd275 [id=a1dea3b74aa94995991d3123fbb2bf8d]). Expected version 1 got 0
I20251027 21:08:15.458097 9305 catalog_manager.cc:5649] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:15.458282 11543 leader_election.cc:290] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.458273 10029 consensus_queue.cc:260] T 95446c4333424f10a71829ce32d857e4 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.458441 9305 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 76bee322d0b94abe9166170733608853 (table default.Table_4b0bc60a7c5e4888b79210296b7dfa20 [id=c9e7aa435d5b48e096c2b67b0e678eeb]). Expected version 1 got 0
I20251027 21:08:15.458662 9305 catalog_manager.cc:5649] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:15.458951 10029 ts_tablet_manager.cc:1434] T 95446c4333424f10a71829ce32d857e4 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.137s user 0.014s sys 0.000s
I20251027 21:08:15.458950 9305 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 93a3b22cdc1543aebcc7fe72f2f2c0ae (table default.Table_c5a552fcdf214416a4b9217758c32b80 [id=8646c99b90414f4fa266cd7a31123d8e]). Expected version 6 got 0
I20251027 21:08:15.459195 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37ae3b6bf590473e8e09092fc5f7df1b" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:15.459224 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37ae3b6bf590473e8e09092fc5f7df1b" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:15.459605 10031 consensus_queue.cc:260] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:15.459657 9859 leader_election.cc:343] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:15.459960 9860 leader_election.cc:343] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.460047 9860 leader_election.cc:304] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.460474 10031 ts_tablet_manager.cc:1434] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20251027 21:08:15.460510 11543 raft_consensus.cc:2749] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.460727 10031 tablet_bootstrap.cc:492] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:08:15.462044 9566 consensus_peers.cc:597] T aaaf70c950044387a63820079db3c829 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aaaf70c950044387a63820079db3c829. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:15.462208 9566 leader_election.cc:343] T 133e474f9133487f89c44dbfa6266ca7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.462294 9566 leader_election.cc:304] T 133e474f9133487f89c44dbfa6266ca7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.286901 10043 ts_tablet_manager.cc:1950] T 7292bc6c22ca4ab5b8f53a69520833ba P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
W20251027 21:08:15.462585 9566 leader_election.cc:343] T e10ae34eaaf3476594813de8e96c7e77 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.462656 9566 leader_election.cc:304] T e10ae34eaaf3476594813de8e96c7e77 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:15.462975 9566 leader_election.cc:343] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.463101 11658 raft_consensus.cc:2749] T 133e474f9133487f89c44dbfa6266ca7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:15.463222 9566 leader_election.cc:343] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.463332 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "986bf148b3e9482fa2fe53bbac54750d" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:15.463459 9566 consensus_peers.cc:597] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 588e2f58f07b4a32a78dffd5b92ceb12. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:15.463227 11658 raft_consensus.cc:2749] T e10ae34eaaf3476594813de8e96c7e77 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.463688 11658 consensus_queue.cc:1048] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:15.463991 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1115) took 6327 ms. Trace:
I20251027 21:08:15.464107 10043 rpcz_store.cc:276] 1027 21:08:09.136031 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.136302 (+ 271us) service_pool.cc:225] Handling call
1027 21:08:15.233448 (+6097146us) ts_tablet_manager.cc:1151] Deleting tablet 7292bc6c22ca4ab5b8f53a69520833ba
1027 21:08:15.233456 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:15.237412 (+ 3956us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:15.280466 (+ 43054us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:15.463954 (+183488us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6068745}
I20251027 21:08:15.464793 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1125) took 6308 ms. Trace:
I20251027 21:08:15.464919 10043 rpcz_store.cc:276] 1027 21:08:09.156699 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.162176 (+ 5477us) service_pool.cc:225] Handling call
1027 21:08:15.464695 (+6302519us) ts_tablet_manager.cc:1151] Deleting tablet f1a931c36fec45029715b01754d0b161
1027 21:08:15.464705 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:15.464779 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6294725}
I20251027 21:08:15.465281 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1126) took 6308 ms. Trace:
I20251027 21:08:15.465394 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 7292bc6c22ca4ab5b8f53a69520833ba (table default.Table_4658a756ae8149cf9e248c65a102bdb6 [id=0a31c72c314e4e5ab0655d9b853c941c]) successfully deleted
I20251027 21:08:15.465402 10043 rpcz_store.cc:276] 1027 21:08:09.156788 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.162585 (+ 5797us) service_pool.cc:225] Handling call
1027 21:08:15.465195 (+6302610us) ts_tablet_manager.cc:1151] Deleting tablet cb5ac0fff70b491e8c8842f7a331005e
1027 21:08:15.465203 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:15.465267 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6295076}
W20251027 21:08:15.465634 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet f1a931c36fec45029715b01754d0b161 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1a931c36fec45029715b01754d0b161 already in progress: creating tablet
W20251027 21:08:15.465873 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet cb5ac0fff70b491e8c8842f7a331005e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb5ac0fff70b491e8c8842f7a331005e already in progress: creating tablet
I20251027 21:08:15.459236 10029 tablet_bootstrap.cc:492] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:15.467423 10029 tablet_bootstrap.cc:654] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.470486 9891 tablet_service.cc:1505] Processing CreateTablet for tablet d86d1a605afe4c29b1f2a139adf0259f (DEFAULT_TABLE table=default.Table_b81999a26dfe4176a646bd057c455f92 [id=5a7302b9fde24d2c9d289e34e0ef4ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.470916 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d86d1a605afe4c29b1f2a139adf0259f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.472223 10040 ts_tablet_manager.cc:1929] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:15.472330 10040 log.cc:1199] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/f4187b2f66f24055b1df170f1062cc88
I20251027 21:08:15.472628 10040 ts_tablet_manager.cc:1950] T f4187b2f66f24055b1df170f1062cc88 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:15.472649 9710 leader_election.cc:304] T 986bf148b3e9482fa2fe53bbac54750d P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:15.473006 10040 tablet_replica.cc:333] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:15.473238 11733 raft_consensus.cc:2749] T 986bf148b3e9482fa2fe53bbac54750d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.473837 10043 tablet_replica.cc:333] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:15.474081 10043 raft_consensus.cc:2243] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.474304 10043 raft_consensus.cc:2272] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.474390 9887 tablet_service.cc:1505] Processing CreateTablet for tablet f954566c02a341e7b445143489f609ff (DEFAULT_TABLE table=default.Table_a01b887ede5f492b9175988cbfec56c1 [id=fbb4b554eabd4888b68d71ae38932da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.474805 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f954566c02a341e7b445143489f609ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.474831 10043 ts_tablet_manager.cc:1916] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.483983 11411 raft_consensus.cc:493] T 5cb8ca3a50c148e6b602e949650353f8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.484105 11411 raft_consensus.cc:515] T 5cb8ca3a50c148e6b602e949650353f8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:15.484954 11411 leader_election.cc:290] T 5cb8ca3a50c148e6b602e949650353f8 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:15.485844 9568 leader_election.cc:343] T 5cb8ca3a50c148e6b602e949650353f8 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 5cb8ca3a50c148e6b602e949650353f8
I20251027 21:08:15.489102 10029 tablet_bootstrap.cc:492] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:15.489236 10029 ts_tablet_manager.cc:1403] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.001s
I20251027 21:08:15.489871 10029 raft_consensus.cc:359] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.490012 10029 raft_consensus.cc:385] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.490072 10029 raft_consensus.cc:740] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.490262 10029 consensus_queue.cc:260] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.490669 10029 ts_tablet_manager.cc:1434] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251027 21:08:15.490916 10029 tablet_bootstrap.cc:492] T cc4635c00e9b40c49ce04debc76a04bf P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:15.493026 11411 raft_consensus.cc:493] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.493566 11411 raft_consensus.cc:515] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.495062 9755 tablet_service.cc:1505] Processing CreateTablet for tablet f954566c02a341e7b445143489f609ff (DEFAULT_TABLE table=default.Table_a01b887ede5f492b9175988cbfec56c1 [id=fbb4b554eabd4888b68d71ae38932da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.495167 9736 tablet_service.cc:1505] Processing CreateTablet for tablet d86d1a605afe4c29b1f2a139adf0259f (DEFAULT_TABLE table=default.Table_b81999a26dfe4176a646bd057c455f92 [id=5a7302b9fde24d2c9d289e34e0ef4ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.495246 11411 leader_election.cc:290] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.495419 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f954566c02a341e7b445143489f609ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.495635 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d86d1a605afe4c29b1f2a139adf0259f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.497177 10031 tablet_bootstrap.cc:654] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.497218 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 3f734b7cd8b9484fbcfe130dc60051ea (DEFAULT_TABLE table=default.Table_5c1f633e896747d6a15a4998a12a1ef9 [id=627d8d7d55e74342b15b0e801ed4974d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.497558 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f734b7cd8b9484fbcfe130dc60051ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.501497 10040 raft_consensus.cc:2243] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:15.501844 10040 pending_rounds.cc:70] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
I20251027 21:08:15.501895 10040 pending_rounds.cc:77] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510714187747328 op_type: NO_OP noop_request { }
I20251027 21:08:15.501915 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet f4187b2f66f24055b1df170f1062cc88 (table default.Table_d276ca9d745f41a58deb7846600ddcc0 [id=a23a34526a1b46db9ce948189242dc02]) successfully deleted
I20251027 21:08:15.501979 10040 raft_consensus.cc:2889] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:15.502059 10040 raft_consensus.cc:2272] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.502473 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 958dafe1e6844f19902524e8fa27d69a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:15.504848 10029 tablet_bootstrap.cc:654] T cc4635c00e9b40c49ce04debc76a04bf P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.505419 11747 tablet.cc:1722] T 87d5af4607f04f92abcc7292b4176cbd P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251027 21:08:15.505846 9308 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 8cccd253ae2a46328dc9f767b8e9ee97 (table default.Table_4ba26495af344e3cb082a4b781f1b208 [id=501b230a824d4a17bb1903c8409407b7]). Expected version 2 got 0
I20251027 21:08:15.508780 11724 raft_consensus.cc:493] T 72c1dc3b9eff47ee81df821deb789307 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.508963 11724 raft_consensus.cc:515] T 72c1dc3b9eff47ee81df821deb789307 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.510022 11724 leader_election.cc:290] T 72c1dc3b9eff47ee81df821deb789307 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:15.510666 9755 tablet_service.cc:1505] Processing CreateTablet for tablet e070a791682e4924970c0d0f5d1ddf83 (DEFAULT_TABLE table=default.Table_877147b58eff438190609ce0142c08f5 [id=05da7a8eb8ea441d90763f0bf6abd69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.511054 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e070a791682e4924970c0d0f5d1ddf83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.511832 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cb8ca3a50c148e6b602e949650353f8" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:15.512531 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72c1dc3b9eff47ee81df821deb789307" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:15.513262 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9351a369dc1e4ac49f6ad419eab3a4c5" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:15.513914 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72c1dc3b9eff47ee81df821deb789307" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:15.516373 9566 leader_election.cc:343] T 5cb8ca3a50c148e6b602e949650353f8 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.516477 9566 leader_election.cc:304] T 5cb8ca3a50c148e6b602e949650353f8 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.518496 11512 raft_consensus.cc:2749] T 5cb8ca3a50c148e6b602e949650353f8 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.521773 9710 leader_election.cc:304] T 72c1dc3b9eff47ee81df821deb789307 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
W20251027 21:08:15.518791 9566 leader_election.cc:343] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.524679 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 05608a3650ef434495a1f2aeb926e79a (DEFAULT_TABLE table=default.Table_d4dbfb9efe5f4b00a0279400110e5779 [id=60a72c058f444e13acded7dd8f76724a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.525017 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05608a3650ef434495a1f2aeb926e79a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.527400 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 3bd720945d1345fea590b01987c0799c (DEFAULT_TABLE table=default.Table_7768a03566534f8bbcacc53a14a957db [id=117f33f59f9948c98a0252a8f65f62e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.527729 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bd720945d1345fea590b01987c0799c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.527944 11626 raft_consensus.cc:493] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.528184 11626 raft_consensus.cc:515] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.528949 11626 leader_election.cc:290] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:15.530372 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 3ea1048622414318b1fdc0ab034cfb31 (DEFAULT_TABLE table=default.Table_9a1b725b46c94ae7a2831ad1e047cb39 [id=569c514ed7a747e89ea937e1a4c65766]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.530654 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ea1048622414318b1fdc0ab034cfb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.531512 11724 raft_consensus.cc:2749] T 72c1dc3b9eff47ee81df821deb789307 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.534655 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e4d35f229bc46e1859ded7c8196ab9d" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:15.534827 9630 raft_consensus.cc:2468] T 0e4d35f229bc46e1859ded7c8196ab9d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
W20251027 21:08:15.538170 9711 leader_election.cc:343] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 0e4d35f229bc46e1859ded7c8196ab9d
I20251027 21:08:15.540787 9710 leader_election.cc:304] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:15.541011 11724 raft_consensus.cc:2804] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:15.541141 11724 raft_consensus.cc:493] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:15.541282 11724 raft_consensus.cc:3060] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.541858 10040 ts_tablet_manager.cc:1916] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.544838 11724 raft_consensus.cc:515] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.545701 11724 leader_election.cc:290] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:15.546254 9711 leader_election.cc:343] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 0e4d35f229bc46e1859ded7c8196ab9d
I20251027 21:08:15.546994 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e4d35f229bc46e1859ded7c8196ab9d" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:15.547147 9630 raft_consensus.cc:3060] T 0e4d35f229bc46e1859ded7c8196ab9d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.549656 9630 raft_consensus.cc:2468] T 0e4d35f229bc46e1859ded7c8196ab9d P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:15.550590 9710 leader_election.cc:304] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:15.551131 11724 raft_consensus.cc:2804] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:15.551294 11724 raft_consensus.cc:697] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:15.551568 11724 consensus_queue.cc:237] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.552008 11713 tablet.cc:1722] T 87d5af4607f04f92abcc7292b4176cbd P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251027 21:08:15.552486 11512 raft_consensus.cc:493] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.552775 10029 tablet_bootstrap.cc:492] T cc4635c00e9b40c49ce04debc76a04bf P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:15.552654 11512 raft_consensus.cc:515] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.553261 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 05608a3650ef434495a1f2aeb926e79a (DEFAULT_TABLE table=default.Table_d4dbfb9efe5f4b00a0279400110e5779 [id=60a72c058f444e13acded7dd8f76724a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.553359 11512 leader_election.cc:290] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.553671 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05608a3650ef434495a1f2aeb926e79a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.554177 10031 tablet_bootstrap.cc:492] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:15.554170 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 3ea1048622414318b1fdc0ab034cfb31 (DEFAULT_TABLE table=default.Table_9a1b725b46c94ae7a2831ad1e047cb39 [id=569c514ed7a747e89ea937e1a4c65766]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.554361 10031 ts_tablet_manager.cc:1403] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.094s user 0.002s sys 0.000s
I20251027 21:08:15.554491 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ea1048622414318b1fdc0ab034cfb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.554991 10031 raft_consensus.cc:359] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.555145 10031 raft_consensus.cc:385] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.555222 10031 raft_consensus.cc:740] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.555392 10031 consensus_queue.cc:260] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.555867 10031 ts_tablet_manager.cc:1434] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 21:08:15.556131 10031 tablet_bootstrap.cc:492] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:15.556572 9911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59300e16b3e148a08a217fc24d1e572f" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:15.556763 9911 raft_consensus.cc:2468] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:15.557749 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 3ea1048622414318b1fdc0ab034cfb31 (DEFAULT_TABLE table=default.Table_9a1b725b46c94ae7a2831ad1e047cb39 [id=569c514ed7a747e89ea937e1a4c65766]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.558202 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ea1048622414318b1fdc0ab034cfb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.559201 10031 tablet_bootstrap.cc:654] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.553610 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 3bd720945d1345fea590b01987c0799c (DEFAULT_TABLE table=default.Table_7768a03566534f8bbcacc53a14a957db [id=117f33f59f9948c98a0252a8f65f62e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.559546 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bd720945d1345fea590b01987c0799c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.560694 11512 raft_consensus.cc:493] T 8dc358445526415d8cd4f1ac6779126d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.560845 11512 raft_consensus.cc:515] T 8dc358445526415d8cd4f1ac6779126d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.561630 11512 leader_election.cc:290] T 8dc358445526415d8cd4f1ac6779126d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.561774 11512 raft_consensus.cc:493] T 371aa8a57aee466ab01652d6061db4c2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.561893 11512 raft_consensus.cc:515] T 371aa8a57aee466ab01652d6061db4c2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.563874 11512 leader_election.cc:290] T 371aa8a57aee466ab01652d6061db4c2 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.564922 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet ae0ee7cd2c1b406f84285f25420588d0 (table default.Table_e2cb137bea0d419b9791305f97bc0efd [id=251aef26011f4b728b1102e4570722b5]) successfully deleted
I20251027 21:08:15.565245 10041 tablet_replica.cc:333] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:15.565490 10041 raft_consensus.cc:2243] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.565682 10041 raft_consensus.cc:2272] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.565924 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 3bd720945d1345fea590b01987c0799c (DEFAULT_TABLE table=default.Table_7768a03566534f8bbcacc53a14a957db [id=117f33f59f9948c98a0252a8f65f62e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.566282 10041 ts_tablet_manager.cc:1916] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.566507 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bd720945d1345fea590b01987c0799c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.567641 9613 tablet_service.cc:1505] Processing CreateTablet for tablet 05608a3650ef434495a1f2aeb926e79a (DEFAULT_TABLE table=default.Table_d4dbfb9efe5f4b00a0279400110e5779 [id=60a72c058f444e13acded7dd8f76724a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.568099 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05608a3650ef434495a1f2aeb926e79a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:15.572319 9568 leader_election.cc:343] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 9351a369dc1e4ac49f6ad419eab3a4c5
I20251027 21:08:15.573031 9568 leader_election.cc:304] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.573418 9773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8dc358445526415d8cd4f1ac6779126d" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:15.573463 11657 raft_consensus.cc:2749] T 9351a369dc1e4ac49f6ad419eab3a4c5 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.573427 9893 tablet_service.cc:1505] Processing CreateTablet for tablet f66db4f14ca942258401e17b085dc490 (DEFAULT_TABLE table=default.Table_20fbc364a16e49e8b9b5a87e7c7a4065 [id=1867d126fec44b2bbbc87123b60288cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.573536 9568 leader_election.cc:304] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
W20251027 21:08:15.573743 9568 leader_election.cc:343] T 8dc358445526415d8cd4f1ac6779126d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 8dc358445526415d8cd4f1ac6779126d
I20251027 21:08:15.573820 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f66db4f14ca942258401e17b085dc490. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.573752 11411 raft_consensus.cc:2804] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:15.573873 11411 raft_consensus.cc:493] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251027 21:08:15.573892 9568 leader_election.cc:343] T 371aa8a57aee466ab01652d6061db4c2 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 371aa8a57aee466ab01652d6061db4c2
I20251027 21:08:15.573930 11411 raft_consensus.cc:3060] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.574923 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "371aa8a57aee466ab01652d6061db4c2" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:15.575569 9566 leader_election.cc:343] T 8dc358445526415d8cd4f1ac6779126d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.575659 9566 leader_election.cc:304] T 8dc358445526415d8cd4f1ac6779126d P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.576363 11657 raft_consensus.cc:2749] T 8dc358445526415d8cd4f1ac6779126d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.576490 11411 raft_consensus.cc:515] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:15.576709 9566 leader_election.cc:343] T 371aa8a57aee466ab01652d6061db4c2 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.576795 9566 leader_election.cc:304] T 371aa8a57aee466ab01652d6061db4c2 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.577039 11411 leader_election.cc:290] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.577167 11411 raft_consensus.cc:2749] T 371aa8a57aee466ab01652d6061db4c2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.578310 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59300e16b3e148a08a217fc24d1e572f" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:15.579063 9566 leader_election.cc:343] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.582247 9607 tablet_service.cc:1505] Processing CreateTablet for tablet f66db4f14ca942258401e17b085dc490 (DEFAULT_TABLE table=default.Table_20fbc364a16e49e8b9b5a87e7c7a4065 [id=1867d126fec44b2bbbc87123b60288cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.582628 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f66db4f14ca942258401e17b085dc490. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.582741 9904 tablet_service.cc:1505] Processing CreateTablet for tablet e070a791682e4924970c0d0f5d1ddf83 (DEFAULT_TABLE table=default.Table_877147b58eff438190609ce0142c08f5 [id=05da7a8eb8ea441d90763f0bf6abd69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.582821 9911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59300e16b3e148a08a217fc24d1e572f" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:15.582970 9911 raft_consensus.cc:3060] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.583092 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e070a791682e4924970c0d0f5d1ddf83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.583464 9742 tablet_service.cc:1505] Processing CreateTablet for tablet f66db4f14ca942258401e17b085dc490 (DEFAULT_TABLE table=default.Table_20fbc364a16e49e8b9b5a87e7c7a4065 [id=1867d126fec44b2bbbc87123b60288cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.583815 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f66db4f14ca942258401e17b085dc490. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.585721 9911 raft_consensus.cc:2468] T 59300e16b3e148a08a217fc24d1e572f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:15.586545 11753 tablet.cc:1722] T 87d5af4607f04f92abcc7292b4176cbd P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251027 21:08:15.589236 9568 leader_election.cc:304] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:15.589537 11658 raft_consensus.cc:2804] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:15.589687 11658 raft_consensus.cc:697] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:15.589972 11658 consensus_queue.cc:237] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.590427 10029 ts_tablet_manager.cc:1403] T cc4635c00e9b40c49ce04debc76a04bf P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.100s user 0.001s sys 0.003s
I20251027 21:08:15.591279 10029 raft_consensus.cc:359] T cc4635c00e9b40c49ce04debc76a04bf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.591444 10029 raft_consensus.cc:385] T cc4635c00e9b40c49ce04debc76a04bf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.591498 10029 raft_consensus.cc:740] T cc4635c00e9b40c49ce04debc76a04bf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.591688 10029 consensus_queue.cc:260] T cc4635c00e9b40c49ce04debc76a04bf P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.592312 10029 ts_tablet_manager.cc:1434] T cc4635c00e9b40c49ce04debc76a04bf P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:15.592602 10029 tablet_bootstrap.cc:492] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:15.596772 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 39ec426bc49f487da74961ee313bfda4 (DEFAULT_TABLE table=default.Table_a823e860186944468dd775400fd777e0 [id=7083248074fa466998ed1720b9cd8a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.598740 9603 tablet_service.cc:1505] Processing CreateTablet for tablet 39ec426bc49f487da74961ee313bfda4 (DEFAULT_TABLE table=default.Table_a823e860186944468dd775400fd777e0 [id=7083248074fa466998ed1720b9cd8a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.599081 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39ec426bc49f487da74961ee313bfda4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.602272 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 0d365070ce6448caa576a5b3e223d2e3 (DEFAULT_TABLE table=default.Table_adad32ac270d4323b3de1c871664d15d [id=720e2b1d8a72426996c997c56162dfd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.602608 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d365070ce6448caa576a5b3e223d2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.605469 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59300e16b3e148a08a217fc24d1e572f" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:15.607479 10043 ts_tablet_manager.cc:1929] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:15.608839 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 0d365070ce6448caa576a5b3e223d2e3 (DEFAULT_TABLE table=default.Table_adad32ac270d4323b3de1c871664d15d [id=720e2b1d8a72426996c997c56162dfd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.613013 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 0d365070ce6448caa576a5b3e223d2e3 (DEFAULT_TABLE table=default.Table_adad32ac270d4323b3de1c871664d15d [id=720e2b1d8a72426996c997c56162dfd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.613799 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d365070ce6448caa576a5b3e223d2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.614745 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 04553cf7c3674d6292f0709c064bde82 (DEFAULT_TABLE table=default.Table_27667baca7d8497a8f70a38f1ed37a30 [id=633781a9664348db9ecfbf6faa56ca1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.612550 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 39ec426bc49f487da74961ee313bfda4 (DEFAULT_TABLE table=default.Table_a823e860186944468dd775400fd777e0 [id=7083248074fa466998ed1720b9cd8a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.615792 10040 ts_tablet_manager.cc:1929] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:15.617429 10040 log.cc:1199] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/952583fd7fde4b4baccdb98c01bdf9e1
I20251027 21:08:15.617707 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39ec426bc49f487da74961ee313bfda4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.617640 9887 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bfa70ded2f54e3fb2ca744c723395e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:11 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:15.612601 9602 tablet_service.cc:1505] Processing CreateTablet for tablet 04553cf7c3674d6292f0709c064bde82 (DEFAULT_TABLE table=default.Table_27667baca7d8497a8f70a38f1ed37a30 [id=633781a9664348db9ecfbf6faa56ca1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.614416 10029 tablet_bootstrap.cc:654] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:15.616701 9566 leader_election.cc:343] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.622447 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04553cf7c3674d6292f0709c064bde82. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.623265 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1284db19502347cdbb233e995dcca267" } modify_external_catalogs: true
I20251027 21:08:15.623728 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1284db19502347cdbb233e995dcca267" } modify_external_catalogs: true
I20251027 21:08:15.618383 10031 tablet_bootstrap.cc:492] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
W20251027 21:08:15.624356 9566 consensus_peers.cc:597] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:15.623417 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39ec426bc49f487da74961ee313bfda4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.626880 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d365070ce6448caa576a5b3e223d2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.623905 10040 ts_tablet_manager.cc:1950] T 952583fd7fde4b4baccdb98c01bdf9e1 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:15.627442 11747 tablet.cc:1722] T 8cccd253ae2a46328dc9f767b8e9ee97 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:15.624006 10031 ts_tablet_manager.cc:1403] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.000s
I20251027 21:08:15.628068 10040 tablet_replica.cc:333] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:15.628309 10040 raft_consensus.cc:2243] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:15.628547 10040 pending_rounds.cc:70] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
I20251027 21:08:15.628638 10040 pending_rounds.cc:77] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510714094878720 op_type: NO_OP noop_request { }
I20251027 21:08:15.628768 10040 raft_consensus.cc:2889] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:15.628890 10040 raft_consensus.cc:2272] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.629259 10043 log.cc:1199] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/520f7142f8af4599ac4880b0692e3151
I20251027 21:08:15.629400 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04553cf7c3674d6292f0709c064bde82. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.629143 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 952583fd7fde4b4baccdb98c01bdf9e1 (table default.Table_df5f4de40f7f46728f38580d23124544 [id=7527117eef9d46a694977cbf2c073974]) successfully deleted
I20251027 21:08:15.628914 10031 raft_consensus.cc:359] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:15.629662 10031 raft_consensus.cc:385] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.629707 10043 ts_tablet_manager.cc:1950] T 520f7142f8af4599ac4880b0692e3151 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:15.629731 10031 raft_consensus.cc:740] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.630082 10031 consensus_queue.cc:260] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:15.630530 10031 ts_tablet_manager.cc:1434] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:15.630914 10031 tablet_bootstrap.cc:492] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:15.633862 10031 tablet_bootstrap.cc:654] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.635488 10040 ts_tablet_manager.cc:1916] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.637288 9887 tablet_service.cc:1505] Processing CreateTablet for tablet 04553cf7c3674d6292f0709c064bde82 (DEFAULT_TABLE table=default.Table_27667baca7d8497a8f70a38f1ed37a30 [id=633781a9664348db9ecfbf6faa56ca1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:15.637760 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04553cf7c3674d6292f0709c064bde82. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:15.638768 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1127) took 6481 ms. Trace:
I20251027 21:08:15.635452 11713 tablet.cc:1722] T 8cccd253ae2a46328dc9f767b8e9ee97 P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:15.638861 10043 rpcz_store.cc:276] 1027 21:08:09.156882 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.162638 (+ 5756us) service_pool.cc:225] Handling call
1027 21:08:15.473798 (+6311160us) ts_tablet_manager.cc:1151] Deleting tablet 520f7142f8af4599ac4880b0692e3151
1027 21:08:15.473805 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:15.530093 (+ 56288us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:15.607398 (+ 77305us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:15.638748 (+ 31350us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6303505}
I20251027 21:08:15.639190 10043 tablet_replica.cc:333] T 95446c4333424f10a71829ce32d857e4 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:15.639400 10043 raft_consensus.cc:2243] T 95446c4333424f10a71829ce32d857e4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.639547 10043 raft_consensus.cc:2272] T 95446c4333424f10a71829ce32d857e4 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.646934 11724 raft_consensus.cc:493] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.647137 11724 raft_consensus.cc:515] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.649739 11724 leader_election.cc:290] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:15.652449 10043 ts_tablet_manager.cc:1916] T 95446c4333424f10a71829ce32d857e4 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.652952 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 520f7142f8af4599ac4880b0692e3151 (table default.Table_e6955caf15904f3bae744add8221897f [id=bd9615f2023e45688ae88cd7d323e61b]) successfully deleted
I20251027 21:08:15.655826 10040 ts_tablet_manager.cc:1929] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:15.656780 10040 log.cc:1199] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/842c1a1298234d9ea2719bab29c54ae3
I20251027 21:08:15.657811 10040 ts_tablet_manager.cc:1950] T 842c1a1298234d9ea2719bab29c54ae3 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:15.659094 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 842c1a1298234d9ea2719bab29c54ae3 (table default.Table_773dc0a1742a4f9194e9b26ea3d07123 [id=4372567fda9b4a178b8adc2cde3e1704]) successfully deleted
I20251027 21:08:15.659511 11657 raft_consensus.cc:493] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fbd567f929a4f6d8e4d78feef7ed678)
I20251027 21:08:15.659641 11657 raft_consensus.cc:515] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.660176 11657 leader_election.cc:290] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:15.661228 9568 leader_election.cc:343] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 90f96c21da9944f1ade05d35787d47e2
I20251027 21:08:15.663064 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6327cacf4ed54d56b522291ad9594784" } modify_external_catalogs: true
I20251027 21:08:15.663241 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6327cacf4ed54d56b522291ad9594784" } modify_external_catalogs: true
I20251027 21:08:15.664731 9307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1f15260477b0409794ec2c672c472be3" } new_table_name: "default.Table_0518e419a0f0438dafaa1951f92ba2d4" modify_external_catalogs: true
I20251027 21:08:15.665338 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_81668449dbd44f34abb9087dd7decf2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:15.665865 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81668449dbd44f34abb9087dd7decf2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:15.666183 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_6fd9f233c7fe4ccc93d3ebaa7383e259"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:15.666661 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6fd9f233c7fe4ccc93d3ebaa7383e259 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:15.666939 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d2e87214f12c4f508a7bf6b4e03b4244"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:15.667372 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2e87214f12c4f508a7bf6b4e03b4244 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:15.667917 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_bef609ceeb424e359bd9b80f5352001c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:15.668311 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bef609ceeb424e359bd9b80f5352001c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:15.674702 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 107c608f9d904d86b74e2ad682e1bdc7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 107c608f9d904d86b74e2ad682e1bdc7 already in progress: creating tablet
I20251027 21:08:15.675298 10040 tablet_replica.cc:333] T 28a2e85021394f9d8a9ea524959b52a1 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:15.675704 10040 raft_consensus.cc:2243] T 28a2e85021394f9d8a9ea524959b52a1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.676086 10040 raft_consensus.cc:2272] T 28a2e85021394f9d8a9ea524959b52a1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.676764 10040 ts_tablet_manager.cc:1916] T 28a2e85021394f9d8a9ea524959b52a1 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.637789 11543 raft_consensus.cc:493] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.681569 11543 raft_consensus.cc:515] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.682811 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "874d5974698c448ab79f19812cb0e886" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:15.683292 11543 leader_election.cc:290] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:15.684222 9860 leader_election.cc:343] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.685168 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "874d5974698c448ab79f19812cb0e886" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:15.685698 9859 leader_election.cc:343] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.685822 9859 leader_election.cc:304] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.686236 11543 raft_consensus.cc:2749] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.652290 10032 raft_consensus.cc:359] T 95f9595784df434dbb0dd29d9a05511e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.688340 10032 raft_consensus.cc:385] T 95f9595784df434dbb0dd29d9a05511e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.688452 10032 raft_consensus.cc:740] T 95f9595784df434dbb0dd29d9a05511e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.688746 10032 consensus_queue.cc:260] T 95f9595784df434dbb0dd29d9a05511e P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.692456 10032 ts_tablet_manager.cc:1434] T 95f9595784df434dbb0dd29d9a05511e P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.041s user 0.003s sys 0.000s
I20251027 21:08:15.693064 10032 tablet_bootstrap.cc:492] T 82366f8bf4154ffd87ef45e4a4d5f54b P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:15.694387 11543 raft_consensus.cc:493] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.694496 11543 raft_consensus.cc:515] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.695005 11543 leader_election.cc:290] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.695551 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3f4e692c08244afa2c79e0a075bbcdb" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:15.695680 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3f4e692c08244afa2c79e0a075bbcdb" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:15.696108 9859 leader_election.cc:343] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:15.696506 9860 leader_election.cc:343] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.696607 9860 leader_election.cc:304] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.696991 11543 raft_consensus.cc:2749] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.697561 10032 tablet_bootstrap.cc:654] T 82366f8bf4154ffd87ef45e4a4d5f54b P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.698400 10040 ts_tablet_manager.cc:1929] T 28a2e85021394f9d8a9ea524959b52a1 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:15.699030 10040 log.cc:1199] T 28a2e85021394f9d8a9ea524959b52a1 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/28a2e85021394f9d8a9ea524959b52a1
I20251027 21:08:15.677300 11760 tablet.cc:1722] T 8cccd253ae2a46328dc9f767b8e9ee97 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251027 21:08:15.699877 10040 ts_tablet_manager.cc:1950] T 28a2e85021394f9d8a9ea524959b52a1 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:15.701052 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 28a2e85021394f9d8a9ea524959b52a1 (table default.Table_b730ec55ffb24974928bd0b0d9416460 [id=d72c1295ca1a4ec6955476a6eb5b0bb4]) successfully deleted
W20251027 21:08:15.701826 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 2044c73d56bf4434879dd65f35c5612b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2044c73d56bf4434879dd65f35c5612b already in progress: creating tablet
W20251027 21:08:15.702672 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 218d244dc8a549408863110fefbc7b20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 218d244dc8a549408863110fefbc7b20 already in progress: creating tablet
I20251027 21:08:15.703874 10032 tablet_bootstrap.cc:492] T 82366f8bf4154ffd87ef45e4a4d5f54b P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:15.704047 10032 ts_tablet_manager.cc:1403] T 82366f8bf4154ffd87ef45e4a4d5f54b P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20251027 21:08:15.704048 10040 tablet_replica.cc:333] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:15.704788 10032 raft_consensus.cc:359] T 82366f8bf4154ffd87ef45e4a4d5f54b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:15.707918 10040 raft_consensus.cc:2243] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.708246 11373 raft_consensus.cc:493] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.708408 11373 raft_consensus.cc:515] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.708971 11373 leader_election.cc:290] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.709967 10040 raft_consensus.cc:2272] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:15.703388 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 5e20fb964a4543ff9eda684acc8eb2bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e20fb964a4543ff9eda684acc8eb2bb already in progress: creating tablet
W20251027 21:08:15.710757 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet afff93eb06ab4bcca2990c8bc534b234 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afff93eb06ab4bcca2990c8bc534b234 already in progress: creating tablet
I20251027 21:08:15.711264 10029 tablet_bootstrap.cc:492] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:15.711375 10029 ts_tablet_manager.cc:1403] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.119s user 0.004s sys 0.004s
I20251027 21:08:15.711494 10040 ts_tablet_manager.cc:1916] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.711977 10029 raft_consensus.cc:359] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.712123 10029 raft_consensus.cc:385] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.712175 10029 raft_consensus.cc:740] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.712359 10029 consensus_queue.cc:260] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.713207 10029 ts_tablet_manager.cc:1434] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:15.713864 11657 consensus_queue.cc:1048] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:15.714164 11724 raft_consensus.cc:493] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.714354 10029 tablet_bootstrap.cc:492] T e477ca4b69c84327b04618d90406e19a P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:15.714283 11724 raft_consensus.cc:515] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.714865 11724 leader_election.cc:290] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:15.722007 10029 tablet_bootstrap.cc:654] T e477ca4b69c84327b04618d90406e19a P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.722342 10031 tablet_bootstrap.cc:492] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:15.722460 10031 ts_tablet_manager.cc:1403] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.092s user 0.016s sys 0.001s
I20251027 21:08:15.723169 10031 raft_consensus.cc:359] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.723312 10031 raft_consensus.cc:385] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.723351 10031 raft_consensus.cc:740] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.723495 10031 consensus_queue.cc:260] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.723886 10031 ts_tablet_manager.cc:1434] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 21:08:15.724182 10031 tablet_bootstrap.cc:492] T 823620e5486e4cf982fdf9105e8cc4f1 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:15.725750 10031 tablet_bootstrap.cc:654] T 823620e5486e4cf982fdf9105e8cc4f1 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.709573 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57ef07c764664deaa4b2e9bcae7e7973" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:15.728107 9925 raft_consensus.cc:2468] T 57ef07c764664deaa4b2e9bcae7e7973 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:15.728514 9568 leader_election.cc:304] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:15.728822 11657 raft_consensus.cc:2804] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:15.728891 11657 raft_consensus.cc:493] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:15.728943 11657 raft_consensus.cc:3060] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.731263 11657 raft_consensus.cc:515] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.732524 11657 leader_election.cc:290] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.733065 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57ef07c764664deaa4b2e9bcae7e7973" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:15.733129 10040 ts_tablet_manager.cc:1929] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:15.733222 10040 log.cc:1199] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/520f7142f8af4599ac4880b0692e3151
I20251027 21:08:15.733217 9925 raft_consensus.cc:3060] T 57ef07c764664deaa4b2e9bcae7e7973 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.733487 10040 ts_tablet_manager.cc:1950] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:15.733954 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 520f7142f8af4599ac4880b0692e3151 (table default.Table_e6955caf15904f3bae744add8221897f [id=bd9615f2023e45688ae88cd7d323e61b]) successfully deleted
I20251027 21:08:15.735952 9925 raft_consensus.cc:2468] T 57ef07c764664deaa4b2e9bcae7e7973 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
W20251027 21:08:15.736469 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet cf1751af4c774d078a4aede9d2bb8108 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf1751af4c774d078a4aede9d2bb8108 already in progress: creating tablet
I20251027 21:08:15.736790 9568 leader_election.cc:304] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:15.737098 11657 raft_consensus.cc:2804] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:15.737244 11657 raft_consensus.cc:697] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:15.737581 11657 consensus_queue.cc:237] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:15.737934 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet d332d8e8f0c949589d1fd29b4bd7f076 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d332d8e8f0c949589d1fd29b4bd7f076 already in progress: creating tablet
W20251027 21:08:15.738703 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 3ef2fff1381b465a89eeb5f82ae4fd12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ef2fff1381b465a89eeb5f82ae4fd12 already in progress: creating tablet
I20251027 21:08:15.739180 10040 tablet_replica.cc:333] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:15.739544 10040 raft_consensus.cc:2243] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.739936 10040 raft_consensus.cc:2272] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.740625 10040 ts_tablet_manager.cc:1916] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.745246 10032 raft_consensus.cc:385] T 82366f8bf4154ffd87ef45e4a4d5f54b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.745389 10032 raft_consensus.cc:740] T 82366f8bf4154ffd87ef45e4a4d5f54b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.745610 10032 consensus_queue.cc:260] T 82366f8bf4154ffd87ef45e4a4d5f54b P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:15.746373 10032 ts_tablet_manager.cc:1434] T 82366f8bf4154ffd87ef45e4a4d5f54b P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.042s user 0.001s sys 0.000s
I20251027 21:08:15.750501 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e070a791682e4924970c0d0f5d1ddf83
I20251027 21:08:15.752934 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_676ac104a790481d9043a383d7c96caf" } new_table_name: "default.Table_b357185a2a1546d7adc2e005528ef3e6" modify_external_catalogs: true
I20251027 21:08:15.750495 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 4e35961fed9c4a4b974140324fe6cc05
I20251027 21:08:15.754933 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_14eab78b90fa4336a693f1c12f10c55e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:15.755429 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14eab78b90fa4336a693f1c12f10c55e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:15.756201 10029 tablet_bootstrap.cc:492] T e477ca4b69c84327b04618d90406e19a P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:15.756292 10040 ts_tablet_manager.cc:1929] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:15.756327 10029 ts_tablet_manager.cc:1403] T e477ca4b69c84327b04618d90406e19a P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.000s
I20251027 21:08:15.756389 10040 log.cc:1199] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/e6ed9d7a700446edafcd0daa02aa7ef6
I20251027 21:08:15.756672 10040 ts_tablet_manager.cc:1950] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:15.757172 10029 raft_consensus.cc:359] T e477ca4b69c84327b04618d90406e19a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.757421 10029 raft_consensus.cc:385] T e477ca4b69c84327b04618d90406e19a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.757477 10029 raft_consensus.cc:740] T e477ca4b69c84327b04618d90406e19a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.757700 10029 consensus_queue.cc:260] T e477ca4b69c84327b04618d90406e19a P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.758075 9755 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e35961fed9c4a4b974140324fe6cc05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:15.758131 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet e070a791682e4924970c0d0f5d1ddf83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:15.758325 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e35961fed9c4a4b974140324fe6cc05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:15.759240 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet e6ed9d7a700446edafcd0daa02aa7ef6 (table default.Table_c814f5ad5b6f46f2aa8fc2ecf7ce2d73 [id=af0612025c07415fa46cf2fad2609787]) successfully deleted
W20251027 21:08:15.759954 9297 catalog_manager.cc:4952] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet aaaf70c950044387a63820079db3c829 because the tablet was not found. No further retry: Not found: Tablet not found: aaaf70c950044387a63820079db3c829
I20251027 21:08:15.760402 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 952583fd7fde4b4baccdb98c01bdf9e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:08:15.760689 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 958dafe1e6844f19902524e8fa27d69a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 958dafe1e6844f19902524e8fa27d69a already in progress: creating tablet
W20251027 21:08:15.761336 9711 leader_election.cc:343] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 520f7142f8af4599ac4880b0692e3151
W20251027 21:08:15.761685 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 4e35961fed9c4a4b974140324fe6cc05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e35961fed9c4a4b974140324fe6cc05 already in progress: creating tablet
W20251027 21:08:15.761907 9297 catalog_manager.cc:4952] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 952583fd7fde4b4baccdb98c01bdf9e1 because the tablet was not found. No further retry: Not found: Tablet not found: 952583fd7fde4b4baccdb98c01bdf9e1
W20251027 21:08:15.762357 9711 leader_election.cc:343] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: e6ed9d7a700446edafcd0daa02aa7ef6
I20251027 21:08:15.763454 10032 tablet_bootstrap.cc:492] T 72a6ca830c904373a4cca1169ae391ce P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:15.764886 10032 tablet_bootstrap.cc:654] T 72a6ca830c904373a4cca1169ae391ce P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.774300 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet e070a791682e4924970c0d0f5d1ddf83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:08:15.775157 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet e070a791682e4924970c0d0f5d1ddf83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e070a791682e4924970c0d0f5d1ddf83 already in progress: creating tablet
I20251027 21:08:15.777220 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57ef07c764664deaa4b2e9bcae7e7973" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:15.777441 9710 leader_election.cc:343] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 520f7142f8af4599ac4880b0692e3151
I20251027 21:08:15.777558 9710 leader_election.cc:304] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:15.777583 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57ef07c764664deaa4b2e9bcae7e7973" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:15.785323 10029 ts_tablet_manager.cc:1434] T e477ca4b69c84327b04618d90406e19a P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.029s user 0.003s sys 0.000s
I20251027 21:08:15.785981 10031 tablet_bootstrap.cc:492] T 823620e5486e4cf982fdf9105e8cc4f1 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:15.786023 10029 tablet_bootstrap.cc:492] T 93a31187eca14b8797737936a80e7292 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:15.786233 10031 ts_tablet_manager.cc:1403] T 823620e5486e4cf982fdf9105e8cc4f1 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.022s
I20251027 21:08:15.787221 10031 raft_consensus.cc:359] T 823620e5486e4cf982fdf9105e8cc4f1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.787453 10031 raft_consensus.cc:385] T 823620e5486e4cf982fdf9105e8cc4f1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.787511 10029 tablet_bootstrap.cc:654] T 93a31187eca14b8797737936a80e7292 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.787559 10031 raft_consensus.cc:740] T 823620e5486e4cf982fdf9105e8cc4f1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.787879 10031 consensus_queue.cc:260] T 823620e5486e4cf982fdf9105e8cc4f1 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.788504 10031 ts_tablet_manager.cc:1434] T 823620e5486e4cf982fdf9105e8cc4f1 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:15.788787 10031 tablet_bootstrap.cc:492] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:15.790038 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 67cb71ce43244d8ab9e71ead6bb315c9
I20251027 21:08:15.790028 10031 tablet_bootstrap.cc:654] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.793103 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9f54e06fe6824db28057d61b77925c57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:15.793582 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f54e06fe6824db28057d61b77925c57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:15.797021 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0c303276539a43feb144ecfce1f3d172" } new_table_name: "default.Table_4f913169315f4d75945d3599a8b6e236" modify_external_catalogs: true
I20251027 21:08:15.797943 10032 tablet_bootstrap.cc:492] T 72a6ca830c904373a4cca1169ae391ce P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:15.798064 10032 ts_tablet_manager.cc:1403] T 72a6ca830c904373a4cca1169ae391ce P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
W20251027 21:08:15.798189 9298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.125:46313, user_credentials={real_user=slave}} blocked reactor thread for 78823.4us
I20251027 21:08:15.798944 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e35961fed9c4a4b974140324fe6cc05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:15.799234 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet e070a791682e4924970c0d0f5d1ddf83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:15.800159 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d2d977e360b4ed29c9ed63a309783d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:15.800657 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 00c90ff84d3f4a7f92fda235416eed1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:03 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:15.800906 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 67cb71ce43244d8ab9e71ead6bb315c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:15.801326 10040 tablet_replica.cc:333] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:15.801537 10040 raft_consensus.cc:2243] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.801720 10040 raft_consensus.cc:2272] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.801750 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet 7292bc6c22ca4ab5b8f53a69520833ba with delete_type TABLET_DATA_DELETED (replaced by tablet 37ae3b6bf590473e8e09092fc5f7df1b) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:15.802129 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet 8dc358445526415d8cd4f1ac6779126d with delete_type TABLET_DATA_DELETED (replaced by tablet dbf29208d34b4e6f92f5394b14dcc1cc) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:15.802256 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet fdd0198d5047458aab8bf3ce08ccc0de
I20251027 21:08:15.802394 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet e10ae34eaaf3476594813de8e96c7e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:15.802636 10040 ts_tablet_manager.cc:1916] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.802896 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_9e349c15adbf444f96f4d83d3acb5990" } new_table_name: "default.Table_2fc0edf132524c0d89efe4b06380695a" modify_external_catalogs: true
I20251027 21:08:15.803012 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet 67cb71ce43244d8ab9e71ead6bb315c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:15.803686 9613 tablet_service.cc:1552] Processing DeleteTablet for tablet fdd0198d5047458aab8bf3ce08ccc0de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:15.803875 9602 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ba13a89a33549af921d3b55023faf15 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:15.799158 9899 tablet_service.cc:1552] Processing DeleteTablet for tablet 95f9595784df434dbb0dd29d9a05511e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:15.799207 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet 67cb71ce43244d8ab9e71ead6bb315c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:15.804713 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet fdd0198d5047458aab8bf3ce08ccc0de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:15.804912 10032 raft_consensus.cc:359] T 72a6ca830c904373a4cca1169ae391ce P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:15.805044 9307 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed8a50cfa7d84c1182b7ce5daf38d7cb consensus state has no leader
I20251027 21:08:15.805174 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet fdd0198d5047458aab8bf3ce08ccc0de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:15.805413 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_8dc0282abb3842ab8ef43161ba3f67d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:15.805801 10043 ts_tablet_manager.cc:1929] T 95446c4333424f10a71829ce32d857e4 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:15.805939 10043 log.cc:1199] T 95446c4333424f10a71829ce32d857e4 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/95446c4333424f10a71829ce32d857e4
W20251027 21:08:15.805925 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8dc0282abb3842ab8ef43161ba3f67d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:15.806313 10043 ts_tablet_manager.cc:1950] T 95446c4333424f10a71829ce32d857e4 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:15.807080 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1124) took 6650 ms. Trace:
I20251027 21:08:15.807196 10043 rpcz_store.cc:276] 1027 21:08:09.156593 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.162913 (+ 6320us) service_pool.cc:225] Handling call
1027 21:08:15.639136 (+6476223us) ts_tablet_manager.cc:1151] Deleting tablet 95446c4333424f10a71829ce32d857e4
1027 21:08:15.639147 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:15.676147 (+ 37000us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:15.805670 (+129523us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:15.807062 (+ 1392us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6468689}
I20251027 21:08:15.807348 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_a272b503b50445a8a4f586fdf6fc5666"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:15.807797 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a272b503b50445a8a4f586fdf6fc5666 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:15.807806 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 95446c4333424f10a71829ce32d857e4 (table default.Table_c40e78267cfd4b84b63987b1c910818f [id=3246662942e949b9ac37a58dd81f0ae9]) successfully deleted
I20251027 21:08:15.808764 11657 raft_consensus.cc:493] T 43611e33512b49d58485cf15de788be7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.808885 11657 raft_consensus.cc:515] T 43611e33512b49d58485cf15de788be7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.809165 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet 83a040332f994055a9ecf4c66720b7e1 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:15.809458 11657 leader_election.cc:290] T 43611e33512b49d58485cf15de788be7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.815611 10040 ts_tablet_manager.cc:1929] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 21:08:15.815707 10040 log.cc:1199] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/67cb71ce43244d8ab9e71ead6bb315c9
I20251027 21:08:15.816047 10040 ts_tablet_manager.cc:1950] T 67cb71ce43244d8ab9e71ead6bb315c9 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:15.816520 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 67cb71ce43244d8ab9e71ead6bb315c9 (table default.Table_6327cacf4ed54d56b522291ad9594784 [id=3eb54d245ec64ee48022c541c2e49b17]) successfully deleted
W20251027 21:08:15.817809 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet fdd0198d5047458aab8bf3ce08ccc0de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdd0198d5047458aab8bf3ce08ccc0de already in progress: creating tablet
I20251027 21:08:15.817862 9769 raft_consensus.cc:3060] T 72c1dc3b9eff47ee81df821deb789307 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.816988 11543 raft_consensus.cc:493] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.818190 11543 raft_consensus.cc:515] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.818828 11543 leader_election.cc:290] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.819381 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93d2f7df14a74731aea84927f0b7ed45" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:15.818653 9568 leader_election.cc:343] T 43611e33512b49d58485cf15de788be7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 43611e33512b49d58485cf15de788be7
W20251027 21:08:15.820197 9860 leader_election.cc:343] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.820725 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93d2f7df14a74731aea84927f0b7ed45" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:15.821412 9859 leader_election.cc:343] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.821511 9859 leader_election.cc:304] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.821908 11543 raft_consensus.cc:2749] T 93d2f7df14a74731aea84927f0b7ed45 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.822119 10032 raft_consensus.cc:385] T 72a6ca830c904373a4cca1169ae391ce P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.822296 10032 raft_consensus.cc:740] T 72a6ca830c904373a4cca1169ae391ce P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.822712 10032 consensus_queue.cc:260] T 72a6ca830c904373a4cca1169ae391ce P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.824288 9309 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 76bee322d0b94abe9166170733608853 (table default.Table_4b0bc60a7c5e4888b79210296b7dfa20 [id=c9e7aa435d5b48e096c2b67b0e678eeb]). Expected version 1 got 0
I20251027 21:08:15.833431 10032 ts_tablet_manager.cc:1434] T 72a6ca830c904373a4cca1169ae391ce P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20251027 21:08:15.838856 10032 tablet_bootstrap.cc:492] T c97cba61e08d42a7bdb3f51db0a63ec8 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:15.838790 11747 tablet.cc:1722] T 76bee322d0b94abe9166170733608853 P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251027 21:08:15.841521 9710 raft_consensus.cc:2686] T 520f7142f8af4599ac4880b0692e3151 P 77b0513f17c0472088901a1ce8b08218: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251027 21:08:15.841761 11753 tablet.cc:1722] T 76bee322d0b94abe9166170733608853 P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:08:15.842383 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1148) took 6339 ms. Trace:
I20251027 21:08:15.846323 10043 rpcz_store.cc:276] 1027 21:08:09.503166 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.503557 (+ 391us) service_pool.cc:225] Handling call
1027 21:08:15.842233 (+6338676us) ts_tablet_manager.cc:1151] Deleting tablet e24312671b4b4106a1790c61a5548a76
1027 21:08:15.842244 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:15.842363 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6338421}
I20251027 21:08:15.846694 10043 tablet_replica.cc:333] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:15.846891 10043 raft_consensus.cc:2243] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251027 21:08:15.842880 9568 consensus_peers.cc:597] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30ce703650c94c06b93bce976cb5d1fd. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:15.847036 10043 raft_consensus.cc:2272] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.847551 10043 ts_tablet_manager.cc:1916] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:15.848598 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet e24312671b4b4106a1790c61a5548a76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e24312671b4b4106a1790c61a5548a76 already in progress: creating tablet
I20251027 21:08:15.851821 10029 tablet_bootstrap.cc:492] T 93a31187eca14b8797737936a80e7292 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:15.851922 10029 ts_tablet_manager.cc:1403] T 93a31187eca14b8797737936a80e7292 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.000s
I20251027 21:08:15.852500 10029 raft_consensus.cc:359] T 93a31187eca14b8797737936a80e7292 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.852635 10029 raft_consensus.cc:385] T 93a31187eca14b8797737936a80e7292 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.852675 10029 raft_consensus.cc:740] T 93a31187eca14b8797737936a80e7292 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.852869 10029 consensus_queue.cc:260] T 93a31187eca14b8797737936a80e7292 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.853312 10029 ts_tablet_manager.cc:1434] T 93a31187eca14b8797737936a80e7292 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 21:08:15.853535 10029 tablet_bootstrap.cc:492] T e1385beb1dc14ee0b42a2f4e1f66f532 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
W20251027 21:08:15.854508 9710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.65:44531, user_credentials={real_user=slave}} blocked reactor thread for 68404.9us
I20251027 21:08:15.854911 10029 tablet_bootstrap.cc:654] T e1385beb1dc14ee0b42a2f4e1f66f532 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.855322 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6ed9d7a700446edafcd0daa02aa7ef6" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:15.855512 9630 raft_consensus.cc:2468] T e6ed9d7a700446edafcd0daa02aa7ef6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
W20251027 21:08:15.855849 9566 leader_election.cc:343] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Not found: Tablet not found: 90f96c21da9944f1ade05d35787d47e2
I20251027 21:08:15.855953 9566 leader_election.cc:304] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:15.856395 9566 consensus_peers.cc:597] T 30ce703650c94c06b93bce976cb5d1fd P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30ce703650c94c06b93bce976cb5d1fd. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:15.856611 9566 leader_election.cc:343] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:15.856945 9566 consensus_peers.cc:597] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:15.857059 9566 leader_election.cc:343] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.857473 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43611e33512b49d58485cf15de788be7" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:15.857668 11658 raft_consensus.cc:2749] T 90f96c21da9944f1ade05d35787d47e2 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251027 21:08:15.858093 9771 raft_consensus.cc:3060] T 986bf148b3e9482fa2fe53bbac54750d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:15.858923 9566 leader_election.cc:343] T 43611e33512b49d58485cf15de788be7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.859010 9566 leader_election.cc:304] T 43611e33512b49d58485cf15de788be7 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.859767 11512 raft_consensus.cc:2749] T 43611e33512b49d58485cf15de788be7 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.860896 11657 consensus_queue.cc:1048] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:15.861052 11411 raft_consensus.cc:493] T 4ba13a89a33549af921d3b55023faf15 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.861163 11411 raft_consensus.cc:515] T 4ba13a89a33549af921d3b55023faf15 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.861706 11411 leader_election.cc:290] T 4ba13a89a33549af921d3b55023faf15 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:15.862890 9568 leader_election.cc:343] T 4ba13a89a33549af921d3b55023faf15 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 4ba13a89a33549af921d3b55023faf15
I20251027 21:08:15.864917 10041 ts_tablet_manager.cc:1929] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:15.865097 10041 log.cc:1199] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/cc4635c00e9b40c49ce04debc76a04bf
I20251027 21:08:15.869601 9303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_dc89ee7309104635b4a2fa61174b0110"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:15.870203 9303 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc89ee7309104635b4a2fa61174b0110 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:15.878360 9710 leader_election.cc:304] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
W20251027 21:08:15.878654 9710 raft_consensus.cc:2686] T e6ed9d7a700446edafcd0daa02aa7ef6 P 77b0513f17c0472088901a1ce8b08218: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251027 21:08:15.879750 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f32a2c6ab909477997b0bbc71e7414a9 consensus state has no leader
I20251027 21:08:15.880038 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba13a89a33549af921d3b55023faf15" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:15.880409 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 801cfdd144e34ec3bd43a49890858edd consensus state has no leader
W20251027 21:08:15.880955 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ada34d5086484ef18e2050ac05b71725 consensus state has no leader
I20251027 21:08:15.884011 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_58ac74f08bd94b5fa9ebfb174833d34a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:15.884541 9566 leader_election.cc:343] T 4ba13a89a33549af921d3b55023faf15 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:15.884565 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58ac74f08bd94b5fa9ebfb174833d34a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:15.884642 9566 leader_election.cc:304] T 4ba13a89a33549af921d3b55023faf15 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.884963 11373 raft_consensus.cc:2749] T 4ba13a89a33549af921d3b55023faf15 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.886052 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_bba03739a01b4ed09f57e8b629ae3690" } modify_external_catalogs: true
I20251027 21:08:15.886240 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_bba03739a01b4ed09f57e8b629ae3690" } modify_external_catalogs: true
I20251027 21:08:15.886813 9308 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_baa19560a48f42c7ab0ecad353eaa283" } new_table_name: "default.Table_e0bac1ee312f4df6835b3bc3b99ad5cb" modify_external_catalogs: true
I20251027 21:08:15.888549 10041 ts_tablet_manager.cc:1950] T cc4635c00e9b40c49ce04debc76a04bf P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:15.894815 11373 consensus_queue.cc:1048] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:15.905644 10043 ts_tablet_manager.cc:1929] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:15.910079 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1335) took 1290 ms. Trace:
I20251027 21:08:15.910362 10041 rpcz_store.cc:276] 1027 21:08:14.619570 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:14.619646 (+ 76us) service_pool.cc:225] Handling call
1027 21:08:15.565197 (+945551us) ts_tablet_manager.cc:1151] Deleting tablet cc4635c00e9b40c49ce04debc76a04bf
1027 21:08:15.565207 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:15.601576 (+ 36369us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:15.864823 (+263247us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:15.910046 (+ 45223us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":945420}
I20251027 21:08:15.910851 10041 tablet_replica.cc:333] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:15.911104 10041 raft_consensus.cc:2243] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.911300 10041 raft_consensus.cc:2272] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.911908 10041 ts_tablet_manager.cc:1916] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.912547 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet cc4635c00e9b40c49ce04debc76a04bf (table default.Table_3c1a4867db0849ceb59d5900eb57eb73 [id=1af04a7db72b4f939ffae873e6e3e688]) successfully deleted
I20251027 21:08:15.929222 10041 ts_tablet_manager.cc:1929] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:15.929440 10041 log.cc:1199] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/c8f0cd910b9d4daea287b3b192a81bd2
I20251027 21:08:15.929841 10041 ts_tablet_manager.cc:1950] T c8f0cd910b9d4daea287b3b192a81bd2 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:15.931559 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1336) took 1311 ms. Trace:
I20251027 21:08:15.931694 10041 rpcz_store.cc:276] 1027 21:08:14.619689 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:14.619800 (+ 111us) service_pool.cc:225] Handling call
1027 21:08:15.910788 (+1290988us) ts_tablet_manager.cc:1151] Deleting tablet c8f0cd910b9d4daea287b3b192a81bd2
1027 21:08:15.910802 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:15.919850 (+ 9048us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:15.929133 (+ 9283us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:15.931543 (+ 2410us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1290839}
I20251027 21:08:15.932104 10041 tablet_replica.cc:333] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:15.932323 10041 raft_consensus.cc:2243] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.932504 10041 raft_consensus.cc:2272] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.932521 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet c8f0cd910b9d4daea287b3b192a81bd2 (table default.Table_7ed2c827a45d4c80a5416d12258c77c1 [id=f7d6376018f64583981d26e14762cbaf]) successfully deleted
I20251027 21:08:15.933132 10041 ts_tablet_manager.cc:1916] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:15.937088 9308 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5611e840dffc4ce6a17e2530fe78fed8 consensus state has no leader
I20251027 21:08:15.949012 11724 raft_consensus.cc:493] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.949139 11724 raft_consensus.cc:515] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:15.949709 11724 leader_election.cc:290] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:15.951226 9711 leader_election.cc:343] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: d2203ee62f7e404c95c5f86b5ae85feb
I20251027 21:08:15.952257 11543 raft_consensus.cc:493] T 82366f8bf4154ffd87ef45e4a4d5f54b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.952389 11543 raft_consensus.cc:515] T 82366f8bf4154ffd87ef45e4a4d5f54b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:15.952945 11543 leader_election.cc:290] T 82366f8bf4154ffd87ef45e4a4d5f54b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.953161 10031 tablet_bootstrap.cc:492] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:15.953536 10031 ts_tablet_manager.cc:1403] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.165s user 0.029s sys 0.027s
I20251027 21:08:15.953630 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82366f8bf4154ffd87ef45e4a4d5f54b" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:15.954061 9859 leader_election.cc:343] T 82366f8bf4154ffd87ef45e4a4d5f54b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.954116 10031 raft_consensus.cc:359] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.954288 10031 raft_consensus.cc:385] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.954341 10031 raft_consensus.cc:740] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.954546 10031 consensus_queue.cc:260] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.956194 10031 ts_tablet_manager.cc:1434] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251027 21:08:15.956496 10031 tablet_bootstrap.cc:492] T 334c580862bd4bf09be3f1710aa6a092 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:15.957970 10031 tablet_bootstrap.cc:654] T 334c580862bd4bf09be3f1710aa6a092 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.958619 10041 ts_tablet_manager.cc:1929] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:15.958726 10041 log.cc:1199] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/e1385beb1dc14ee0b42a2f4e1f66f532
I20251027 21:08:15.959023 10041 ts_tablet_manager.cc:1950] T e1385beb1dc14ee0b42a2f4e1f66f532 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:15.959781 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1338) took 1339 ms. Trace:
I20251027 21:08:15.959863 10041 rpcz_store.cc:276] 1027 21:08:14.619885 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:14.619973 (+ 88us) service_pool.cc:225] Handling call
1027 21:08:15.932062 (+1312089us) ts_tablet_manager.cc:1151] Deleting tablet e1385beb1dc14ee0b42a2f4e1f66f532
1027 21:08:15.932072 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:15.943122 (+ 11050us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:15.958528 (+ 15406us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:15.959762 (+ 1234us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1311959}
I20251027 21:08:15.960862 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet e1385beb1dc14ee0b42a2f4e1f66f532 (table default.Table_92242f48713b48e5969d30e05bed6686 [id=3867dc60b7dd4012b55fc6e187d76b3d]) successfully deleted
I20251027 21:08:15.961054 10041 tablet_replica.cc:333] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:15.961225 10041 raft_consensus.cc:2243] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.961364 10041 raft_consensus.cc:2272] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.961804 10041 ts_tablet_manager.cc:1916] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.967206 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_deb3d240fb84407e810d7077b5bda502" } modify_external_catalogs: true
I20251027 21:08:15.967396 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_deb3d240fb84407e810d7077b5bda502" } modify_external_catalogs: true
I20251027 21:08:15.971016 10031 tablet_bootstrap.cc:492] T 334c580862bd4bf09be3f1710aa6a092 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:15.971138 10031 ts_tablet_manager.cc:1403] T 334c580862bd4bf09be3f1710aa6a092 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.001s
I20251027 21:08:15.919446 10029 tablet_bootstrap.cc:492] T e1385beb1dc14ee0b42a2f4e1f66f532 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:15.922704 11373 consensus_queue.cc:1048] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:15.950639 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2203ee62f7e404c95c5f86b5ae85feb" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:15.953073 9566 consensus_peers.cc:597] T 334c580862bd4bf09be3f1710aa6a092 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:15.971366 10029 ts_tablet_manager.cc:1403] T e1385beb1dc14ee0b42a2f4e1f66f532 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.118s user 0.003s sys 0.000s
I20251027 21:08:15.971413 9630 raft_consensus.cc:2468] T d2203ee62f7e404c95c5f86b5ae85feb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:08:15.953373 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82366f8bf4154ffd87ef45e4a4d5f54b" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:15.971586 11373 raft_consensus.cc:493] T cc4635c00e9b40c49ce04debc76a04bf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:15.971701 11373 raft_consensus.cc:515] T cc4635c00e9b40c49ce04debc76a04bf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:15.972167 9860 leader_election.cc:343] T 82366f8bf4154ffd87ef45e4a4d5f54b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.972237 11373 leader_election.cc:290] T cc4635c00e9b40c49ce04debc76a04bf P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:15.972199 10029 raft_consensus.cc:359] T e1385beb1dc14ee0b42a2f4e1f66f532 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.972275 9860 leader_election.cc:304] T 82366f8bf4154ffd87ef45e4a4d5f54b P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.972330 10029 raft_consensus.cc:385] T e1385beb1dc14ee0b42a2f4e1f66f532 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.972378 10029 raft_consensus.cc:740] T e1385beb1dc14ee0b42a2f4e1f66f532 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.972555 10029 consensus_queue.cc:260] T e1385beb1dc14ee0b42a2f4e1f66f532 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.972651 11543 raft_consensus.cc:2749] T 82366f8bf4154ffd87ef45e4a4d5f54b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.973045 10029 ts_tablet_manager.cc:1434] T e1385beb1dc14ee0b42a2f4e1f66f532 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:15.972976 10031 raft_consensus.cc:359] T 334c580862bd4bf09be3f1710aa6a092 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.973119 10031 raft_consensus.cc:385] T 334c580862bd4bf09be3f1710aa6a092 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.973165 10031 raft_consensus.cc:740] T 334c580862bd4bf09be3f1710aa6a092 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.973285 10029 tablet_bootstrap.cc:492] T 2752b86151a6428c88bfaac1ff72cfb2 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:15.973325 10031 consensus_queue.cc:260] T 334c580862bd4bf09be3f1710aa6a092 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:15.974223 9568 leader_election.cc:343] T cc4635c00e9b40c49ce04debc76a04bf P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: cc4635c00e9b40c49ce04debc76a04bf
I20251027 21:08:15.974661 10029 tablet_bootstrap.cc:654] T 2752b86151a6428c88bfaac1ff72cfb2 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.974660 9710 leader_election.cc:304] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:15.974974 11724 raft_consensus.cc:2804] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:15.975041 11724 raft_consensus.cc:493] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:15.975103 11724 raft_consensus.cc:3060] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.978433 10041 ts_tablet_manager.cc:1929] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:15.978559 10041 log.cc:1199] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/c548d2413a5940c8b114ced667b4dc41
I20251027 21:08:15.979628 10031 ts_tablet_manager.cc:1434] T 334c580862bd4bf09be3f1710aa6a092 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251027 21:08:15.980599 11724 raft_consensus.cc:515] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:15.980830 10032 tablet_bootstrap.cc:654] T c97cba61e08d42a7bdb3f51db0a63ec8 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.980808 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7a535e8483da496498c4c7f7e76d6670" } modify_external_catalogs: true
I20251027 21:08:15.980971 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7a535e8483da496498c4c7f7e76d6670" } modify_external_catalogs: true
I20251027 21:08:15.981633 10041 ts_tablet_manager.cc:1950] T c548d2413a5940c8b114ced667b4dc41 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
W20251027 21:08:15.982041 9711 leader_election.cc:343] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: d2203ee62f7e404c95c5f86b5ae85feb
I20251027 21:08:15.982295 11724 leader_election.cc:290] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:15.982353 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1340) took 1362 ms. Trace:
I20251027 21:08:15.982430 10041 rpcz_store.cc:276] 1027 21:08:14.620046 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:14.620150 (+ 104us) service_pool.cc:225] Handling call
1027 21:08:15.960691 (+1340541us) ts_tablet_manager.cc:1151] Deleting tablet c548d2413a5940c8b114ced667b4dc41
1027 21:08:15.960701 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:15.964785 (+ 4084us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:15.978334 (+ 13549us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:15.982338 (+ 4004us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1340407}
I20251027 21:08:15.982802 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet c548d2413a5940c8b114ced667b4dc41 (table default.Table_907ada415b694cf4817ef1e6f6f68e9d [id=ec3ae7f22bea4f9d8b7ba9c5c4a3b2c6]) successfully deleted
W20251027 21:08:15.984128 9568 consensus_peers.cc:597] T 7b8156d9ab2e4dfba383ddc72137aab4 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b8156d9ab2e4dfba383ddc72137aab4. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:15.984467 10041 tablet_replica.cc:333] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:15.984970 10031 tablet_bootstrap.cc:492] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:15.985574 10041 raft_consensus.cc:2243] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:15.985721 10041 raft_consensus.cc:2272] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:15.986400 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2203ee62f7e404c95c5f86b5ae85feb" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:15.986552 9635 raft_consensus.cc:3060] T d2203ee62f7e404c95c5f86b5ae85feb P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:15.986716 10031 tablet_bootstrap.cc:654] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:15.986806 10041 ts_tablet_manager.cc:1916] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:15.986786 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc4635c00e9b40c49ce04debc76a04bf" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:15.989776 9635 raft_consensus.cc:2468] T d2203ee62f7e404c95c5f86b5ae85feb P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
W20251027 21:08:15.990154 9566 leader_election.cc:343] T cc4635c00e9b40c49ce04debc76a04bf P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:15.990265 9566 leader_election.cc:304] T cc4635c00e9b40c49ce04debc76a04bf P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:15.991302 11373 raft_consensus.cc:2749] T cc4635c00e9b40c49ce04debc76a04bf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:15.991907 10031 tablet_bootstrap.cc:492] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:15.992049 10031 ts_tablet_manager.cc:1403] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20251027 21:08:15.992750 10031 raft_consensus.cc:359] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.992900 10031 raft_consensus.cc:385] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:15.992960 10031 raft_consensus.cc:740] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:15.993160 10031 consensus_queue.cc:260] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:15.993247 10041 ts_tablet_manager.cc:1929] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:15.993341 10041 log.cc:1199] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/4d2c40f983e743b0b53f987b6a7ba437
I20251027 21:08:15.993644 10041 ts_tablet_manager.cc:1950] T 4d2c40f983e743b0b53f987b6a7ba437 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:15.998011 10031 ts_tablet_manager.cc:1434] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251027 21:08:16.002038 9710 leader_election.cc:304] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:16.002614 11724 raft_consensus.cc:2804] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:16.002874 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1334) took 1383 ms. Trace:
I20251027 21:08:16.003019 11373 raft_consensus.cc:493] T e477ca4b69c84327b04618d90406e19a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.003135 11373 raft_consensus.cc:515] T e477ca4b69c84327b04618d90406e19a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.003165 11724 raft_consensus.cc:697] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:16.003458 11724 consensus_queue.cc:237] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:16.003633 11373 leader_election.cc:290] T e477ca4b69c84327b04618d90406e19a P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:16.002943 10041 rpcz_store.cc:276] 1027 21:08:14.619478 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:14.620450 (+ 972us) service_pool.cc:225] Handling call
1027 21:08:15.982693 (+1362243us) ts_tablet_manager.cc:1151] Deleting tablet 4d2c40f983e743b0b53f987b6a7ba437
1027 21:08:15.984418 (+ 1725us) spinlock_profiling.cc:243] Waited 1.5 ms on lock 0x61800000d518. stack: 00007fc04d4329f6 00007fc04d4327b0 00007fc04c40e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fc04d41b2fd 00007fc04d41c36e 00007fc061057c9d 00007fc060fd6203 00007fc05d209646 00007fc05d20a9c9 00007fc05d23e2d4 00007fc05d21ed70 00007fc05d21eafd 0000000000a8867c
1027 21:08:15.984428 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:15.990269 (+ 5841us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:15.993174 (+ 2905us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:16.002854 (+ 9680us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3458304,"tablet-delete.queue_time_us":1362068}
I20251027 21:08:16.004390 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 4d2c40f983e743b0b53f987b6a7ba437 (table default.Table_73afc9d8e0d740d098eb77c4018c3cb2 [id=272264f0d3d347f0bd782edcebfec854]) successfully deleted
I20251027 21:08:16.004709 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e477ca4b69c84327b04618d90406e19a" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:16.005038 10031 tablet_bootstrap.cc:492] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:08:16.005045 9568 leader_election.cc:343] T e477ca4b69c84327b04618d90406e19a P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: e477ca4b69c84327b04618d90406e19a
I20251027 21:08:16.005103 10041 tablet_replica.cc:333] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:16.005254 10041 raft_consensus.cc:2243] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:16.005385 10041 raft_consensus.cc:2272] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:16.006000 9303 master_service.cc:954] ReplaceTablet: received request to replace tablet b57c38c7a97f45feb1b7f90e85577a40 from {username='slave'} at 127.0.0.1:41710
W20251027 21:08:16.006520 9566 leader_election.cc:343] T e477ca4b69c84327b04618d90406e19a P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:16.006657 9566 leader_election.cc:304] T e477ca4b69c84327b04618d90406e19a P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:16.007081 11373 raft_consensus.cc:2749] T e477ca4b69c84327b04618d90406e19a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:16.007491 10041 ts_tablet_manager.cc:1916] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:16.007503 9566 consensus_peers.cc:597] T 9be6b996df8145b9b0df1a2d2dc0ef03 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:16.007702 10031 tablet_bootstrap.cc:654] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:16.009418 10043 log.cc:1199] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/0e64d21abef342de86555e876f020787
I20251027 21:08:16.009791 10043 ts_tablet_manager.cc:1950] T 0e64d21abef342de86555e876f020787 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:16.010715 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1147) took 6507 ms. Trace:
I20251027 21:08:16.010816 10043 rpcz_store.cc:276] 1027 21:08:09.503067 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.504028 (+ 961us) service_pool.cc:225] Handling call
1027 21:08:15.846647 (+6342619us) ts_tablet_manager.cc:1151] Deleting tablet 0e64d21abef342de86555e876f020787
1027 21:08:15.846657 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:15.901153 (+ 54496us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:15.905536 (+ 4383us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:16.010692 (+105156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6332818}
I20251027 21:08:16.021265 10041 ts_tablet_manager.cc:1929] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:16.021677 10041 log.cc:1199] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/23673513189f47638dbbc737da0991d1
I20251027 21:08:16.021862 10032 tablet_bootstrap.cc:492] T c97cba61e08d42a7bdb3f51db0a63ec8 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:16.022024 10032 ts_tablet_manager.cc:1403] T c97cba61e08d42a7bdb3f51db0a63ec8 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.183s user 0.001s sys 0.009s
I20251027 21:08:16.022320 10041 ts_tablet_manager.cc:1950] T 23673513189f47638dbbc737da0991d1 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:16.022778 10032 raft_consensus.cc:359] T c97cba61e08d42a7bdb3f51db0a63ec8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.023029 10032 raft_consensus.cc:385] T c97cba61e08d42a7bdb3f51db0a63ec8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:16.023131 10032 raft_consensus.cc:740] T c97cba61e08d42a7bdb3f51db0a63ec8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:16.023447 10032 consensus_queue.cc:260] T c97cba61e08d42a7bdb3f51db0a63ec8 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.023154 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1342) took 1402 ms. Trace:
I20251027 21:08:16.023993 10041 rpcz_store.cc:276] 1027 21:08:14.620203 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:14.623739 (+ 3536us) service_pool.cc:225] Handling call
1027 21:08:16.005061 (+1381322us) ts_tablet_manager.cc:1151] Deleting tablet 23673513189f47638dbbc737da0991d1
1027 21:08:16.005071 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:16.013616 (+ 8545us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:16.021187 (+ 7571us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:16.023138 (+ 1951us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1380969}
I20251027 21:08:16.024785 10032 ts_tablet_manager.cc:1434] T c97cba61e08d42a7bdb3f51db0a63ec8 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:08:16.013029 11373 consensus_queue.cc:1048] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:16.025267 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1145) took 6541 ms. Trace:
I20251027 21:08:16.025349 10043 rpcz_store.cc:276] 1027 21:08:09.502774 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.505009 (+ 2235us) service_pool.cc:225] Handling call
1027 21:08:16.025141 (+6520132us) ts_tablet_manager.cc:1151] Deleting tablet 784df834432244e8812d1802f9e1a1bc
1027 21:08:16.025150 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:16.025251 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6511104}
I20251027 21:08:16.025635 10043 tablet_replica.cc:333] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:16.025836 10043 raft_consensus.cc:2243] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:16.025867 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 0e64d21abef342de86555e876f020787 (table default.Table_8ebde3403af34f4589ee6222700b2283 [id=cfdf2d0e9745403ea4bbeca6b4ed7439]) successfully deleted
W20251027 21:08:16.025924 9566 consensus_peers.cc:597] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:16.025969 10043 raft_consensus.cc:2272] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:16.026103 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 784df834432244e8812d1802f9e1a1bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 784df834432244e8812d1802f9e1a1bc already in progress: creating tablet
I20251027 21:08:16.026113 11747 tablet.cc:1722] T 371051fb3491426e8cf7797317baa3db P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:08:16.026891 11753 tablet.cc:1722] T 371051fb3491426e8cf7797317baa3db P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:08:16.030262 11543 raft_consensus.cc:493] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.030354 10031 tablet_bootstrap.cc:492] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:16.030385 11543 raft_consensus.cc:515] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.030478 10031 ts_tablet_manager.cc:1403] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.002s
I20251027 21:08:16.030865 11543 leader_election.cc:290] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:16.031792 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_b8502129426844198c5df5434968d674"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:16.032145 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93a31187eca14b8797737936a80e7292" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:16.032508 9310 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6acc073c9e6b45958ee3033e7f0695f3" } modify_external_catalogs: true
W20251027 21:08:16.032655 9859 leader_election.cc:343] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:16.032683 9310 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6acc073c9e6b45958ee3033e7f0695f3" } modify_external_catalogs: true
I20251027 21:08:16.034027 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93a31187eca14b8797737936a80e7292" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:16.034204 9635 raft_consensus.cc:2468] T 93a31187eca14b8797737936a80e7292 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 0.
I20251027 21:08:16.035080 9307 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_35f4f0dcffb34a5e9b565ebd86d78445" } modify_external_catalogs: true
I20251027 21:08:16.035218 9307 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_35f4f0dcffb34a5e9b565ebd86d78445" } modify_external_catalogs: true
I20251027 21:08:16.035343 9860 leader_election.cc:304] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:16.035563 11543 raft_consensus.cc:2804] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:16.035629 11543 raft_consensus.cc:493] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:16.035684 11543 raft_consensus.cc:3060] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:16.036988 10031 raft_consensus.cc:359] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.037161 10031 raft_consensus.cc:385] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:16.037215 10031 raft_consensus.cc:740] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:16.037417 10031 consensus_queue.cc:260] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.037990 10031 ts_tablet_manager.cc:1434] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251027 21:08:16.038276 10031 tablet_bootstrap.cc:492] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:16.039455 10032 tablet_bootstrap.cc:492] T 19f3e1a9db434c519e26e864eb790706 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:16.041265 10032 tablet_bootstrap.cc:654] T 19f3e1a9db434c519e26e864eb790706 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:16.041232 10029 tablet_bootstrap.cc:492] T 2752b86151a6428c88bfaac1ff72cfb2 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
W20251027 21:08:16.042263 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8502129426844198c5df5434968d674 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:16.043308 11543 raft_consensus.cc:515] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.044099 11543 leader_election.cc:290] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:16.044649 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93a31187eca14b8797737936a80e7292" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:16.044776 9635 raft_consensus.cc:3060] T 93a31187eca14b8797737936a80e7292 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:16.045125 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93a31187eca14b8797737936a80e7292" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:16.045544 9859 leader_election.cc:343] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:16.048254 10043 ts_tablet_manager.cc:1916] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:16.024561 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 23673513189f47638dbbc737da0991d1 (table default.Table_0a6d8856e582400da9228702189fc7a9 [id=cc1da530aa95455e90889dd5ea3768c4]) successfully deleted
I20251027 21:08:16.049998 9312 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_69ff6879149248aa9f620f4398ec1e3e" } new_table_name: "default.Table_ae820b20073c46e0b83047ee462c057d" modify_external_catalogs: true
I20251027 21:08:16.048398 9635 raft_consensus.cc:2468] T 93a31187eca14b8797737936a80e7292 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 1.
I20251027 21:08:16.050629 9860 leader_election.cc:304] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:16.051015 11543 raft_consensus.cc:2804] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:16.051195 11543 raft_consensus.cc:697] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Becoming Leader. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Running, Role: LEADER
I20251027 21:08:16.051409 11543 consensus_queue.cc:237] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:16.055435 9566 consensus_peers.cc:597] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:16.055500 10031 tablet_bootstrap.cc:654] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:16.055583 9909 raft_consensus.cc:1275] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Refusing update from remote peer 51902b6b61ff4d5584ff21896e847e14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 21:08:16.056150 11658 consensus_queue.cc:1048] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:16.059737 10032 tablet_bootstrap.cc:492] T 19f3e1a9db434c519e26e864eb790706 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:16.059883 10032 ts_tablet_manager.cc:1403] T 19f3e1a9db434c519e26e864eb790706 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20251027 21:08:16.060459 10032 raft_consensus.cc:359] T 19f3e1a9db434c519e26e864eb790706 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.060644 10032 raft_consensus.cc:385] T 19f3e1a9db434c519e26e864eb790706 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:16.060722 10032 raft_consensus.cc:740] T 19f3e1a9db434c519e26e864eb790706 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:16.060947 10032 consensus_queue.cc:260] T 19f3e1a9db434c519e26e864eb790706 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.061461 10032 ts_tablet_manager.cc:1434] T 19f3e1a9db434c519e26e864eb790706 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 21:08:16.061776 10032 tablet_bootstrap.cc:492] T 7828f466a5ad4d7997721b07f815f56d P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:16.062492 10031 tablet_bootstrap.cc:492] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:16.062623 10031 ts_tablet_manager.cc:1403] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
W20251027 21:08:16.063031 9312 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c3d07b891a248efb0860cd000e44432 consensus state has no leader
I20251027 21:08:16.063280 10031 raft_consensus.cc:359] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.063431 10031 raft_consensus.cc:385] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:16.063500 10031 raft_consensus.cc:740] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:16.063680 10031 consensus_queue.cc:260] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.063846 10032 tablet_bootstrap.cc:654] T 7828f466a5ad4d7997721b07f815f56d P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:16.063393 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_592c1afc91f1411a87d60f2a52e3482f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:16.064296 10031 ts_tablet_manager.cc:1434] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:16.064703 10031 tablet_bootstrap.cc:492] T 371051fb3491426e8cf7797317baa3db P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:16.066023 11747 tablet.cc:1722] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 51902b6b61ff4d5584ff21896e847e14: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20251027 21:08:16.066203 10031 tablet_bootstrap.cc:654] T 371051fb3491426e8cf7797317baa3db P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:16.071055 11753 tablet.cc:1722] T 93a3b22cdc1543aebcc7fe72f2f2c0ae P 0fbd567f929a4f6d8e4d78feef7ed678: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
W20251027 21:08:16.073037 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_592c1afc91f1411a87d60f2a52e3482f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:16.083582 10032 tablet_bootstrap.cc:492] T 7828f466a5ad4d7997721b07f815f56d P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:16.083776 10032 ts_tablet_manager.cc:1403] T 7828f466a5ad4d7997721b07f815f56d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20251027 21:08:16.085108 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f95fb451d6a342f081a72bdbfe6c6e9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:16.042011 10029 ts_tablet_manager.cc:1403] T 2752b86151a6428c88bfaac1ff72cfb2 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.069s user 0.014s sys 0.010s
I20251027 21:08:16.087047 10029 raft_consensus.cc:359] T 2752b86151a6428c88bfaac1ff72cfb2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.087327 10029 raft_consensus.cc:385] T 2752b86151a6428c88bfaac1ff72cfb2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:16.087404 10029 raft_consensus.cc:740] T 2752b86151a6428c88bfaac1ff72cfb2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:16.087664 10029 consensus_queue.cc:260] T 2752b86151a6428c88bfaac1ff72cfb2 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.089244 10029 ts_tablet_manager.cc:1434] T 2752b86151a6428c88bfaac1ff72cfb2 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 21:08:16.089186 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1341) took 1469 ms. Trace:
W20251027 21:08:16.090456 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f95fb451d6a342f081a72bdbfe6c6e9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:16.093869 10032 raft_consensus.cc:359] T 7828f466a5ad4d7997721b07f815f56d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.095295 10032 raft_consensus.cc:385] T 7828f466a5ad4d7997721b07f815f56d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:16.095662 10032 raft_consensus.cc:740] T 7828f466a5ad4d7997721b07f815f56d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:16.095947 10032 consensus_queue.cc:260] T 7828f466a5ad4d7997721b07f815f56d P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.097677 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_dcdcc2cc86764e80918034bdea343516"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:16.098415 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dcdcc2cc86764e80918034bdea343516 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:16.101089 10031 tablet_bootstrap.cc:492] T 371051fb3491426e8cf7797317baa3db P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:16.101226 10031 ts_tablet_manager.cc:1403] T 371051fb3491426e8cf7797317baa3db P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20251027 21:08:16.102013 10031 raft_consensus.cc:359] T 371051fb3491426e8cf7797317baa3db P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.102192 10031 raft_consensus.cc:385] T 371051fb3491426e8cf7797317baa3db P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:16.102248 10031 raft_consensus.cc:740] T 371051fb3491426e8cf7797317baa3db P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:16.102509 10031 consensus_queue.cc:260] T 371051fb3491426e8cf7797317baa3db P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.104730 10032 ts_tablet_manager.cc:1434] T 7828f466a5ad4d7997721b07f815f56d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20251027 21:08:16.105443 10031 ts_tablet_manager.cc:1434] T 371051fb3491426e8cf7797317baa3db P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20251027 21:08:16.106891 9566 consensus_peers.cc:597] T 67cb71ce43244d8ab9e71ead6bb315c9 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67cb71ce43244d8ab9e71ead6bb315c9. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:16.108093 10031 tablet_bootstrap.cc:492] T 0e64d21abef342de86555e876f020787 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:16.109723 10031 tablet_bootstrap.cc:654] T 0e64d21abef342de86555e876f020787 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:16.111241 11543 raft_consensus.cc:493] T 95f9595784df434dbb0dd29d9a05511e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.111426 11543 raft_consensus.cc:515] T 95f9595784df434dbb0dd29d9a05511e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.112057 11543 leader_election.cc:290] T 95f9595784df434dbb0dd29d9a05511e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:16.112571 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95f9595784df434dbb0dd29d9a05511e" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:16.112993 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95f9595784df434dbb0dd29d9a05511e" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:16.113675 9859 leader_election.cc:343] T 95f9595784df434dbb0dd29d9a05511e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:16.113912 9860 leader_election.cc:343] T 95f9595784df434dbb0dd29d9a05511e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:16.113999 9860 leader_election.cc:304] T 95f9595784df434dbb0dd29d9a05511e P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:16.114313 10032 tablet_bootstrap.cc:492] T efbabe357ae947e09b1902b1fbc0ee40 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:16.114393 9305 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet dd9e6656d8df43afbb7f70706bc14585 (table default.Table_f78d839900634631b6963b2d63f0d084 [id=3cdf432ec43d42e8bc2d253a92eac8d3]). Expected version 3 got 0
I20251027 21:08:16.114456 11543 raft_consensus.cc:2749] T 95f9595784df434dbb0dd29d9a05511e P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:16.114701 9305 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 76bee322d0b94abe9166170733608853 (table default.Table_4b0bc60a7c5e4888b79210296b7dfa20 [id=c9e7aa435d5b48e096c2b67b0e678eeb]). Expected version 1 got 0
I20251027 21:08:16.115032 9305 catalog_manager.cc:5649] T a9558d53008640739ac6fb45f1352fc0 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:16.115372 9305 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet a9558d53008640739ac6fb45f1352fc0 (table default.Table_9821265dde764aee9571790dd10b4910 [id=c575fb3af44c46db8c198569241a425f]). Expected version 2 got 0
I20251027 21:08:16.115810 10032 tablet_bootstrap.cc:654] T efbabe357ae947e09b1902b1fbc0ee40 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:16.115624 9305 catalog_manager.cc:5649] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:16.115937 9305 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 59300e16b3e148a08a217fc24d1e572f (table default.Table_37d44f2ffd8b4816b4bff0b1960cd275 [id=a1dea3b74aa94995991d3123fbb2bf8d]). Expected version 1 got 0
I20251027 21:08:16.116160 9305 catalog_manager.cc:5649] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:16.116475 9305 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 57ef07c764664deaa4b2e9bcae7e7973 (table default.Table_7d6b8d0ba5ad42fa9fdbcf13c7647d41 [id=a541c8d30a9540f5a7d4dd98a5d3739a]). Expected version 1 got 0
I20251027 21:08:16.121070 9887 tablet_service.cc:1505] Processing CreateTablet for tablet ac640677d2ca4c568450468b7a8bf8f9 (DEFAULT_TABLE table=default.Table_a656c74e16c24bde86338f71c0836028 [id=7b00cf7d0e6642ff94245963ef7c9d04]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.121088 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 1a9663a7a4274af19661132cea3f383e (DEFAULT_TABLE table=default.Table_6f6da8160dd64d67b34d80b00ab4029b [id=3e00e9e1e221475d9fb2d9db6d345bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.121394 9887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac640677d2ca4c568450468b7a8bf8f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.121569 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a9663a7a4274af19661132cea3f383e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.121812 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 63c869d4fe0b4cc5ade446347e2ddf0f (DEFAULT_TABLE table=default.Table_0f0e665cd21f443b9c1e7f3fa32c66da [id=84f5e6ed829240e6afde4cd67d0fc69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.122083 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63c869d4fe0b4cc5ade446347e2ddf0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.125530 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 373d05c0268d454ebd02740792835cd7 (DEFAULT_TABLE table=default.Table_e6ea09cd17ac421eb0f6a9411d2532e3 [id=81063e5015954aa29b0733ad3e03aeaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.125869 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 373d05c0268d454ebd02740792835cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.130405 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 9d01980e56b54878aa8a1ea48ab0ef59 (DEFAULT_TABLE table=default.Table_9c71cb1852864cb1ae2e8e650536ab77 [id=f00803c94bd045e98941da64efc0cb8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.130697 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d01980e56b54878aa8a1ea48ab0ef59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.132642 11657 raft_consensus.cc:493] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.132761 11657 raft_consensus.cc:515] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.133291 11657 leader_election.cc:290] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:16.133780 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0c53b7b32c84a51a967e2a6a22258c6" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:16.134218 9566 leader_election.cc:343] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:16.134343 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 66d6602e6b28453d83e74ee7480721fd (DEFAULT_TABLE table=default.Table_501dbc8bbb924002872a8eb8d1d7b517 [id=d94de78740134991b3ae8db235333f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.134629 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66d6602e6b28453d83e74ee7480721fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.138594 10032 tablet_bootstrap.cc:492] T efbabe357ae947e09b1902b1fbc0ee40 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:16.138703 10032 ts_tablet_manager.cc:1403] T efbabe357ae947e09b1902b1fbc0ee40 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.001s
I20251027 21:08:16.139273 10032 raft_consensus.cc:359] T efbabe357ae947e09b1902b1fbc0ee40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.139418 10032 raft_consensus.cc:385] T efbabe357ae947e09b1902b1fbc0ee40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:16.139463 10032 raft_consensus.cc:740] T efbabe357ae947e09b1902b1fbc0ee40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:16.139652 10032 consensus_queue.cc:260] T efbabe357ae947e09b1902b1fbc0ee40 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.143754 9752 tablet_service.cc:1505] Processing CreateTablet for tablet ac640677d2ca4c568450468b7a8bf8f9 (DEFAULT_TABLE table=default.Table_a656c74e16c24bde86338f71c0836028 [id=7b00cf7d0e6642ff94245963ef7c9d04]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.144116 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac640677d2ca4c568450468b7a8bf8f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.144205 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 66d6602e6b28453d83e74ee7480721fd (DEFAULT_TABLE table=default.Table_501dbc8bbb924002872a8eb8d1d7b517 [id=d94de78740134991b3ae8db235333f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.144515 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66d6602e6b28453d83e74ee7480721fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.146013 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 373d05c0268d454ebd02740792835cd7 (DEFAULT_TABLE table=default.Table_e6ea09cd17ac421eb0f6a9411d2532e3 [id=81063e5015954aa29b0733ad3e03aeaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.146389 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 373d05c0268d454ebd02740792835cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.146358 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 63c869d4fe0b4cc5ade446347e2ddf0f (DEFAULT_TABLE table=default.Table_0f0e665cd21f443b9c1e7f3fa32c66da [id=84f5e6ed829240e6afde4cd67d0fc69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.146373 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 9d01980e56b54878aa8a1ea48ab0ef59 (DEFAULT_TABLE table=default.Table_9c71cb1852864cb1ae2e8e650536ab77 [id=f00803c94bd045e98941da64efc0cb8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.146474 9603 tablet_service.cc:1505] Processing CreateTablet for tablet 66d6602e6b28453d83e74ee7480721fd (DEFAULT_TABLE table=default.Table_501dbc8bbb924002872a8eb8d1d7b517 [id=d94de78740134991b3ae8db235333f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.146667 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63c869d4fe0b4cc5ade446347e2ddf0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.146698 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d01980e56b54878aa8a1ea48ab0ef59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.147253 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66d6602e6b28453d83e74ee7480721fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.148392 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 9d01980e56b54878aa8a1ea48ab0ef59 (DEFAULT_TABLE table=default.Table_9c71cb1852864cb1ae2e8e650536ab77 [id=f00803c94bd045e98941da64efc0cb8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.148691 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d01980e56b54878aa8a1ea48ab0ef59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.149124 9602 tablet_service.cc:1505] Processing CreateTablet for tablet ac640677d2ca4c568450468b7a8bf8f9 (DEFAULT_TABLE table=default.Table_a656c74e16c24bde86338f71c0836028 [id=7b00cf7d0e6642ff94245963ef7c9d04]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.149447 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac640677d2ca4c568450468b7a8bf8f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.150394 9615 tablet_service.cc:1505] Processing CreateTablet for tablet eebc9b690a714b58a657bbf5e7cdc2b4 (DEFAULT_TABLE table=default.Table_5d6d514c34fb46a6826cf5e730cddd8c [id=52e57fb685344e5cbe616fab5e600ab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.150732 9615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eebc9b690a714b58a657bbf5e7cdc2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.153189 10032 ts_tablet_manager.cc:1434] T efbabe357ae947e09b1902b1fbc0ee40 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20251027 21:08:16.153486 10032 tablet_bootstrap.cc:492] T d1f24d21bb954ac9b62ce458c6a22462 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:16.153429 9755 tablet_service.cc:1505] Processing CreateTablet for tablet 373d05c0268d454ebd02740792835cd7 (DEFAULT_TABLE table=default.Table_e6ea09cd17ac421eb0f6a9411d2532e3 [id=81063e5015954aa29b0733ad3e03aeaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.153825 9613 tablet_service.cc:1505] Processing CreateTablet for tablet 63c869d4fe0b4cc5ade446347e2ddf0f (DEFAULT_TABLE table=default.Table_0f0e665cd21f443b9c1e7f3fa32c66da [id=84f5e6ed829240e6afde4cd67d0fc69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.154129 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63c869d4fe0b4cc5ade446347e2ddf0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.154387 9598 tablet_service.cc:1505] Processing CreateTablet for tablet 8300816cebb84d1090221c4ebbf883f9 (DEFAULT_TABLE table=default.Table_f63a59e9a2bb49b29a8477d90871afde [id=4d029edd43d94372b1864c889b3e0052]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.154680 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8300816cebb84d1090221c4ebbf883f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.155324 11626 raft_consensus.cc:493] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.155434 11626 raft_consensus.cc:515] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.156459 10032 tablet_bootstrap.cc:654] T d1f24d21bb954ac9b62ce458c6a22462 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:16.158409 9604 tablet_service.cc:1505] Processing CreateTablet for tablet 1a9663a7a4274af19661132cea3f383e (DEFAULT_TABLE table=default.Table_6f6da8160dd64d67b34d80b00ab4029b [id=3e00e9e1e221475d9fb2d9db6d345bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.158699 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a9663a7a4274af19661132cea3f383e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.161995 11724 raft_consensus.cc:493] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.162115 11724 raft_consensus.cc:515] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.163045 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 8300816cebb84d1090221c4ebbf883f9 (DEFAULT_TABLE table=default.Table_f63a59e9a2bb49b29a8477d90871afde [id=4d029edd43d94372b1864c889b3e0052]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.163161 9755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 373d05c0268d454ebd02740792835cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.163271 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 7767c0ca2c8e4fe6874aa766aa28dacf (DEFAULT_TABLE table=default.Table_3c8bffe5fd42431fa97808af0d8c634d [id=78a2f2a1e52740a2b041fe391729f114]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.163370 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8300816cebb84d1090221c4ebbf883f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.163585 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7767c0ca2c8e4fe6874aa766aa28dacf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.163616 11543 raft_consensus.cc:493] T 4bfa70ded2f54e3fb2ca744c723395e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.163746 11543 raft_consensus.cc:515] T 4bfa70ded2f54e3fb2ca744c723395e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.163793 9890 tablet_service.cc:1505] Processing CreateTablet for tablet d2d7b7545ce74581a1548e404e004f6f (DEFAULT_TABLE table=default.Table_c572071929f9429f9d8f7bfa30fbf964 [id=3ff1ab5cbf8b45d4afee1d8c6f7aa404]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.164021 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 3dc44d6b0d5549fcb01308f0eaa53bd2 (DEFAULT_TABLE table=default.Table_7cc32a1529164529b4a58e2ea9df9058 [id=b47198eb136246538df2d9b4c0fb720f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.164098 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2d7b7545ce74581a1548e404e004f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.164330 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dc44d6b0d5549fcb01308f0eaa53bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.164528 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 0a5a95f40a56495f85468817ce3c7a9e (DEFAULT_TABLE table=default.Table_9a0390f5f62d4b9da2fbc3425e8781f6 [id=af4eab915d994bb18ebba939fbe4e6ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.164819 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a5a95f40a56495f85468817ce3c7a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.166430 9903 tablet_service.cc:1505] Processing CreateTablet for tablet a109a8b9225f45cda6cbea7df02b8254 (DEFAULT_TABLE table=default.Table_dc77fcc2c86242789d1775592ee6a417 [id=54f1daf3c69047f584a881bf94c0a32d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.166757 9894 tablet_service.cc:1505] Processing CreateTablet for tablet 8a3985daaf24490d972e7e65f85d87d4 (DEFAULT_TABLE table=default.Table_be9e649e9750412b8350f0af57294c85 [id=5e67a606c874479580d878a811bb1f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.167251 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 6c0fc37a1cb34302975ce24c347354a9 (DEFAULT_TABLE table=default.Table_1f3761445da945bbb906e156fe5be872 [id=6c6be282a0774389ab577563d2c86233]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.166483 9898 tablet_service.cc:1505] Processing CreateTablet for tablet a1b528a8761e4df98a1b48ccbba1ad6d (DEFAULT_TABLE table=default.Table_2db01113aba5414ba6dad68dc4f2bb56 [id=aedf3b3e233248739dc51824076b5ebe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.166777 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a109a8b9225f45cda6cbea7df02b8254. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.170274 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1b528a8761e4df98a1b48ccbba1ad6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.171088 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bfa70ded2f54e3fb2ca744c723395e2" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:16.164330 11543 leader_election.cc:290] T 4bfa70ded2f54e3fb2ca744c723395e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:16.173043 9860 leader_election.cc:343] T 4bfa70ded2f54e3fb2ca744c723395e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:16.174054 9605 tablet_service.cc:1505] Processing CreateTablet for tablet a109a8b9225f45cda6cbea7df02b8254 (DEFAULT_TABLE table=default.Table_dc77fcc2c86242789d1775592ee6a417 [id=54f1daf3c69047f584a881bf94c0a32d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.174079 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 7767c0ca2c8e4fe6874aa766aa28dacf (DEFAULT_TABLE table=default.Table_3c8bffe5fd42431fa97808af0d8c634d [id=78a2f2a1e52740a2b041fe391729f114]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.174422 9610 tablet_service.cc:1505] Processing CreateTablet for tablet a1b528a8761e4df98a1b48ccbba1ad6d (DEFAULT_TABLE table=default.Table_2db01113aba5414ba6dad68dc4f2bb56 [id=aedf3b3e233248739dc51824076b5ebe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.174582 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a3985daaf24490d972e7e65f85d87d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.175647 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 28d5507913044975b7fa6e34ba52a1b6 (DEFAULT_TABLE table=default.Table_f639bf82a7f6412ea1aecca4f90193e1 [id=fc0f10f083b644229fd3cdb435cfc951]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.175863 9895 tablet_service.cc:1505] Processing CreateTablet for tablet c0e96c3547a34448b3d6e374825657a4 (DEFAULT_TABLE table=default.Table_c6a9ddd35e0549cc81fe330827797023 [id=f08e140ca25b455794cdb08cf5bd7cec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.176000 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28d5507913044975b7fa6e34ba52a1b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.176172 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0e96c3547a34448b3d6e374825657a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.177114 10043 ts_tablet_manager.cc:1929] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:16.177203 10043 log.cc:1199] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/952583fd7fde4b4baccdb98c01bdf9e1
I20251027 21:08:16.178793 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c0fc37a1cb34302975ce24c347354a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.181937 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 0f7569995c8e4974b5ee3a8b0ec3fc8b (DEFAULT_TABLE table=default.Table_bba03739a01b4ed09f57e8b629ae3690 [id=9852e2eef5794d38b50b0f181f3ba161]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.174427 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a109a8b9225f45cda6cbea7df02b8254. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.182269 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f7569995c8e4974b5ee3a8b0ec3fc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.186265 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1b528a8761e4df98a1b48ccbba1ad6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.186965 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7767c0ca2c8e4fe6874aa766aa28dacf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.188786 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 481199bc97514ca88e19603b3bf38dc7 (DEFAULT_TABLE table=default.Table_f1f34db7d6324d4c8cb9f5a8922e27b2 [id=205f1ddf7e84422e890f08e6ba8128da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.189131 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 481199bc97514ca88e19603b3bf38dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.190430 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 5bf2904d365243c6a261ef620203499f (DEFAULT_TABLE table=default.Table_d94bcf0455894a20baa4a30010869f62 [id=61bb9aa1a0f541029ded0842a69d4da3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.190732 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bf2904d365243c6a261ef620203499f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:16.192265 9566 consensus_peers.cc:597] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:16.194797 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bfa70ded2f54e3fb2ca744c723395e2" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:16.198868 9925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0c53b7b32c84a51a967e2a6a22258c6" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:16.198839 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 6443f2950db143bda9f098453ec4de25 (DEFAULT_TABLE table=default.Table_6f18157ff71c4f37b28566ed2e696c74 [id=cf0b25c72b984df1bc0e139bfdb3f0b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.199086 9925 raft_consensus.cc:2468] T d0c53b7b32c84a51a967e2a6a22258c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:16.199227 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6443f2950db143bda9f098453ec4de25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.199433 9305 catalog_manager.cc:5649] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:16.199834 9305 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 1a48067b40324c79b1ee4eb431ce4a46 (table default.Table_42823df1b61240fa9dc0e59703660dd5 [id=00404f9f36a34a2e8e1244799468d85e]). Expected version 1 got 0
I20251027 21:08:16.200075 9305 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 93a3b22cdc1543aebcc7fe72f2f2c0ae (table default.Table_c5a552fcdf214416a4b9217758c32b80 [id=8646c99b90414f4fa266cd7a31123d8e]). Expected version 6 got 0
I20251027 21:08:16.200260 9309 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_3c5157c8728d40dfb6ffe4ecb2ab80d3 [id=cdb023218cda4e9ea04a226b37a220d9] alter complete (version 3)
I20251027 21:08:16.204246 9603 tablet_service.cc:1505] Processing CreateTablet for tablet 0a5a95f40a56495f85468817ce3c7a9e (DEFAULT_TABLE table=default.Table_9a0390f5f62d4b9da2fbc3425e8781f6 [id=af4eab915d994bb18ebba939fbe4e6ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.204607 9603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a5a95f40a56495f85468817ce3c7a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.205395 9599 tablet_service.cc:1505] Processing CreateTablet for tablet 6c0fc37a1cb34302975ce24c347354a9 (DEFAULT_TABLE table=default.Table_1f3761445da945bbb906e156fe5be872 [id=6c6be282a0774389ab577563d2c86233]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.205736 9599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c0fc37a1cb34302975ce24c347354a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.206390 9604 tablet_service.cc:1505] Processing CreateTablet for tablet f1cde7c514ab408e9e5bc6da891ba493 (DEFAULT_TABLE table=default.Table_b9dc7d4bed0d40c29eab4ab96fd00d20 [id=e1e982e8f8c243bd98c70bc51fa5a1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.206691 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1cde7c514ab408e9e5bc6da891ba493. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.208108 9595 tablet_service.cc:1505] Processing CreateTablet for tablet c0e96c3547a34448b3d6e374825657a4 (DEFAULT_TABLE table=default.Table_c6a9ddd35e0549cc81fe330827797023 [id=f08e140ca25b455794cdb08cf5bd7cec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.208179 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 28d5507913044975b7fa6e34ba52a1b6 (DEFAULT_TABLE table=default.Table_f639bf82a7f6412ea1aecca4f90193e1 [id=fc0f10f083b644229fd3cdb435cfc951]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.208300 9608 tablet_service.cc:1505] Processing CreateTablet for tablet 8a3985daaf24490d972e7e65f85d87d4 (DEFAULT_TABLE table=default.Table_be9e649e9750412b8350f0af57294c85 [id=5e67a606c874479580d878a811bb1f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.208397 9595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0e96c3547a34448b3d6e374825657a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.208606 9594 tablet_service.cc:1505] Processing CreateTablet for tablet 481199bc97514ca88e19603b3bf38dc7 (DEFAULT_TABLE table=default.Table_f1f34db7d6324d4c8cb9f5a8922e27b2 [id=205f1ddf7e84422e890f08e6ba8128da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.208832 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 0f7569995c8e4974b5ee3a8b0ec3fc8b (DEFAULT_TABLE table=default.Table_bba03739a01b4ed09f57e8b629ae3690 [id=9852e2eef5794d38b50b0f181f3ba161]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.208606 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 3dc44d6b0d5549fcb01308f0eaa53bd2 (DEFAULT_TABLE table=default.Table_7cc32a1529164529b4a58e2ea9df9058 [id=b47198eb136246538df2d9b4c0fb720f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:16.209066 9568 consensus_peers.cc:597] T aaa8910d9fdb4146945621ec4ebf969e P 51902b6b61ff4d5584ff21896e847e14 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aaa8910d9fdb4146945621ec4ebf969e. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:16.213497 11626 leader_election.cc:290] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:16.213640 11626 raft_consensus.cc:493] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.213763 11626 raft_consensus.cc:515] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.215862 11724 leader_election.cc:290] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:16.216151 9607 tablet_service.cc:1505] Processing CreateTablet for tablet d2d7b7545ce74581a1548e404e004f6f (DEFAULT_TABLE table=default.Table_c572071929f9429f9d8f7bfa30fbf964 [id=3ff1ab5cbf8b45d4afee1d8c6f7aa404]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.218009 9890 tablet_service.cc:1505] Processing CreateTablet for tablet ab12fe6c9ad44f80ab610a0f03b611ec (DEFAULT_TABLE table=default.Table_6b527fc023dd45f4ab080eddeeb8bb76 [id=5888beb9284242a8bf5a7c8f2649d62c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.218593 9568 leader_election.cc:304] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters: 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:16.218632 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27bd3b2a011648b2b8f2f92944513751" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:16.218786 9630 raft_consensus.cc:2468] T 27bd3b2a011648b2b8f2f92944513751 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:08:16.219026 9902 tablet_service.cc:1505] Processing CreateTablet for tablet b339d71e9cd948598b369ef5afa2d835 (DEFAULT_TABLE table=default.Table_55c1322435df4656993fef22102a4f62 [id=f335766496e14db4ac3d442956f509dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.219158 11724 raft_consensus.cc:493] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.219267 11724 raft_consensus.cc:515] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.219240 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab12fe6c9ad44f80ab610a0f03b611ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.219772 9710 leader_election.cc:304] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:16.220404 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 1a9663a7a4274af19661132cea3f383e (DEFAULT_TABLE table=default.Table_6f6da8160dd64d67b34d80b00ab4029b [id=3e00e9e1e221475d9fb2d9db6d345bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.220742 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a9663a7a4274af19661132cea3f383e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.223867 9740 tablet_service.cc:1505] Processing CreateTablet for tablet eebc9b690a714b58a657bbf5e7cdc2b4 (DEFAULT_TABLE table=default.Table_5d6d514c34fb46a6826cf5e730cddd8c [id=52e57fb685344e5cbe616fab5e600ab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.224175 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eebc9b690a714b58a657bbf5e7cdc2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.224154 9891 tablet_service.cc:1505] Processing CreateTablet for tablet eebc9b690a714b58a657bbf5e7cdc2b4 (DEFAULT_TABLE table=default.Table_5d6d514c34fb46a6826cf5e730cddd8c [id=52e57fb685344e5cbe616fab5e600ab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.224527 10043 ts_tablet_manager.cc:1950] T 952583fd7fde4b4baccdb98c01bdf9e1 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:16.224702 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eebc9b690a714b58a657bbf5e7cdc2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.224947 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b339d71e9cd948598b369ef5afa2d835. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.225327 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1149) took 6664 ms. Trace:
I20251027 21:08:16.225411 10043 rpcz_store.cc:276] 1027 21:08:09.560919 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.561351 (+ 432us) service_pool.cc:225] Handling call
1027 21:08:16.025592 (+6464241us) ts_tablet_manager.cc:1151] Deleting tablet 952583fd7fde4b4baccdb98c01bdf9e1
1027 21:08:16.025601 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:16.060589 (+ 34988us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:16.177035 (+116446us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:16.225305 (+ 48270us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6464058}
I20251027 21:08:16.225806 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1146) took 6722 ms. Trace:
I20251027 21:08:16.225881 10043 rpcz_store.cc:276] 1027 21:08:09.502934 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.594531 (+ 91597us) service_pool.cc:225] Handling call
1027 21:08:16.225669 (+6631138us) ts_tablet_manager.cc:1151] Deleting tablet ece28e82383540bea3457b1cbc65558b
1027 21:08:16.225679 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:16.225792 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6630751}
I20251027 21:08:16.226112 10043 tablet_replica.cc:333] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:16.226322 10043 raft_consensus.cc:2243] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:16.226649 10043 raft_consensus.cc:2272] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:16.228050 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28d5507913044975b7fa6e34ba52a1b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.228211 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a3985daaf24490d972e7e65f85d87d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.228410 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f7569995c8e4974b5ee3a8b0ec3fc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.228333 9612 tablet_service.cc:1505] Processing CreateTablet for tablet 6443f2950db143bda9f098453ec4de25 (DEFAULT_TABLE table=default.Table_6f18157ff71c4f37b28566ed2e696c74 [id=cf0b25c72b984df1bc0e139bfdb3f0b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.228724 9612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6443f2950db143bda9f098453ec4de25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.228225 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 5bf2904d365243c6a261ef620203499f (DEFAULT_TABLE table=default.Table_d94bcf0455894a20baa4a30010869f62 [id=61bb9aa1a0f541029ded0842a69d4da3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.229319 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bf2904d365243c6a261ef620203499f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.229351 11626 leader_election.cc:290] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:16.229482 11626 raft_consensus.cc:2804] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:16.229548 11626 raft_consensus.cc:493] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:16.229610 11626 raft_consensus.cc:3060] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:16.229616 9594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 481199bc97514ca88e19603b3bf38dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.230368 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 0c88e7aecf3f4e28987158f725e093da
I20251027 21:08:16.230825 11724 leader_election.cc:290] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:16.231711 9737 tablet_service.cc:1505] Processing CreateTablet for tablet 8300816cebb84d1090221c4ebbf883f9 (DEFAULT_TABLE table=default.Table_f63a59e9a2bb49b29a8477d90871afde [id=4d029edd43d94372b1864c889b3e0052]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.232024 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8300816cebb84d1090221c4ebbf883f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.232153 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 0f7569995c8e4974b5ee3a8b0ec3fc8b
I20251027 21:08:16.233529 9748 tablet_service.cc:1505] Processing CreateTablet for tablet d2d7b7545ce74581a1548e404e004f6f (DEFAULT_TABLE table=default.Table_c572071929f9429f9d8f7bfa30fbf964 [id=3ff1ab5cbf8b45d4afee1d8c6f7aa404]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.233879 9748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2d7b7545ce74581a1548e404e004f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.234175 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 7767c0ca2c8e4fe6874aa766aa28dacf (DEFAULT_TABLE table=default.Table_3c8bffe5fd42431fa97808af0d8c634d [id=78a2f2a1e52740a2b041fe391729f114]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.234521 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7767c0ca2c8e4fe6874aa766aa28dacf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.235036 9741 tablet_service.cc:1505] Processing CreateTablet for tablet 0a5a95f40a56495f85468817ce3c7a9e (DEFAULT_TABLE table=default.Table_9a0390f5f62d4b9da2fbc3425e8781f6 [id=af4eab915d994bb18ebba939fbe4e6ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.235473 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 6c0fc37a1cb34302975ce24c347354a9 (DEFAULT_TABLE table=default.Table_1f3761445da945bbb906e156fe5be872 [id=6c6be282a0774389ab577563d2c86233]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.235922 9749 tablet_service.cc:1505] Processing CreateTablet for tablet a109a8b9225f45cda6cbea7df02b8254 (DEFAULT_TABLE table=default.Table_dc77fcc2c86242789d1775592ee6a417 [id=54f1daf3c69047f584a881bf94c0a32d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.236339 9751 tablet_service.cc:1505] Processing CreateTablet for tablet 8a3985daaf24490d972e7e65f85d87d4 (DEFAULT_TABLE table=default.Table_be9e649e9750412b8350f0af57294c85 [id=5e67a606c874479580d878a811bb1f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.236764 9744 tablet_service.cc:1505] Processing CreateTablet for tablet 28d5507913044975b7fa6e34ba52a1b6 (DEFAULT_TABLE table=default.Table_f639bf82a7f6412ea1aecca4f90193e1 [id=fc0f10f083b644229fd3cdb435cfc951]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.237191 9747 tablet_service.cc:1505] Processing CreateTablet for tablet c0e96c3547a34448b3d6e374825657a4 (DEFAULT_TABLE table=default.Table_c6a9ddd35e0549cc81fe330827797023 [id=f08e140ca25b455794cdb08cf5bd7cec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.237644 9739 tablet_service.cc:1505] Processing CreateTablet for tablet 0f7569995c8e4974b5ee3a8b0ec3fc8b (DEFAULT_TABLE table=default.Table_bba03739a01b4ed09f57e8b629ae3690 [id=9852e2eef5794d38b50b0f181f3ba161]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.237994 9750 tablet_service.cc:1505] Processing CreateTablet for tablet a1b528a8761e4df98a1b48ccbba1ad6d (DEFAULT_TABLE table=default.Table_2db01113aba5414ba6dad68dc4f2bb56 [id=aedf3b3e233248739dc51824076b5ebe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.238088 9743 tablet_service.cc:1505] Processing CreateTablet for tablet 5bf2904d365243c6a261ef620203499f (DEFAULT_TABLE table=default.Table_d94bcf0455894a20baa4a30010869f62 [id=61bb9aa1a0f541029ded0842a69d4da3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.238467 9753 tablet_service.cc:1505] Processing CreateTablet for tablet 3dc44d6b0d5549fcb01308f0eaa53bd2 (DEFAULT_TABLE table=default.Table_7cc32a1529164529b4a58e2ea9df9058 [id=b47198eb136246538df2d9b4c0fb720f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.239090 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 348898b63dca46f4a5c9a7e34b545d86
W20251027 21:08:16.239089 9859 leader_election.cc:343] T 4bfa70ded2f54e3fb2ca744c723395e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:16.239197 9859 leader_election.cc:304] T 4bfa70ded2f54e3fb2ca744c723395e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:16.240487 9738 tablet_service.cc:1505] Processing CreateTablet for tablet 481199bc97514ca88e19603b3bf38dc7 (DEFAULT_TABLE table=default.Table_f1f34db7d6324d4c8cb9f5a8922e27b2 [id=205f1ddf7e84422e890f08e6ba8128da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.240921 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet aeae07c3906145149cea31e5a7c466fe
I20251027 21:08:16.242600 9303 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet b57c38c7a97f45feb1b7f90e85577a40
I20251027 21:08:16.242997 9303 catalog_manager.cc:6719] ReplaceTablet: tablet b57c38c7a97f45feb1b7f90e85577a40 deleted and replaced by tablet 56de83676c414abab0763bcac1f45fad
I20251027 21:08:16.243177 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 6443f2950db143bda9f098453ec4de25 (DEFAULT_TABLE table=default.Table_6f18157ff71c4f37b28566ed2e696c74 [id=cf0b25c72b984df1bc0e139bfdb3f0b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.243757 9740 tablet_service.cc:1505] Processing CreateTablet for tablet f1cde7c514ab408e9e5bc6da891ba493 (DEFAULT_TABLE table=default.Table_b9dc7d4bed0d40c29eab4ab96fd00d20 [id=e1e982e8f8c243bd98c70bc51fa5a1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.244071 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_70046cf1a7e44ca287756df96f94e51e" } new_table_name: "default.Table_b1daedcf87f6427393432861f93f3c34" modify_external_catalogs: true
I20251027 21:08:16.245149 9737 tablet_service.cc:1505] Processing CreateTablet for tablet b339d71e9cd948598b369ef5afa2d835 (DEFAULT_TABLE table=default.Table_55c1322435df4656993fef22102a4f62 [id=f335766496e14db4ac3d442956f509dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.245251 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_4a062664ca414e45a619e155863a314b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:16.245718 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a062664ca414e45a619e155863a314b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:16.246941 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_0424277af07b491e8869abb490f253a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:16.247401 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0424277af07b491e8869abb490f253a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:16.248114 11543 raft_consensus.cc:2749] T 4bfa70ded2f54e3fb2ca744c723395e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:16.248440 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_50b924ddfd184108bc67c4f821badb10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:16.248863 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50b924ddfd184108bc67c4f821badb10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:16.249926 9303 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_55f3c5dc04064eb7b371b99b5e64287a" } new_table_name: "default.Table_e64c357814f243deb3402b7bcc3c2876" modify_external_catalogs: true
I20251027 21:08:16.251642 9310 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet b8390bd307784327bacc17d0c0dd43d2
I20251027 21:08:16.252300 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_1cda8ade41884ccc959f290f1b26ffdb" } new_table_name: "default.Table_82fff28e8f234f10aa1b304ca24f2783" modify_external_catalogs: true
I20251027 21:08:16.255568 9903 tablet_service.cc:1552] Processing DeleteTablet for tablet b57c38c7a97f45feb1b7f90e85577a40 with delete_type TABLET_DATA_DELETED (replaced by tablet 56de83676c414abab0763bcac1f45fad) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:16.255854 9894 tablet_service.cc:1505] Processing CreateTablet for tablet 71f787da203c48b0bd11b838d680f24c (DEFAULT_TABLE table=default.Table_1bd31d83bf85421ea90591d545184406 [id=e5a72a856dae4dbaa634a5d100c38746]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.256700 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cdbde6272af147829ecadd67d634f049" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:16.256882 9634 raft_consensus.cc:2468] T cdbde6272af147829ecadd67d634f049 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:08:16.257004 9894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71f787da203c48b0bd11b838d680f24c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.257177 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "465be1b24d16457e834023a2b2194049" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:16.257349 9630 raft_consensus.cc:2468] T 465be1b24d16457e834023a2b2194049 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:08:16.257972 9710 leader_election.cc:304] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:16.258255 9710 leader_election.cc:304] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:16.258558 11724 raft_consensus.cc:2804] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:16.258630 11724 raft_consensus.cc:493] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:16.258708 11724 raft_consensus.cc:3060] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:16.255928 9898 tablet_service.cc:1552] Processing DeleteTablet for tablet b8390bd307784327bacc17d0c0dd43d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:16.261029 11724 raft_consensus.cc:515] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.265025 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22dbfa50996b47269983c107cc399000" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:16.265116 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2d7b7545ce74581a1548e404e004f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.265211 9635 raft_consensus.cc:2393] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77b0513f17c0472088901a1ce8b08218 in current term 1: Already voted for candidate 51902b6b61ff4d5584ff21896e847e14 in this term.
W20251027 21:08:16.265741 9568 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 47180.1us
I20251027 21:08:16.265947 11657 raft_consensus.cc:2804] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:16.266026 11657 raft_consensus.cc:493] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:16.266098 11657 raft_consensus.cc:3060] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:16.267627 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dc44d6b0d5549fcb01308f0eaa53bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.267746 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 952583fd7fde4b4baccdb98c01bdf9e1 (table default.Table_df5f4de40f7f46728f38580d23124544 [id=7527117eef9d46a694977cbf2c073974]) successfully deleted
W20251027 21:08:16.268617 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet ece28e82383540bea3457b1cbc65558b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ece28e82383540bea3457b1cbc65558b already in progress: creating tablet
I20251027 21:08:16.271174 11724 leader_election.cc:290] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:16.272130 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cdbde6272af147829ecadd67d634f049" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:16.272277 9635 raft_consensus.cc:3060] T cdbde6272af147829ecadd67d634f049 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:16.267475 9892 tablet_service.cc:1505] Processing CreateTablet for tablet f1cde7c514ab408e9e5bc6da891ba493 (DEFAULT_TABLE table=default.Table_b9dc7d4bed0d40c29eab4ab96fd00d20 [id=e1e982e8f8c243bd98c70bc51fa5a1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.256291 9904 tablet_service.cc:1552] Processing DeleteTablet for tablet 348898b63dca46f4a5c9a7e34b545d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:16.268254 9888 tablet_service.cc:1552] Processing DeleteTablet for tablet aeae07c3906145149cea31e5a7c466fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:16.272925 11724 raft_consensus.cc:2804] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:16.273541 11724 raft_consensus.cc:493] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:16.273924 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1cde7c514ab408e9e5bc6da891ba493. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.274088 11724 raft_consensus.cc:3060] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:16.277905 9902 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f7569995c8e4974b5ee3a8b0ec3fc8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:16.278353 11373 raft_consensus.cc:493] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.280536 11373 raft_consensus.cc:515] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.281081 11373 leader_election.cc:290] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:16.285853 9602 tablet_service.cc:1505] Processing CreateTablet for tablet b339d71e9cd948598b369ef5afa2d835 (DEFAULT_TABLE table=default.Table_55c1322435df4656993fef22102a4f62 [id=f335766496e14db4ac3d442956f509dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.286199 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b339d71e9cd948598b369ef5afa2d835. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.089462 10041 rpcz_store.cc:276] 1027 21:08:14.620108 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:14.624143 (+ 4035us) service_pool.cc:225] Handling call
1027 21:08:16.088800 (+1464657us) ts_tablet_manager.cc:1151] Deleting tablet e419e588b31d48c8bc19d7948081cc81
1027 21:08:16.088904 (+ 104us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:16.089167 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1464503}
I20251027 21:08:16.288681 10029 tablet_bootstrap.cc:492] T e419e588b31d48c8bc19d7948081cc81 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:16.288681 9752 tablet_service.cc:1505] Processing CreateTablet for tablet ab12fe6c9ad44f80ab610a0f03b611ec (DEFAULT_TABLE table=default.Table_6b527fc023dd45f4ab080eddeeb8bb76 [id=5888beb9284242a8bf5a7c8f2649d62c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.288993 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1337) took 1669 ms. Trace:
I20251027 21:08:16.289086 10041 rpcz_store.cc:276] 1027 21:08:14.619779 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:14.624352 (+ 4573us) service_pool.cc:225] Handling call
1027 21:08:16.288889 (+1664537us) ts_tablet_manager.cc:1151] Deleting tablet 815e12edcd3847ff8485ba768291a824
1027 21:08:16.288898 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:16.288976 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1664395}
I20251027 21:08:16.289372 10041 tablet_replica.cc:333] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:16.292598 9635 raft_consensus.cc:2468] T cdbde6272af147829ecadd67d634f049 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:16.292809 11373 raft_consensus.cc:493] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.292930 11373 raft_consensus.cc:515] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.293531 11373 leader_election.cc:290] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:16.290066 9613 tablet_service.cc:1505] Processing CreateTablet for tablet ab12fe6c9ad44f80ab610a0f03b611ec (DEFAULT_TABLE table=default.Table_6b527fc023dd45f4ab080eddeeb8bb76 [id=5888beb9284242a8bf5a7c8f2649d62c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.294606 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab12fe6c9ad44f80ab610a0f03b611ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.300323 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f7569995c8e4974b5ee3a8b0ec3fc8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:16.300797 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet 348898b63dca46f4a5c9a7e34b545d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:16.306066 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet aeae07c3906145149cea31e5a7c466fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:16.306823 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet b57c38c7a97f45feb1b7f90e85577a40 with delete_type TABLET_DATA_DELETED (replaced by tablet 56de83676c414abab0763bcac1f45fad) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:16.306833 10041 raft_consensus.cc:2243] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:16.307001 10041 raft_consensus.cc:2272] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:16.307199 9606 tablet_service.cc:1552] Processing DeleteTablet for tablet b8390bd307784327bacc17d0c0dd43d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:16.307498 10041 ts_tablet_manager.cc:1916] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:16.307622 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 71f787da203c48b0bd11b838d680f24c (DEFAULT_TABLE table=default.Table_1bd31d83bf85421ea90591d545184406 [id=e5a72a856dae4dbaa634a5d100c38746]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.307832 9605 tablet_service.cc:1505] Processing CreateTablet for tablet de1d777792d84fef92512783e289a696 (DEFAULT_TABLE table=default.Table_e68bfc4835674033aaef4620bb648593 [id=6608256694b846e8888fe7aefbefdd70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.308471 10029 tablet_bootstrap.cc:654] T e419e588b31d48c8bc19d7948081cc81 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:16.311036 11373 consensus_queue.cc:1048] T 1a48067b40324c79b1ee4eb431ce4a46 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:16.255442 9890 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c88e7aecf3f4e28987158f725e093da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:08:16.312057 9711 leader_election.cc:343] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 27bd3b2a011648b2b8f2f92944513751
I20251027 21:08:16.312570 9309 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_4ba26495af344e3cb082a4b781f1b208 [id=501b230a824d4a17bb1903c8409407b7] alter complete (version 2)
I20251027 21:08:16.290438 9598 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c88e7aecf3f4e28987158f725e093da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:16.313751 9711 leader_election.cc:343] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 465be1b24d16457e834023a2b2194049
I20251027 21:08:16.317179 9598 tablet_service.cc:1505] Processing CreateTablet for tablet 12f0e78425f441a7ae0b82d726103f3d (DEFAULT_TABLE table=default.Table_2f6ebdb6b30146d3af2e3911749f8c9c [id=e4ad08fd810e4efd979fccbaac1cb299]), partition=RANGE (key) PARTITION UNBOUNDED
W20251027 21:08:16.319968 9299 catalog_manager.cc:4952] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet e419e588b31d48c8bc19d7948081cc81 because the tablet was not found. No further retry: Not found: Tablet not found: e419e588b31d48c8bc19d7948081cc81
W20251027 21:08:16.320191 9299 catalog_manager.cc:4952] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 815e12edcd3847ff8485ba768291a824 because the tablet was not found. No further retry: Not found: Tablet not found: 815e12edcd3847ff8485ba768291a824
I20251027 21:08:16.320865 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_5853dee850df4fbd862a0aed5c5ad7bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:16.321990 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5853dee850df4fbd862a0aed5c5ad7bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:16.324617 11626 raft_consensus.cc:515] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.325357 11626 leader_election.cc:290] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:16.328233 9711 leader_election.cc:343] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: cdbde6272af147829ecadd67d634f049
I20251027 21:08:16.330163 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22dbfa50996b47269983c107cc399000" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:16.330328 9769 raft_consensus.cc:2468] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 0.
I20251027 21:08:16.330355 9710 leader_election.cc:304] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:16.331144 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c88e7aecf3f4e28987158f725e093da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:16.331219 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f7569995c8e4974b5ee3a8b0ec3fc8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:16.331524 11626 raft_consensus.cc:2804] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:16.331647 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 71f787da203c48b0bd11b838d680f24c (DEFAULT_TABLE table=default.Table_1bd31d83bf85421ea90591d545184406 [id=e5a72a856dae4dbaa634a5d100c38746]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.331691 11626 raft_consensus.cc:697] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:16.331846 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_5291bf5018e443118fabd8171b612b07" } new_table_name: "default.Table_7e009f7134fd4824a6bf6c6c79e2dec1" modify_external_catalogs: true
I20251027 21:08:16.332227 11626 consensus_queue.cc:237] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.332569 9309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d926e8bd576149279654e4290c084a03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:16.332790 9566 leader_election.cc:304] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
W20251027 21:08:16.333022 9309 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d926e8bd576149279654e4290c084a03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:16.339589 11657 raft_consensus.cc:515] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.340288 11657 leader_election.cc:290] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:16.342471 9566 leader_election.cc:343] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Not found: Tablet not found: 28a2e85021394f9d8a9ea524959b52a1
I20251027 21:08:16.344007 11657 raft_consensus.cc:2804] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251027 21:08:16.344079 11657 raft_consensus.cc:493] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:16.344130 11657 raft_consensus.cc:3060] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Advancing to term 2
I20251027 21:08:16.357638 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27bd3b2a011648b2b8f2f92944513751" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:16.357822 9635 raft_consensus.cc:3060] T 27bd3b2a011648b2b8f2f92944513751 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:16.358767 9612 tablet_service.cc:1552] Processing DeleteTablet for tablet 43611e33512b49d58485cf15de788be7 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e372257733148199d2dbe70d0887cbb) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:16.358932 9612 tablet_service.cc:1552] Processing DeleteTablet for tablet 9351a369dc1e4ac49f6ad419eab3a4c5 with delete_type TABLET_DATA_DELETED (replaced by tablet d5f4f3990da34f5e8f0385595f457427) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:16.361092 9612 tablet_service.cc:1552] Processing DeleteTablet for tablet 133e474f9133487f89c44dbfa6266ca7 with delete_type TABLET_DATA_DELETED (replaced by tablet 6cd39721fd28422896d517d349009c66) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:16.361258 9612 tablet_service.cc:1552] Processing DeleteTablet for tablet cc4635c00e9b40c49ce04debc76a04bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:00 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:16.361768 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71f787da203c48b0bd11b838d680f24c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.362241 9599 tablet_service.cc:1552] Processing DeleteTablet for tablet 95446c4333424f10a71829ce32d857e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:08 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:16.362290 9615 tablet_service.cc:1552] Processing DeleteTablet for tablet e477ca4b69c84327b04618d90406e19a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:16.361409 9612 tablet_service.cc:1552] Processing DeleteTablet for tablet 371aa8a57aee466ab01652d6061db4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:16.362663 11657 raft_consensus.cc:515] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.364109 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12f0e78425f441a7ae0b82d726103f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.367911 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de1d777792d84fef92512783e289a696. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.375327 9635 raft_consensus.cc:2468] T 27bd3b2a011648b2b8f2f92944513751 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:16.378089 9710 leader_election.cc:304] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:16.378089 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0c53b7b32c84a51a967e2a6a22258c6" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:16.395008 11657 leader_election.cc:290] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 election: Requested vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:16.395279 11543 raft_consensus.cc:493] T 72a6ca830c904373a4cca1169ae391ce P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.395438 11543 raft_consensus.cc:515] T 72a6ca830c904373a4cca1169ae391ce P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.396076 11543 leader_election.cc:290] T 72a6ca830c904373a4cca1169ae391ce P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:16.398370 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72a6ca830c904373a4cca1169ae391ce" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:16.399613 9860 leader_election.cc:343] T 72a6ca830c904373a4cca1169ae391ce P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:16.401777 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_2c6494231cc54b2f960bfdc57c7e376a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:16.402864 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46ed8d7e48c24914b8166940fc8d1842 consensus state has no leader
I20251027 21:08:16.403344 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f7b1c7b66f8d47a1821734d9795abbe6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:16.405264 9311 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_7768a03566534f8bbcacc53a14a957db" } new_table_name: "default.Table_ea026ae7f2874fa6b4574bfc3d837f0b" modify_external_catalogs: true
I20251027 21:08:16.406836 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_cc7058dae30a4bf2bf37506ff5f42be4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:16.415844 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c6494231cc54b2f960bfdc57c7e376a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:16.416942 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7b1c7b66f8d47a1821734d9795abbe6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:16.418069 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc7058dae30a4bf2bf37506ff5f42be4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:16.425774 9895 tablet_service.cc:1505] Processing CreateTablet for tablet de1d777792d84fef92512783e289a696 (DEFAULT_TABLE table=default.Table_e68bfc4835674033aaef4620bb648593 [id=6608256694b846e8888fe7aefbefdd70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.426280 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de1d777792d84fef92512783e289a696. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:16.429045 9303 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 793aebd92b65423796d6ad2d845519c2 consensus state has no leader
I20251027 21:08:16.430629 9909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0c53b7b32c84a51a967e2a6a22258c6" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678"
I20251027 21:08:16.430814 9909 raft_consensus.cc:3060] T d0c53b7b32c84a51a967e2a6a22258c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:16.433317 9909 raft_consensus.cc:2468] T d0c53b7b32c84a51a967e2a6a22258c6 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 1.
I20251027 21:08:16.444196 11724 raft_consensus.cc:515] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:16.445160 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a0d3203d12fc4823b3b3b1689f6bde62 consensus state has no leader
W20251027 21:08:16.445909 9711 leader_election.cc:343] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 22dbfa50996b47269983c107cc399000
I20251027 21:08:16.446035 9711 leader_election.cc:304] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:16.449008 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_9c9c668efe9c48438694310ecca64af4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:16.449553 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c9c668efe9c48438694310ecca64af4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:16.450040 10029 tablet_bootstrap.cc:492] T e419e588b31d48c8bc19d7948081cc81 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:16.450469 10029 ts_tablet_manager.cc:1403] T e419e588b31d48c8bc19d7948081cc81 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.162s user 0.007s sys 0.001s
I20251027 21:08:16.451740 11794 raft_consensus.cc:2804] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:16.451938 11794 raft_consensus.cc:697] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:16.452246 11794 consensus_queue.cc:237] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.453308 10029 raft_consensus.cc:359] T e419e588b31d48c8bc19d7948081cc81 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:16.454485 10029 raft_consensus.cc:385] T e419e588b31d48c8bc19d7948081cc81 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:16.455078 10029 raft_consensus.cc:740] T e419e588b31d48c8bc19d7948081cc81 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:16.456961 10029 consensus_queue.cc:260] T e419e588b31d48c8bc19d7948081cc81 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
W20251027 21:08:16.459199 9297 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 46ed8d7e48c24914b8166940fc8d1842 (table default.Table_b1daedcf87f6427393432861f93f3c34 [id=8d019f69a74c4ca3aff21211946556cd], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 46ed8d7e48c24914b8166940fc8d1842 consensus state has no leader
I20251027 21:08:16.463567 9312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_effb3e21eeb54ede9520b087b294612c" } modify_external_catalogs: true
I20251027 21:08:16.463832 9312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_effb3e21eeb54ede9520b087b294612c" } modify_external_catalogs: true
I20251027 21:08:16.468748 9305 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 334c580862bd4bf09be3f1710aa6a092 (table default.Table_06c15ff076cd4c24bc49b16b01bff965 [id=cf9029e184114ac0a5b641a496701122]). Expected version 1 got 0
I20251027 21:08:16.470288 10029 ts_tablet_manager.cc:1434] T e419e588b31d48c8bc19d7948081cc81 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.019s user 0.002s sys 0.002s
I20251027 21:08:16.476490 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 82366f8bf4154ffd87ef45e4a4d5f54b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:16.476806 10029 tablet_bootstrap.cc:492] T 7c15b74f53bc45e1af33b043f2cf225f P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:16.477298 11543 raft_consensus.cc:493] T 7828f466a5ad4d7997721b07f815f56d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.477442 11543 raft_consensus.cc:515] T 7828f466a5ad4d7997721b07f815f56d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:16.477692 9568 leader_election.cc:343] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 28a2e85021394f9d8a9ea524959b52a1
I20251027 21:08:16.477841 9568 leader_election.cc:304] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:16.478050 11657 raft_consensus.cc:2749] T 28a2e85021394f9d8a9ea524959b52a1 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:16.478072 11543 leader_election.cc:290] T 7828f466a5ad4d7997721b07f815f56d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:16.478822 9568 leader_election.cc:343] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 22dbfa50996b47269983c107cc399000
I20251027 21:08:16.479552 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7828f466a5ad4d7997721b07f815f56d" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:16.480019 9860 leader_election.cc:343] T 7828f466a5ad4d7997721b07f815f56d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:16.480573 9568 leader_election.cc:304] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:16.480798 11657 raft_consensus.cc:2804] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:16.481215 11657 raft_consensus.cc:697] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:16.481910 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 12f0e78425f441a7ae0b82d726103f3d (DEFAULT_TABLE table=default.Table_2f6ebdb6b30146d3af2e3911749f8c9c [id=e4ad08fd810e4efd979fccbaac1cb299]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.481952 11657 consensus_queue.cc:237] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:16.474388 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 013dc05778a047f691789cf21d241151 consensus state has no leader
I20251027 21:08:16.487884 9305 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 9be6b996df8145b9b0df1a2d2dc0ef03 (table default.Table_d09f1941d15f4517ac7561a66dd9b5e9 [id=b57564104d854b67a653c03de0650c09]). Expected version 1 got 0
I20251027 21:08:16.488564 9305 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet efbabe357ae947e09b1902b1fbc0ee40 (table default.Table_2a8ec86d85e7427597fe25a3c76c28b0 [id=c85b86cfe6494e919eb1eff998cc3ac7]). Expected version 1 got 0
I20251027 21:08:16.489315 9305 catalog_manager.cc:5649] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67). New cstate: current_term: 1 leader_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:16.490041 9306 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_18c39792431548bf80997615fa3dd505" } modify_external_catalogs: true
I20251027 21:08:16.501647 9306 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_18c39792431548bf80997615fa3dd505" } modify_external_catalogs: true
I20251027 21:08:16.499147 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_db3806bdc5c2498294b20102a1767b93" } modify_external_catalogs: true
I20251027 21:08:16.503054 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_db3806bdc5c2498294b20102a1767b93" } modify_external_catalogs: true
I20251027 21:08:16.503548 9305 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 93a31187eca14b8797737936a80e7292 (table default.Table_92fa91c5b7c24ec2bd881c758ba33889 [id=69a8a630d7ab4b2d8c7b580b41ebe3ba]). Expected version 1 got 0
I20251027 21:08:16.503854 9305 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet a9558d53008640739ac6fb45f1352fc0 (table default.Table_9821265dde764aee9571790dd10b4910 [id=c575fb3af44c46db8c198569241a425f]). Expected version 2 got 0
W20251027 21:08:16.509603 9311 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3bd720945d1345fea590b01987c0799c consensus state has no leader
I20251027 21:08:16.513417 9311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_bfc1511d230f4df297686bac2acbeac2" } modify_external_catalogs: true
I20251027 21:08:16.513600 9311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_bfc1511d230f4df297686bac2acbeac2" } modify_external_catalogs: true
I20251027 21:08:16.518659 10029 tablet_bootstrap.cc:654] T 7c15b74f53bc45e1af33b043f2cf225f P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:16.521114 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1b528a8761e4df98a1b48ccbba1ad6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.521328 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28d5507913044975b7fa6e34ba52a1b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.521508 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6443f2950db143bda9f098453ec4de25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.521677 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0e96c3547a34448b3d6e374825657a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.521870 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 481199bc97514ca88e19603b3bf38dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.522028 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f7569995c8e4974b5ee3a8b0ec3fc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.525218 9308 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6b1655f44b7f422f888954117b126d5f" } modify_external_catalogs: true
I20251027 21:08:16.525358 9308 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_6b1655f44b7f422f888954117b126d5f" } modify_external_catalogs: true
I20251027 21:08:16.526155 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1cde7c514ab408e9e5bc6da891ba493. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.526362 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a5a95f40a56495f85468817ce3c7a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.526667 9753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dc44d6b0d5549fcb01308f0eaa53bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.526849 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a3985daaf24490d972e7e65f85d87d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.527037 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71f787da203c48b0bd11b838d680f24c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.527204 9737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b339d71e9cd948598b369ef5afa2d835. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.527386 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bf2904d365243c6a261ef620203499f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.534161 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d564b20e53f6441dac2d1cd33475c3ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:16.534562 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c0fc37a1cb34302975ce24c347354a9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:16.534713 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d564b20e53f6441dac2d1cd33475c3ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:16.538244 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a109a8b9225f45cda6cbea7df02b8254. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.540095 9307 master_service.cc:954] ReplaceTablet: received request to replace tablet d715976e84754e31bfaf2cc9246342f4 from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:16.556526 9312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet c6a3cf8372184825a18d734ce8f94d62
I20251027 21:08:16.558074 9598 tablet_service.cc:1552] Processing DeleteTablet for tablet c6a3cf8372184825a18d734ce8f94d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:16.559187 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab12fe6c9ad44f80ab610a0f03b611ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.561312 10043 ts_tablet_manager.cc:1916] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:16.565992 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet c6a3cf8372184825a18d734ce8f94d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:16.570385 9306 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 7e32e33235b841aeb2a1eabf25bd9ff3
I20251027 21:08:16.571604 9598 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e32e33235b841aeb2a1eabf25bd9ff3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:16.571919 11800 raft_consensus.cc:493] T 823620e5486e4cf982fdf9105e8cc4f1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.572090 11800 raft_consensus.cc:515] T 823620e5486e4cf982fdf9105e8cc4f1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.572772 11800 leader_election.cc:290] T 823620e5486e4cf982fdf9105e8cc4f1 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:16.572971 11800 raft_consensus.cc:3060] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:16.576484 9710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.65:44531, user_credentials={real_user=slave}} blocked reactor thread for 198429us
I20251027 21:08:16.577185 9772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22dbfa50996b47269983c107cc399000" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
I20251027 21:08:16.578208 9773 raft_consensus.cc:3060] T 371051fb3491426e8cf7797317baa3db P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:16.590042 9711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.67:36705, user_credentials={real_user=slave}} blocked reactor thread for 144169us
I20251027 21:08:16.591884 9303 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet a9558d53008640739ac6fb45f1352fc0 (table default.Table_9821265dde764aee9571790dd10b4910 [id=c575fb3af44c46db8c198569241a425f]). Expected version 2 got 0
I20251027 21:08:16.592150 9303 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 57ef07c764664deaa4b2e9bcae7e7973 (table default.Table_7d6b8d0ba5ad42fa9fdbcf13c7647d41 [id=a541c8d30a9540f5a7d4dd98a5d3739a]). Expected version 1 got 0
I20251027 21:08:16.592339 9303 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 1a48067b40324c79b1ee4eb431ce4a46 (table default.Table_42823df1b61240fa9dc0e59703660dd5 [id=00404f9f36a34a2e8e1244799468d85e]). Expected version 1 got 0
I20251027 21:08:16.592649 9303 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet dd9e6656d8df43afbb7f70706bc14585 (table default.Table_f78d839900634631b6963b2d63f0d084 [id=3cdf432ec43d42e8bc2d253a92eac8d3]). Expected version 3 got 0
I20251027 21:08:16.593104 9303 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 59300e16b3e148a08a217fc24d1e572f (table default.Table_37d44f2ffd8b4816b4bff0b1960cd275 [id=a1dea3b74aa94995991d3123fbb2bf8d]). Expected version 1 got 0
I20251027 21:08:16.593173 9750 tablet_service.cc:1552] Processing DeleteTablet for tablet aeae07c3906145149cea31e5a7c466fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:16.593348 9742 tablet_service.cc:1552] Processing DeleteTablet for tablet 348898b63dca46f4a5c9a7e34b545d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:16.593355 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet b57c38c7a97f45feb1b7f90e85577a40 with delete_type TABLET_DATA_DELETED (replaced by tablet 56de83676c414abab0763bcac1f45fad) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:16.593742 11724 leader_election.cc:290] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:16.594409 10031 tablet_bootstrap.cc:492] T 0e64d21abef342de86555e876f020787 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:16.594533 10031 ts_tablet_manager.cc:1403] T 0e64d21abef342de86555e876f020787 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.487s user 0.004s sys 0.003s
I20251027 21:08:16.595111 10031 raft_consensus.cc:359] T 0e64d21abef342de86555e876f020787 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.591127 11626 raft_consensus.cc:493] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.595914 11626 raft_consensus.cc:515] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.596403 11626 leader_election.cc:290] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:16.598430 9303 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 9be6b996df8145b9b0df1a2d2dc0ef03 (table default.Table_d09f1941d15f4517ac7561a66dd9b5e9 [id=b57564104d854b67a653c03de0650c09]). Expected version 1 got 0
I20251027 21:08:16.598630 9303 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 334c580862bd4bf09be3f1710aa6a092 (table default.Table_06c15ff076cd4c24bc49b16b01bff965 [id=cf9029e184114ac0a5b641a496701122]). Expected version 1 got 0
I20251027 21:08:16.598846 9303 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 93a31187eca14b8797737936a80e7292 (table default.Table_92fa91c5b7c24ec2bd881c758ba33889 [id=69a8a630d7ab4b2d8c7b580b41ebe3ba]). Expected version 1 got 0
I20251027 21:08:16.601738 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet b8390bd307784327bacc17d0c0dd43d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:16.601990 9743 tablet_service.cc:1552] Processing DeleteTablet for tablet 22dbfa50996b47269983c107cc399000 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:16.602476 9738 tablet_service.cc:1552] Processing DeleteTablet for tablet c6a3cf8372184825a18d734ce8f94d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:16.602591 9753 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e4d35f229bc46e1859ded7c8196ab9d with delete_type TABLET_DATA_DELETED (replaced by tablet 280ee23d80d64587b18811ff06340eee) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:16.602630 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e32e33235b841aeb2a1eabf25bd9ff3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:16.602393 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 12f0e78425f441a7ae0b82d726103f3d (DEFAULT_TABLE table=default.Table_2f6ebdb6b30146d3af2e3911749f8c9c [id=e4ad08fd810e4efd979fccbaac1cb299]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.602797 9743 tablet_service.cc:1552] Processing DeleteTablet for tablet 67cb71ce43244d8ab9e71ead6bb315c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:15 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:16.602394 9745 tablet_service.cc:1552] Processing DeleteTablet for tablet 27bd3b2a011648b2b8f2f92944513751 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:16.602969 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12f0e78425f441a7ae0b82d726103f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.602993 9736 tablet_service.cc:1552] Processing DeleteTablet for tablet cdbde6272af147829ecadd67d634f049 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:54 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:16.603385 9742 tablet_service.cc:1505] Processing CreateTablet for tablet de1d777792d84fef92512783e289a696 (DEFAULT_TABLE table=default.Table_e68bfc4835674033aaef4620bb648593 [id=6608256694b846e8888fe7aefbefdd70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.603564 9304 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 72c1dc3b9eff47ee81df821deb789307 (table default.Table_ff5458dda9f042e189bd8cd69c06d025 [id=2a96776404334cf79ddb611a234a5844]). Expected version 2 got 0
I20251027 21:08:16.603709 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de1d777792d84fef92512783e289a696. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.603823 9304 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 371051fb3491426e8cf7797317baa3db (table default.Table_d481a540123e4620874b629ce6db65a7 [id=3b0044836ccc4b4082775de6eb111e49]). Expected version 1 got 0
I20251027 21:08:16.603984 9304 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 334c580862bd4bf09be3f1710aa6a092 (table default.Table_06c15ff076cd4c24bc49b16b01bff965 [id=cf9029e184114ac0a5b641a496701122]). Expected version 1 got 0
I20251027 21:08:16.605865 11797 tablet_replica.cc:333] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
W20251027 21:08:16.607127 9566 leader_election.cc:343] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:16.607581 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "823620e5486e4cf982fdf9105e8cc4f1" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:16.609059 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e32e33235b841aeb2a1eabf25bd9ff3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:16.612746 10041 ts_tablet_manager.cc:1929] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:16.612926 10041 log.cc:1199] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/7c6c2292ad0e4066958897fff01370af
I20251027 21:08:16.613291 10041 ts_tablet_manager.cc:1950] T 7c6c2292ad0e4066958897fff01370af P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:16.613747 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_dc89ee7309104635b4a2fa61174b0110" } new_table_name: "default.Table_098c2c6de1524f37b849095ed44c2714" modify_external_catalogs: true
I20251027 21:08:16.614193 10031 raft_consensus.cc:385] T 0e64d21abef342de86555e876f020787 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:16.614261 10031 raft_consensus.cc:740] T 0e64d21abef342de86555e876f020787 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:16.615252 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_cceefe8cd7854613b6cd45ae7b0b2d40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:16.615715 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cceefe8cd7854613b6cd45ae7b0b2d40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:16.617326 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_26fc21b6e1314363b30940ddb4326a1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:16.617734 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26fc21b6e1314363b30940ddb4326a1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 21:08:16.618525 9568 leader_election.cc:343] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 22dbfa50996b47269983c107cc399000
I20251027 21:08:16.620221 11658 consensus_queue.cc:1048] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:16.635082 9307 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet d715976e84754e31bfaf2cc9246342f4
I20251027 21:08:16.635617 9307 catalog_manager.cc:6719] ReplaceTablet: tablet d715976e84754e31bfaf2cc9246342f4 deleted and replaced by tablet 0846e67d795b42748f102264dc5e6ff9
I20251027 21:08:16.635900 11543 raft_consensus.cc:493] T efbabe357ae947e09b1902b1fbc0ee40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.636090 11543 raft_consensus.cc:515] T efbabe357ae947e09b1902b1fbc0ee40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.636540 9905 tablet_service.cc:1552] Processing DeleteTablet for tablet d715976e84754e31bfaf2cc9246342f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 0846e67d795b42748f102264dc5e6ff9) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:16.637115 9598 tablet_service.cc:1552] Processing DeleteTablet for tablet d715976e84754e31bfaf2cc9246342f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 0846e67d795b42748f102264dc5e6ff9) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:16.636921 11543 leader_election.cc:290] T efbabe357ae947e09b1902b1fbc0ee40 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:16.640290 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_f344a44c37a444fd8ce01f2a8a354907"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:16.640851 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f344a44c37a444fd8ce01f2a8a354907 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:16.649132 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12f0e78425f441a7ae0b82d726103f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.660961 9305 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_d481a540123e4620874b629ce6db65a7 [id=3b0044836ccc4b4082775de6eb111e49] alter complete (version 1)
I20251027 21:08:16.661409 10031 consensus_queue.cc:260] T 0e64d21abef342de86555e876f020787 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:16.661540 10032 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.011s sys 0.025s
W20251027 21:08:16.661609 10032 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.505s user 0.011s sys 0.025s
I20251027 21:08:16.662000 10031 ts_tablet_manager.cc:1434] T 0e64d21abef342de86555e876f020787 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.067s user 0.001s sys 0.001s
I20251027 21:08:16.665584 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "465be1b24d16457e834023a2b2194049" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:16.665759 9630 raft_consensus.cc:3060] T 465be1b24d16457e834023a2b2194049 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:16.667390 9566 consensus_peers.cc:597] T 59300e16b3e148a08a217fc24d1e572f P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:16.668443 9630 raft_consensus.cc:2468] T 465be1b24d16457e834023a2b2194049 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:16.665634 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9c86253ce914916bee6584675a82067" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:16.668885 11724 raft_consensus.cc:493] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.668944 9634 raft_consensus.cc:2468] T c9c86253ce914916bee6584675a82067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:08:16.668996 11724 raft_consensus.cc:515] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.669476 11724 leader_election.cc:290] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:16.665071 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_79a3420c8b6542868231c740293ab1fb" } new_table_name: "default.Table_0dc9764df87744e6aef303a10968595e" modify_external_catalogs: true
I20251027 21:08:16.671813 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1343) took 2051 ms. Trace:
I20251027 21:08:16.671945 10041 rpcz_store.cc:276] 1027 21:08:14.620305 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:14.624604 (+ 4299us) service_pool.cc:225] Handling call
1027 21:08:16.289326 (+1664722us) ts_tablet_manager.cc:1151] Deleting tablet 7c6c2292ad0e4066958897fff01370af
1027 21:08:16.289335 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:16.609632 (+320297us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:16.612660 (+ 3028us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:16.671788 (+ 59128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1664571}
I20251027 21:08:16.672436 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1339) took 2052 ms. Trace:
I20251027 21:08:16.672559 10041 rpcz_store.cc:276] 1027 21:08:14.619963 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:14.624831 (+ 4868us) service_pool.cc:225] Handling call
1027 21:08:16.672346 (+2047515us) ts_tablet_manager.cc:1151] Deleting tablet 618f1657a10947fd98976820564c20b3
1027 21:08:16.672355 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:16.672422 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2047377}
I20251027 21:08:16.672894 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1344) took 2037 ms. Trace:
I20251027 21:08:16.672991 10041 rpcz_store.cc:276] 1027 21:08:14.635219 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:14.635297 (+ 78us) service_pool.cc:225] Handling call
1027 21:08:16.672786 (+2037489us) ts_tablet_manager.cc:1151] Deleting tablet e8fe63ea8aba4614900ac254189464de
1027 21:08:16.672810 (+ 24us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:16.672883 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2037288}
I20251027 21:08:16.673244 10041 tablet_replica.cc:333] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:16.673450 10041 raft_consensus.cc:2243] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:16.673622 10041 raft_consensus.cc:2272] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:16.674129 10041 ts_tablet_manager.cc:1916] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:16.675676 10032 tablet_bootstrap.cc:492] T d1f24d21bb954ac9b62ce458c6a22462 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:16.675840 10032 ts_tablet_manager.cc:1403] T d1f24d21bb954ac9b62ce458c6a22462 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.522s user 0.011s sys 0.026s
W20251027 21:08:16.676371 9299 catalog_manager.cc:4952] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 618f1657a10947fd98976820564c20b3 because the tablet was not found. No further retry: Not found: Tablet not found: 618f1657a10947fd98976820564c20b3
W20251027 21:08:16.676731 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet e8fe63ea8aba4614900ac254189464de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8fe63ea8aba4614900ac254189464de already in progress: creating tablet
I20251027 21:08:16.676647 10032 raft_consensus.cc:359] T d1f24d21bb954ac9b62ce458c6a22462 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.676831 10032 raft_consensus.cc:385] T d1f24d21bb954ac9b62ce458c6a22462 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:16.676890 10032 raft_consensus.cc:740] T d1f24d21bb954ac9b62ce458c6a22462 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:16.677117 10032 consensus_queue.cc:260] T d1f24d21bb954ac9b62ce458c6a22462 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.677222 9303 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_c5a552fcdf214416a4b9217758c32b80 [id=8646c99b90414f4fa266cd7a31123d8e] alter complete (version 6)
I20251027 21:08:16.677873 10032 ts_tablet_manager.cc:1434] T d1f24d21bb954ac9b62ce458c6a22462 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:16.678210 10032 tablet_bootstrap.cc:492] T 6c61c360653346fabff0718f0990d100 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:16.679934 10032 tablet_bootstrap.cc:654] T 6c61c360653346fabff0718f0990d100 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:16.681432 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 0c88e7aecf3f4e28987158f725e093da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c88e7aecf3f4e28987158f725e093da already in progress: creating tablet
W20251027 21:08:16.681629 9297 catalog_manager.cc:4952] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 0f7569995c8e4974b5ee3a8b0ec3fc8b because the tablet was not found. No further retry: Not found: Tablet not found: 0f7569995c8e4974b5ee3a8b0ec3fc8b
W20251027 21:08:16.681826 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet aeae07c3906145149cea31e5a7c466fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aeae07c3906145149cea31e5a7c466fe already in progress: creating tablet
W20251027 21:08:16.682008 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 348898b63dca46f4a5c9a7e34b545d86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 348898b63dca46f4a5c9a7e34b545d86 already in progress: creating tablet
W20251027 21:08:16.682235 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet b57c38c7a97f45feb1b7f90e85577a40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b57c38c7a97f45feb1b7f90e85577a40 already in progress: creating tablet
I20251027 21:08:16.682592 9913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "823620e5486e4cf982fdf9105e8cc4f1" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" is_pre_election: true
I20251027 21:08:16.682845 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72a6ca830c904373a4cca1169ae391ce" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:16.683219 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7828f466a5ad4d7997721b07f815f56d" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:16.683512 9768 raft_consensus.cc:3060] T 823620e5486e4cf982fdf9105e8cc4f1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:16.684057 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "efbabe357ae947e09b1902b1fbc0ee40" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:16.687417 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet b8390bd307784327bacc17d0c0dd43d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8390bd307784327bacc17d0c0dd43d2 already in progress: creating tablet
I20251027 21:08:16.687829 11657 raft_consensus.cc:493] T e1385beb1dc14ee0b42a2f4e1f66f532 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.687937 11657 raft_consensus.cc:515] T e1385beb1dc14ee0b42a2f4e1f66f532 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.689162 11657 leader_election.cc:290] T e1385beb1dc14ee0b42a2f4e1f66f532 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:16.689416 9568 leader_election.cc:343] T e1385beb1dc14ee0b42a2f4e1f66f532 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: e1385beb1dc14ee0b42a2f4e1f66f532
I20251027 21:08:16.696882 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "efbabe357ae947e09b1902b1fbc0ee40" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:16.700409 10031 tablet_bootstrap.cc:492] T 00c90ff84d3f4a7f92fda235416eed1d P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
W20251027 21:08:16.705047 9860 leader_election.cc:343] T efbabe357ae947e09b1902b1fbc0ee40 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:16.710934 10032 tablet_bootstrap.cc:492] T 6c61c360653346fabff0718f0990d100 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:16.711076 10032 ts_tablet_manager.cc:1403] T 6c61c360653346fabff0718f0990d100 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20251027 21:08:16.711920 10032 raft_consensus.cc:359] T 6c61c360653346fabff0718f0990d100 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.712107 10032 raft_consensus.cc:385] T 6c61c360653346fabff0718f0990d100 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:16.712155 10032 raft_consensus.cc:740] T 6c61c360653346fabff0718f0990d100 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:16.712327 10032 consensus_queue.cc:260] T 6c61c360653346fabff0718f0990d100 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.712862 10032 ts_tablet_manager.cc:1434] T 6c61c360653346fabff0718f0990d100 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:16.713155 10032 tablet_bootstrap.cc:492] T 64e39cc210664c3d83eaf1fc576f83ce P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:16.714828 10032 tablet_bootstrap.cc:654] T 64e39cc210664c3d83eaf1fc576f83ce P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:16.715916 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b5e64e9079c44a4bbf34fb802068304f has no consensus state
I20251027 21:08:16.719578 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_0424277af07b491e8869abb490f253a3" } new_table_name: "default.Table_8ab2cb3624c24b35bca9f783fc9ffb4c" modify_external_catalogs: true
I20251027 21:08:16.726429 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8691aa22fe84cf29893ed4e79d156cc" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:16.726675 9635 raft_consensus.cc:2468] T d8691aa22fe84cf29893ed4e79d156cc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:08:16.730647 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_466a230c82db40f887726ce6ade334ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:16.731137 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_466a230c82db40f887726ce6ade334ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:16.736050 11658 raft_consensus.cc:493] T 2752b86151a6428c88bfaac1ff72cfb2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.736167 11658 raft_consensus.cc:515] T 2752b86151a6428c88bfaac1ff72cfb2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.736646 11658 leader_election.cc:290] T 2752b86151a6428c88bfaac1ff72cfb2 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:16.737564 9568 leader_election.cc:343] T 2752b86151a6428c88bfaac1ff72cfb2 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 2752b86151a6428c88bfaac1ff72cfb2
I20251027 21:08:16.738382 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet d2203ee62f7e404c95c5f86b5ae85feb with delete_type TABLET_DATA_DELETED (replaced by tablet 00bfb0bdf2ca4b288eac0c1dbee5765a) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:16.739061 9310 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_503f3a993de6475ea92ce352816fc954" } new_table_name: "default.Table_5c046a557b1c4ca2a40f25f7557e972b" modify_external_catalogs: true
I20251027 21:08:16.739180 10032 tablet_bootstrap.cc:492] T 64e39cc210664c3d83eaf1fc576f83ce P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:16.739324 10032 ts_tablet_manager.cc:1403] T 64e39cc210664c3d83eaf1fc576f83ce P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.003s
I20251027 21:08:16.740211 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1385beb1dc14ee0b42a2f4e1f66f532" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:16.740556 10032 raft_consensus.cc:359] T 64e39cc210664c3d83eaf1fc576f83ce P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.740746 10032 raft_consensus.cc:385] T 64e39cc210664c3d83eaf1fc576f83ce P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:16.740813 10032 raft_consensus.cc:740] T 64e39cc210664c3d83eaf1fc576f83ce P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:16.741003 10032 consensus_queue.cc:260] T 64e39cc210664c3d83eaf1fc576f83ce P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.741497 10032 ts_tablet_manager.cc:1434] T 64e39cc210664c3d83eaf1fc576f83ce P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 21:08:16.741787 10032 tablet_bootstrap.cc:492] T 2f641bc4dbed4c0abd0a55b253352507 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:16.743342 10032 tablet_bootstrap.cc:654] T 2f641bc4dbed4c0abd0a55b253352507 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:16.743801 10041 ts_tablet_manager.cc:1929] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:16.745695 9741 tablet_service.cc:1552] Processing DeleteTablet for tablet d715976e84754e31bfaf2cc9246342f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 0846e67d795b42748f102264dc5e6ff9) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:16.745962 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet d8691aa22fe84cf29893ed4e79d156cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:01 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:16.747089 11800 raft_consensus.cc:2749] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:16.747303 9772 raft_consensus.cc:3060] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Advancing to term 2
W20251027 21:08:16.744880 9859 leader_election.cc:343] T 72a6ca830c904373a4cca1169ae391ce P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:16.747766 10041 log.cc:1199] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/0820648719a34f379507efced9cc434b
I20251027 21:08:16.747766 9859 leader_election.cc:304] T 72a6ca830c904373a4cca1169ae391ce P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:16.748126 10041 ts_tablet_manager.cc:1950] T 0820648719a34f379507efced9cc434b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
W20251027 21:08:16.748291 9859 leader_election.cc:343] T 7828f466a5ad4d7997721b07f815f56d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:16.748369 9859 leader_election.cc:304] T 7828f466a5ad4d7997721b07f815f56d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:16.748804 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1346) took 1891 ms. Trace:
I20251027 21:08:16.752219 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet 6210d3d10fc44b9e9e2b0780883186ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:16.752472 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2752b86151a6428c88bfaac1ff72cfb2" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:16.756731 9772 raft_consensus.cc:2468] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51902b6b61ff4d5584ff21896e847e14 in term 2.
W20251027 21:08:16.758576 9711 leader_election.cc:343] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: cdbde6272af147829ecadd67d634f049
I20251027 21:08:16.758836 11797 raft_consensus.cc:2243] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:16.759613 9751 tablet_service.cc:1552] Processing DeleteTablet for tablet 465be1b24d16457e834023a2b2194049 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:56 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:16.765625 10043 ts_tablet_manager.cc:1929] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:16.765799 10043 log.cc:1199] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/588e2f58f07b4a32a78dffd5b92ceb12
W20251027 21:08:16.758858 9711 leader_election.cc:343] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 465be1b24d16457e834023a2b2194049
W20251027 21:08:16.770624 9711 leader_election.cc:343] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: c9c86253ce914916bee6584675a82067
I20251027 21:08:16.770809 9711 leader_election.cc:304] T 823620e5486e4cf982fdf9105e8cc4f1 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
W20251027 21:08:16.771247 9711 leader_election.cc:343] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 27bd3b2a011648b2b8f2f92944513751
I20251027 21:08:16.773694 11657 consensus_queue.cc:1048] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:16.775981 9752 tablet_service.cc:1552] Processing DeleteTablet for tablet c9c86253ce914916bee6584675a82067 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:16.776273 9710 leader_election.cc:304] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:16.776690 11626 raft_consensus.cc:2804] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:16.776861 11626 raft_consensus.cc:697] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:16.777171 11626 consensus_queue.cc:237] T 465be1b24d16457e834023a2b2194049 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.777957 9710 leader_election.cc:304] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:16.778218 11724 raft_consensus.cc:2804] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:16.778337 11724 raft_consensus.cc:493] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:16.778462 11724 raft_consensus.cc:3060] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:16.779201 10043 ts_tablet_manager.cc:1950] T 588e2f58f07b4a32a78dffd5b92ceb12 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:16.779901 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1150) took 7115 ms. Trace:
I20251027 21:08:16.780005 10043 rpcz_store.cc:276] 1027 21:08:09.664418 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.664596 (+ 178us) service_pool.cc:225] Handling call
1027 21:08:16.226077 (+6561481us) ts_tablet_manager.cc:1151] Deleting tablet 588e2f58f07b4a32a78dffd5b92ceb12
1027 21:08:16.226084 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:16.600010 (+373926us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:16.765514 (+165504us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:16.779884 (+ 14370us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6561235}
I20251027 21:08:16.780411 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet 588e2f58f07b4a32a78dffd5b92ceb12 (table default.Table_7a3ebf1ecf35428495d2d7bb5540530a [id=c4838f30ee9f4aec96e6c73b4ce15c7b]) successfully deleted
I20251027 21:08:16.780768 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1152) took 7085 ms. Trace:
I20251027 21:08:16.780869 10043 rpcz_store.cc:276] 1027 21:08:09.695759 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.701877 (+ 6118us) service_pool.cc:225] Handling call
1027 21:08:16.780664 (+7078787us) ts_tablet_manager.cc:1151] Deleting tablet 280ee23d80d64587b18811ff06340eee
1027 21:08:16.780675 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:16.780752 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7078583}
I20251027 21:08:16.780926 11799 consensus_queue.cc:1048] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251027 21:08:16.781383 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 280ee23d80d64587b18811ff06340eee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 280ee23d80d64587b18811ff06340eee already in progress: creating tablet
I20251027 21:08:16.781764 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1157) took 7040 ms. Trace:
I20251027 21:08:16.781839 10043 rpcz_store.cc:276] 1027 21:08:09.741468 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.741510 (+ 42us) service_pool.cc:225] Handling call
1027 21:08:16.781652 (+7040142us) ts_tablet_manager.cc:1151] Deleting tablet bc387163e2d64532b87c32c5842a285a
1027 21:08:16.781664 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:16.781750 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7039988}
W20251027 21:08:16.782235 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet bc387163e2d64532b87c32c5842a285a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc387163e2d64532b87c32c5842a285a already in progress: creating tablet
I20251027 21:08:16.782514 9710 leader_election.cc:304] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:16.782685 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1158) took 7040 ms. Trace:
I20251027 21:08:16.782809 11799 raft_consensus.cc:2804] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:16.782768 10043 rpcz_store.cc:276] 1027 21:08:09.741842 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.741885 (+ 43us) service_pool.cc:225] Handling call
1027 21:08:16.782588 (+7040703us) ts_tablet_manager.cc:1151] Deleting tablet 53948dc94c334329bdf81f1eba94863c
1027 21:08:16.782599 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:16.782671 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7040539}
I20251027 21:08:16.782967 11799 raft_consensus.cc:493] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:16.783094 11799 raft_consensus.cc:3060] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
W20251027 21:08:16.783134 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 53948dc94c334329bdf81f1eba94863c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53948dc94c334329bdf81f1eba94863c already in progress: creating tablet
I20251027 21:08:16.783574 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1165) took 7021 ms. Trace:
I20251027 21:08:16.783666 10043 rpcz_store.cc:276] 1027 21:08:09.762104 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.762662 (+ 558us) service_pool.cc:225] Handling call
1027 21:08:16.783356 (+7020694us) ts_tablet_manager.cc:1151] Deleting tablet d042a15408484fbc9ee9666476b103d5
1027 21:08:16.783368 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:16.783554 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7020475}
W20251027 21:08:16.784078 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet d042a15408484fbc9ee9666476b103d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d042a15408484fbc9ee9666476b103d5 already in progress: creating tablet
W20251027 21:08:16.784523 9566 consensus_peers.cc:597] T 57ef07c764664deaa4b2e9bcae7e7973 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:16.784673 9566 leader_election.cc:343] T e1385beb1dc14ee0b42a2f4e1f66f532 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:16.784744 9566 leader_election.cc:304] T e1385beb1dc14ee0b42a2f4e1f66f532 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:16.785393 11657 raft_consensus.cc:2749] T e1385beb1dc14ee0b42a2f4e1f66f532 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:16.790019 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_41d42b9036d140c3966516fb22753ca9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:16.790735 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41d42b9036d140c3966516fb22753ca9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:16.751468 11813 raft_consensus.cc:2749] T 72a6ca830c904373a4cca1169ae391ce P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:16.751672 11814 raft_consensus.cc:2749] T 7828f466a5ad4d7997721b07f815f56d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:16.797426 10041 rpcz_store.cc:276] 1027 21:08:14.857050 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:14.857108 (+ 58us) service_pool.cc:225] Handling call
1027 21:08:16.673208 (+1816100us) ts_tablet_manager.cc:1151] Deleting tablet 0820648719a34f379507efced9cc434b
1027 21:08:16.673216 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:16.694445 (+ 21229us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:16.743676 (+ 49231us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:16.748787 (+ 5111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1815534}
I20251027 21:08:16.797799 10029 tablet_bootstrap.cc:492] T 7c15b74f53bc45e1af33b043f2cf225f P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:16.797923 10029 ts_tablet_manager.cc:1403] T 7c15b74f53bc45e1af33b043f2cf225f P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.323s user 0.021s sys 0.053s
I20251027 21:08:16.799173 10029 raft_consensus.cc:359] T 7c15b74f53bc45e1af33b043f2cf225f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.799335 10029 raft_consensus.cc:385] T 7c15b74f53bc45e1af33b043f2cf225f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:16.754951 10032 tablet_bootstrap.cc:492] T 2f641bc4dbed4c0abd0a55b253352507 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:16.799904 10032 ts_tablet_manager.cc:1403] T 2f641bc4dbed4c0abd0a55b253352507 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.000s
I20251027 21:08:16.800006 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 0820648719a34f379507efced9cc434b (table default.Table_5a979a51f58942db97c3897dcd5feb08 [id=b0301c76f48f461c8ee32b925c91234b]) successfully deleted
I20251027 21:08:16.800421 10029 raft_consensus.cc:740] T 7c15b74f53bc45e1af33b043f2cf225f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:16.800645 10029 consensus_queue.cc:260] T 7c15b74f53bc45e1af33b043f2cf225f P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.800698 10032 raft_consensus.cc:359] T 2f641bc4dbed4c0abd0a55b253352507 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:16.800833 10032 raft_consensus.cc:385] T 2f641bc4dbed4c0abd0a55b253352507 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:16.800889 10032 raft_consensus.cc:740] T 2f641bc4dbed4c0abd0a55b253352507 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:16.801103 10032 consensus_queue.cc:260] T 2f641bc4dbed4c0abd0a55b253352507 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:16.801570 10032 ts_tablet_manager.cc:1434] T 2f641bc4dbed4c0abd0a55b253352507 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:16.801867 10032 tablet_bootstrap.cc:492] T 4d88aacbf47144cb837b99d68f6331e2 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:16.803854 10029 ts_tablet_manager.cc:1434] T 7c15b74f53bc45e1af33b043f2cf225f P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251027 21:08:16.804208 10032 tablet_bootstrap.cc:654] T 4d88aacbf47144cb837b99d68f6331e2 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:16.800457 10041 tablet_replica.cc:333] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:16.804793 10029 tablet_bootstrap.cc:492] T 46185d382f0a4dea803aad692ed3c89a P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
W20251027 21:08:16.804910 9711 consensus_peers.cc:597] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e4d35f229bc46e1859ded7c8196ab9d. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:16.805068 9711 leader_election.cc:343] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: d8691aa22fe84cf29893ed4e79d156cc
I20251027 21:08:16.805825 10041 raft_consensus.cc:2243] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:16.811137 10029 tablet_bootstrap.cc:654] T 46185d382f0a4dea803aad692ed3c89a P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:16.806387 9859 leader_election.cc:343] T efbabe357ae947e09b1902b1fbc0ee40 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:16.816171 10041 raft_consensus.cc:2272] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:16.816134 9859 leader_election.cc:304] T efbabe357ae947e09b1902b1fbc0ee40 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:16.816672 10041 ts_tablet_manager.cc:1916] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:16.816732 11814 raft_consensus.cc:493] T c97cba61e08d42a7bdb3f51db0a63ec8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.816845 11813 raft_consensus.cc:2749] T efbabe357ae947e09b1902b1fbc0ee40 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:16.816939 11814 raft_consensus.cc:515] T c97cba61e08d42a7bdb3f51db0a63ec8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.817711 10029 tablet_bootstrap.cc:492] T 46185d382f0a4dea803aad692ed3c89a P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:16.817833 10029 ts_tablet_manager.cc:1403] T 46185d382f0a4dea803aad692ed3c89a P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20251027 21:08:16.818287 9772 raft_consensus.cc:3060] T 334c580862bd4bf09be3f1710aa6a092 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:16.818532 10029 raft_consensus.cc:359] T 46185d382f0a4dea803aad692ed3c89a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.818674 10029 raft_consensus.cc:385] T 46185d382f0a4dea803aad692ed3c89a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:16.818715 10029 raft_consensus.cc:740] T 46185d382f0a4dea803aad692ed3c89a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:16.818867 11814 leader_election.cc:290] T c97cba61e08d42a7bdb3f51db0a63ec8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:16.818863 10029 consensus_queue.cc:260] T 46185d382f0a4dea803aad692ed3c89a P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.819304 11797 raft_consensus.cc:2272] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:16.819799 11797 ts_tablet_manager.cc:1916] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:16.821209 11799 raft_consensus.cc:515] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.821811 11799 leader_election.cc:290] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:16.822614 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c97cba61e08d42a7bdb3f51db0a63ec8" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:16.823184 9860 leader_election.cc:343] T c97cba61e08d42a7bdb3f51db0a63ec8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:16.824055 10032 tablet_bootstrap.cc:492] T 4d88aacbf47144cb837b99d68f6331e2 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:16.824196 10032 ts_tablet_manager.cc:1403] T 4d88aacbf47144cb837b99d68f6331e2 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.003s
I20251027 21:08:16.825024 10032 raft_consensus.cc:359] T 4d88aacbf47144cb837b99d68f6331e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:16.825205 10032 raft_consensus.cc:385] T 4d88aacbf47144cb837b99d68f6331e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:16.825280 10032 raft_consensus.cc:740] T 4d88aacbf47144cb837b99d68f6331e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:16.825486 10032 consensus_queue.cc:260] T 4d88aacbf47144cb837b99d68f6331e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:16.826061 10032 ts_tablet_manager.cc:1434] T 4d88aacbf47144cb837b99d68f6331e2 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:16.826366 10032 tablet_bootstrap.cc:492] T da23505b0fe742ddadfb829e37ca5499 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:16.828608 10029 ts_tablet_manager.cc:1434] T 46185d382f0a4dea803aad692ed3c89a P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251027 21:08:16.828898 10029 tablet_bootstrap.cc:492] T 93d2f7df14a74731aea84927f0b7ed45 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
W20251027 21:08:16.830582 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed66f9eeeb2b4b4bb77062c241d2140f consensus state has no leader
I20251027 21:08:16.830991 9304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_279dfb819b974d2085826df64b14d411" } new_table_name: "default.Table_91a95ff1364546eea4708137b670aee7" modify_external_catalogs: true
I20251027 21:08:16.833254 9891 tablet_service.cc:1505] Processing CreateTablet for tablet 2be0e24f77a04841bbd14f650528a850 (DEFAULT_TABLE table=default.Table_592c1afc91f1411a87d60f2a52e3482f [id=43b4192979f446e2add93e4987b2cc77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.833639 9891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2be0e24f77a04841bbd14f650528a850. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.835145 9305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_733ce9831862428581215e925ead54c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:16.835330 10032 tablet_bootstrap.cc:654] T da23505b0fe742ddadfb829e37ca5499 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:16.835824 9305 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_733ce9831862428581215e925ead54c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:16.837556 9896 tablet_service.cc:1505] Processing CreateTablet for tablet 426b564c3f714f689c7434445c0f9f51 (DEFAULT_TABLE table=default.Table_dcdcc2cc86764e80918034bdea343516 [id=988a4e84f72b47edb9c96aec823112c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.837881 9896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 426b564c3f714f689c7434445c0f9f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.838994 9598 tablet_service.cc:1505] Processing CreateTablet for tablet 2be0e24f77a04841bbd14f650528a850 (DEFAULT_TABLE table=default.Table_592c1afc91f1411a87d60f2a52e3482f [id=43b4192979f446e2add93e4987b2cc77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.839352 9598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2be0e24f77a04841bbd14f650528a850. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.842917 9892 tablet_service.cc:1552] Processing DeleteTablet for tablet c97cba61e08d42a7bdb3f51db0a63ec8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:10 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:16.843264 9902 tablet_service.cc:1505] Processing CreateTablet for tablet eb347e91740c4c588bece11f305c9f11 (DEFAULT_TABLE table=default.Table_a272b503b50445a8a4f586fdf6fc5666 [id=ee8c52e71e1244e3b3dec5d5ebd400d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.843606 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb347e91740c4c588bece11f305c9f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.843957 9905 tablet_service.cc:1505] Processing CreateTablet for tablet 7b86fac1ac464f79a138743fedbd1312 (DEFAULT_TABLE table=default.Table_58ac74f08bd94b5fa9ebfb174833d34a [id=274e2a7265ee4df099eea17aa6b6c9de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.844094 9888 tablet_service.cc:1505] Processing CreateTablet for tablet 81e12547f9dc4b138748cf70facdfeb1 (DEFAULT_TABLE table=default.Table_6fd9f233c7fe4ccc93d3ebaa7383e259 [id=db7c4f496be94b569af407fe2b7d6724]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.844223 9905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b86fac1ac464f79a138743fedbd1312. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.844457 9888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e12547f9dc4b138748cf70facdfeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.844534 9894 tablet_service.cc:1552] Processing DeleteTablet for tablet 19f3e1a9db434c519e26e864eb790706 with delete_type TABLET_DATA_DELETED (replaced by tablet 1778f4a7f81541bc8932201ead4189b4) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:16.844828 9890 tablet_service.cc:1505] Processing CreateTablet for tablet b5e64e9079c44a4bbf34fb802068304f (DEFAULT_TABLE table=default.Table_098c2c6de1524f37b849095ed44c2714 [id=da14b0b580eb410fb14464931583ea76]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.845090 9890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5e64e9079c44a4bbf34fb802068304f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.846378 9904 tablet_service.cc:1505] Processing CreateTablet for tablet 55e5d0d404fa42159b61c60467863d47 (DEFAULT_TABLE table=default.Table_81668449dbd44f34abb9087dd7decf2c [id=1a277ea035a34df28803dce964d1e174]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.846733 9904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55e5d0d404fa42159b61c60467863d47. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:16.831887 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 46713.8us
W20251027 21:08:16.848346 9566 leader_election.cc:343] T 2752b86151a6428c88bfaac1ff72cfb2 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:16.848460 9566 leader_election.cc:304] T 2752b86151a6428c88bfaac1ff72cfb2 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:16.848763 9898 tablet_service.cc:1505] Processing CreateTablet for tablet 1f4b18c6c0e8438b908dae6671a5a044 (DEFAULT_TABLE table=default.Table_bef609ceeb424e359bd9b80f5352001c [id=e8e3a9efabf9442baa2379581952df88]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.849037 9898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f4b18c6c0e8438b908dae6671a5a044. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.850387 9893 tablet_service.cc:1505] Processing CreateTablet for tablet 65f134685028466fa324e7bdb7f4e07a (DEFAULT_TABLE table=default.Table_14eab78b90fa4336a693f1c12f10c55e [id=e8e0ee20b40d4f8eb4cf31669af4c582]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.850754 9893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f134685028466fa324e7bdb7f4e07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.851565 9742 tablet_service.cc:1505] Processing CreateTablet for tablet 47cdefa148ba4e82b9050b2f7d557491 (DEFAULT_TABLE table=default.Table_f95fb451d6a342f081a72bdbfe6c6e9d [id=ab801d7bc55a49388e32d65776eef01d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.852017 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47cdefa148ba4e82b9050b2f7d557491. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.852203 9903 tablet_service.cc:1505] Processing CreateTablet for tablet 12e010e11e2a49f987a5396e7461e332 (DEFAULT_TABLE table=default.Table_8dc0282abb3842ab8ef43161ba3f67d2 [id=58e454015d924c23a7db31b6d2bc6324]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.852463 9903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12e010e11e2a49f987a5396e7461e332. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.853842 9892 tablet_service.cc:1505] Processing CreateTablet for tablet 42bc3a7870774e7bbf0ac6a5140a8975 (DEFAULT_TABLE table=default.Table_9f54e06fe6824db28057d61b77925c57 [id=a8fe500f72374c6db45f1ed918427f0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.854103 9892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42bc3a7870774e7bbf0ac6a5140a8975. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.854408 9899 tablet_service.cc:1505] Processing CreateTablet for tablet 12ebded1eb40494a82587836f8d8d859 (DEFAULT_TABLE table=default.Table_d2e87214f12c4f508a7bf6b4e03b4244 [id=0baf585fa5d944c4af056ee16e406245]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.854740 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12ebded1eb40494a82587836f8d8d859. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.858218 11373 raft_consensus.cc:2749] T 2752b86151a6428c88bfaac1ff72cfb2 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:16.858858 9566 leader_election.cc:304] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:16.859022 9613 tablet_service.cc:1505] Processing CreateTablet for tablet 7b86fac1ac464f79a138743fedbd1312 (DEFAULT_TABLE table=default.Table_58ac74f08bd94b5fa9ebfb174833d34a [id=274e2a7265ee4df099eea17aa6b6c9de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.859102 11373 raft_consensus.cc:2804] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [term 2 FOLLOWER]: Leader election won for term 2
I20251027 21:08:16.859181 11373 raft_consensus.cc:697] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [term 2 LEADER]: Becoming Leader. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Running, Role: LEADER
I20251027 21:08:16.859400 11373 consensus_queue.cc:237] T 22dbfa50996b47269983c107cc399000 P 51902b6b61ff4d5584ff21896e847e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.859840 9601 tablet_service.cc:1505] Processing CreateTablet for tablet 12ebded1eb40494a82587836f8d8d859 (DEFAULT_TABLE table=default.Table_d2e87214f12c4f508a7bf6b4e03b4244 [id=0baf585fa5d944c4af056ee16e406245]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.860026 9606 tablet_service.cc:1505] Processing CreateTablet for tablet 62ff6043cfb3494e993f636492ace3dc (DEFAULT_TABLE table=default.Table_b8502129426844198c5df5434968d674 [id=549fd8aec4a4447491bd0d93d949504e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.860460 9611 tablet_service.cc:1505] Processing CreateTablet for tablet 65f134685028466fa324e7bdb7f4e07a (DEFAULT_TABLE table=default.Table_14eab78b90fa4336a693f1c12f10c55e [id=e8e0ee20b40d4f8eb4cf31669af4c582]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.860544 9602 tablet_service.cc:1505] Processing CreateTablet for tablet b5e64e9079c44a4bbf34fb802068304f (DEFAULT_TABLE table=default.Table_098c2c6de1524f37b849095ed44c2714 [id=da14b0b580eb410fb14464931583ea76]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.860961 9605 tablet_service.cc:1505] Processing CreateTablet for tablet eb347e91740c4c588bece11f305c9f11 (DEFAULT_TABLE table=default.Table_a272b503b50445a8a4f586fdf6fc5666 [id=ee8c52e71e1244e3b3dec5d5ebd400d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.861352 9609 tablet_service.cc:1505] Processing CreateTablet for tablet 42bc3a7870774e7bbf0ac6a5140a8975 (DEFAULT_TABLE table=default.Table_9f54e06fe6824db28057d61b77925c57 [id=a8fe500f72374c6db45f1ed918427f0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.861672 9754 tablet_service.cc:1505] Processing CreateTablet for tablet 426b564c3f714f689c7434445c0f9f51 (DEFAULT_TABLE table=default.Table_dcdcc2cc86764e80918034bdea343516 [id=988a4e84f72b47edb9c96aec823112c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.861855 9608 tablet_service.cc:1505] Processing CreateTablet for tablet 81e12547f9dc4b138748cf70facdfeb1 (DEFAULT_TABLE table=default.Table_6fd9f233c7fe4ccc93d3ebaa7383e259 [id=db7c4f496be94b569af407fe2b7d6724]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.861964 9754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 426b564c3f714f689c7434445c0f9f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.862572 9600 tablet_service.cc:1505] Processing CreateTablet for tablet 1f4b18c6c0e8438b908dae6671a5a044 (DEFAULT_TABLE table=default.Table_bef609ceeb424e359bd9b80f5352001c [id=e8e3a9efabf9442baa2379581952df88]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.863020 9614 tablet_service.cc:1505] Processing CreateTablet for tablet 12e010e11e2a49f987a5396e7461e332 (DEFAULT_TABLE table=default.Table_8dc0282abb3842ab8ef43161ba3f67d2 [id=58e454015d924c23a7db31b6d2bc6324]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.863226 9303 catalog_manager.cc:6194] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: default.Table_4b0bc60a7c5e4888b79210296b7dfa20 [id=c9e7aa435d5b48e096c2b67b0e678eeb] alter complete (version 1)
I20251027 21:08:16.863404 9610 tablet_service.cc:1505] Processing CreateTablet for tablet 55e5d0d404fa42159b61c60467863d47 (DEFAULT_TABLE table=default.Table_81668449dbd44f34abb9087dd7decf2c [id=1a277ea035a34df28803dce964d1e174]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.863865 10032 tablet_bootstrap.cc:492] T da23505b0fe742ddadfb829e37ca5499 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:16.863991 10032 ts_tablet_manager.cc:1403] T da23505b0fe742ddadfb829e37ca5499 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.001s
I20251027 21:08:16.864657 10032 raft_consensus.cc:359] T da23505b0fe742ddadfb829e37ca5499 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.864796 10032 raft_consensus.cc:385] T da23505b0fe742ddadfb829e37ca5499 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:16.864838 10032 raft_consensus.cc:740] T da23505b0fe742ddadfb829e37ca5499 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:16.864988 10032 consensus_queue.cc:260] T da23505b0fe742ddadfb829e37ca5499 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.865406 10032 ts_tablet_manager.cc:1434] T da23505b0fe742ddadfb829e37ca5499 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:16.865639 10032 tablet_bootstrap.cc:492] T c25be263facd42c5bb6fa421160fbef0 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:16.866402 9595 tablet_service.cc:1552] Processing DeleteTablet for tablet 588e2f58f07b4a32a78dffd5b92ceb12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:53 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:16.866565 9594 tablet_service.cc:1552] Processing DeleteTablet for tablet e1385beb1dc14ee0b42a2f4e1f66f532 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:59 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:16.866673 9595 tablet_service.cc:1552] Processing DeleteTablet for tablet 2752b86151a6428c88bfaac1ff72cfb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:16.869616 9752 tablet_service.cc:1505] Processing CreateTablet for tablet 2be0e24f77a04841bbd14f650528a850 (DEFAULT_TABLE table=default.Table_592c1afc91f1411a87d60f2a52e3482f [id=43b4192979f446e2add93e4987b2cc77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.869748 9595 tablet_service.cc:1505] Processing CreateTablet for tablet 56de83676c414abab0763bcac1f45fad (DEFAULT_TABLE table=default.Table_f433bb2ad7be45fc82c323420fe9808f [id=37247477866142efab09f520c780f952]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.870003 9752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2be0e24f77a04841bbd14f650528a850. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.870709 10032 tablet_bootstrap.cc:654] T c25be263facd42c5bb6fa421160fbef0 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:16.876056 9303 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 1a48067b40324c79b1ee4eb431ce4a46 (table default.Table_42823df1b61240fa9dc0e59703660dd5 [id=00404f9f36a34a2e8e1244799468d85e]). Expected version 1 got 0
I20251027 21:08:16.876283 9303 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 59300e16b3e148a08a217fc24d1e572f (table default.Table_37d44f2ffd8b4816b4bff0b1960cd275 [id=a1dea3b74aa94995991d3123fbb2bf8d]). Expected version 1 got 0
I20251027 21:08:16.876549 9303 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 57ef07c764664deaa4b2e9bcae7e7973 (table default.Table_7d6b8d0ba5ad42fa9fdbcf13c7647d41 [id=a541c8d30a9540f5a7d4dd98a5d3739a]). Expected version 1 got 0
I20251027 21:08:16.876753 9303 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 72c1dc3b9eff47ee81df821deb789307 (table default.Table_ff5458dda9f042e189bd8cd69c06d025 [id=2a96776404334cf79ddb611a234a5844]). Expected version 2 got 0
I20251027 21:08:16.880568 11814 raft_consensus.cc:493] T 19f3e1a9db434c519e26e864eb790706 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.881031 11814 raft_consensus.cc:515] T 19f3e1a9db434c519e26e864eb790706 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.881140 9902 tablet_service.cc:1505] Processing CreateTablet for tablet 56de83676c414abab0763bcac1f45fad (DEFAULT_TABLE table=default.Table_f433bb2ad7be45fc82c323420fe9808f [id=37247477866142efab09f520c780f952]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.882543 11814 leader_election.cc:290] T 19f3e1a9db434c519e26e864eb790706 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:16.883299 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12ebded1eb40494a82587836f8d8d859. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.883473 9606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62ff6043cfb3494e993f636492ace3dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.883653 9611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f134685028466fa324e7bdb7f4e07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.883818 9595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56de83676c414abab0763bcac1f45fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.883956 9613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b86fac1ac464f79a138743fedbd1312. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.884338 9605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb347e91740c4c588bece11f305c9f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.884549 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19f3e1a9db434c519e26e864eb790706" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:16.884684 9614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12e010e11e2a49f987a5396e7461e332. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.885025 9608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e12547f9dc4b138748cf70facdfeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.885347 9609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42bc3a7870774e7bbf0ac6a5140a8975. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.885726 9902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56de83676c414abab0763bcac1f45fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.885738 9602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5e64e9079c44a4bbf34fb802068304f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.886121 9610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55e5d0d404fa42159b61c60467863d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.889691 9600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f4b18c6c0e8438b908dae6671a5a044. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:16.894712 9860 leader_election.cc:343] T 19f3e1a9db434c519e26e864eb790706 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:16.895349 10043 tablet_replica.cc:333] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:16.896497 10043 raft_consensus.cc:2243] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:16.905243 10043 raft_consensus.cc:2272] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:16.907002 11823 raft_consensus.cc:493] T 2f641bc4dbed4c0abd0a55b253352507 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251027 21:08:16.911831 9306 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7f56fb962be24bbba5e7b57e631e9d2e has no consensus state
I20251027 21:08:16.919018 9306 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 59300e16b3e148a08a217fc24d1e572f (table default.Table_37d44f2ffd8b4816b4bff0b1960cd275 [id=a1dea3b74aa94995991d3123fbb2bf8d]). Expected version 1 got 0
I20251027 21:08:16.919437 9306 catalog_manager.cc:5649] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65). New cstate: current_term: 1 leader_uuid: "51902b6b61ff4d5584ff21896e847e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20251027 21:08:16.920312 9306 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 7c15b74f53bc45e1af33b043f2cf225f (table default.Table_c97bb7def2964a3aacb4b0627420bf5a [id=32fe13ae5b414c62aba38f34c648d6ce]). Expected version 2 got 0
I20251027 21:08:16.920766 9306 catalog_manager.cc:5741] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531) does not have the latest schema for tablet 1a48067b40324c79b1ee4eb431ce4a46 (table default.Table_42823df1b61240fa9dc0e59703660dd5 [id=00404f9f36a34a2e8e1244799468d85e]). Expected version 1 got 0
I20251027 21:08:16.923677 10043 ts_tablet_manager.cc:1916] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:16.930538 11814 raft_consensus.cc:493] T 6c61c360653346fabff0718f0990d100 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.930730 11814 raft_consensus.cc:515] T 6c61c360653346fabff0718f0990d100 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.931543 11814 leader_election.cc:290] T 6c61c360653346fabff0718f0990d100 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:16.933012 9312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_55c3083fa17541dd9e553192e6eb657b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:16.933549 9312 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55c3083fa17541dd9e553192e6eb657b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:16.938853 9894 tablet_service.cc:1552] Processing DeleteTablet for tablet d1f24d21bb954ac9b62ce458c6a22462 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:57 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:16.939213 9894 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f641bc4dbed4c0abd0a55b253352507 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:02 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:16.939538 9894 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d88aacbf47144cb837b99d68f6331e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:58 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:16.939854 9894 tablet_service.cc:1552] Processing DeleteTablet for tablet 64e39cc210664c3d83eaf1fc576f83ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:07:55 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:16.943666 11799 raft_consensus.cc:493] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.943830 11799 raft_consensus.cc:515] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.944360 11799 leader_election.cc:290] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:16.907207 11823 raft_consensus.cc:515] T 2f641bc4dbed4c0abd0a55b253352507 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:16.955703 11823 leader_election.cc:290] T 2f641bc4dbed4c0abd0a55b253352507 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:16.958214 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c61c360653346fabff0718f0990d100" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:16.958904 9897 tablet_service.cc:1505] Processing CreateTablet for tablet 62ff6043cfb3494e993f636492ace3dc (DEFAULT_TABLE table=default.Table_b8502129426844198c5df5434968d674 [id=549fd8aec4a4447491bd0d93d949504e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.959100 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f641bc4dbed4c0abd0a55b253352507" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:16.959380 9897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62ff6043cfb3494e993f636492ace3dc. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 21:08:16.959645 9860 leader_election.cc:343] T 6c61c360653346fabff0718f0990d100 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:16.960014 9860 leader_election.cc:343] T 2f641bc4dbed4c0abd0a55b253352507 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:16.963444 9310 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 918a4a19c64f4261b58bea03d1b8251e consensus state has no leader
I20251027 21:08:16.963878 9310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_693e20fac5a342359ccabdad457e0444"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:16.964464 9310 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_693e20fac5a342359ccabdad457e0444 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:16.966219 9307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_09a9d967ab8141449bc217be3ab4ef95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:16.966768 9307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09a9d967ab8141449bc217be3ab4ef95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:16.974108 11823 consensus_queue.cc:1048] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:16.979393 11823 raft_consensus.cc:493] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:16.979611 11823 raft_consensus.cc:515] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:16.980345 11823 leader_election.cc:290] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:16.981588 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bc5fbdbbea0406a961e1a4df51b8762" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:16.982064 9860 leader_election.cc:343] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:16.985045 9604 tablet_service.cc:1505] Processing CreateTablet for tablet 47cdefa148ba4e82b9050b2f7d557491 (DEFAULT_TABLE table=default.Table_f95fb451d6a342f081a72bdbfe6c6e9d [id=ab801d7bc55a49388e32d65776eef01d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.985452 9604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47cdefa148ba4e82b9050b2f7d557491. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.992089 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8691aa22fe84cf29893ed4e79d156cc" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:16.992247 9634 raft_consensus.cc:3060] T d8691aa22fe84cf29893ed4e79d156cc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:16.992621 9895 tablet_service.cc:1505] Processing CreateTablet for tablet 47cdefa148ba4e82b9050b2f7d557491 (DEFAULT_TABLE table=default.Table_f95fb451d6a342f081a72bdbfe6c6e9d [id=ab801d7bc55a49388e32d65776eef01d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.993070 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47cdefa148ba4e82b9050b2f7d557491. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.995355 9634 raft_consensus.cc:2468] T d8691aa22fe84cf29893ed4e79d156cc P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:16.995651 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6210d3d10fc44b9e9e2b0780883186ce" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:16.995806 9630 raft_consensus.cc:2468] T 6210d3d10fc44b9e9e2b0780883186ce P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 0.
I20251027 21:08:16.997713 9749 tablet_service.cc:1505] Processing CreateTablet for tablet 7b86fac1ac464f79a138743fedbd1312 (DEFAULT_TABLE table=default.Table_58ac74f08bd94b5fa9ebfb174833d34a [id=274e2a7265ee4df099eea17aa6b6c9de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.998065 9751 tablet_service.cc:1505] Processing CreateTablet for tablet b5e64e9079c44a4bbf34fb802068304f (DEFAULT_TABLE table=default.Table_098c2c6de1524f37b849095ed44c2714 [id=da14b0b580eb410fb14464931583ea76]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.998430 9751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5e64e9079c44a4bbf34fb802068304f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.997713 9747 tablet_service.cc:1505] Processing CreateTablet for tablet 62ff6043cfb3494e993f636492ace3dc (DEFAULT_TABLE table=default.Table_b8502129426844198c5df5434968d674 [id=549fd8aec4a4447491bd0d93d949504e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.998800 9607 tablet_service.cc:1505] Processing CreateTablet for tablet 426b564c3f714f689c7434445c0f9f51 (DEFAULT_TABLE table=default.Table_dcdcc2cc86764e80918034bdea343516 [id=988a4e84f72b47edb9c96aec823112c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:16.999122 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62ff6043cfb3494e993f636492ace3dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:16.999224 9607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 426b564c3f714f689c7434445c0f9f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:17.002772 9749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b86fac1ac464f79a138743fedbd1312. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:17.007532 11823 raft_consensus.cc:493] T 64e39cc210664c3d83eaf1fc576f83ce P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:17.007812 11823 raft_consensus.cc:515] T 64e39cc210664c3d83eaf1fc576f83ce P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.008816 11823 leader_election.cc:290] T 64e39cc210664c3d83eaf1fc576f83ce P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:17.009104 11826 raft_consensus.cc:493] T 46185d382f0a4dea803aad692ed3c89a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:17.009308 11826 raft_consensus.cc:515] T 46185d382f0a4dea803aad692ed3c89a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.010018 11826 leader_election.cc:290] T 46185d382f0a4dea803aad692ed3c89a P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:17.010792 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64e39cc210664c3d83eaf1fc576f83ce" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:17.013486 9568 leader_election.cc:343] T 46185d382f0a4dea803aad692ed3c89a P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 46185d382f0a4dea803aad692ed3c89a
I20251027 21:08:17.015695 11823 raft_consensus.cc:493] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:17.015970 11823 raft_consensus.cc:515] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.016863 11823 leader_election.cc:290] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
W20251027 21:08:17.017524 9860 leader_election.cc:343] T 64e39cc210664c3d83eaf1fc576f83ce P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:17.017518 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c15b74f53bc45e1af33b043f2cf225f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
I20251027 21:08:17.017675 9630 raft_consensus.cc:2468] T 7c15b74f53bc45e1af33b043f2cf225f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 0.
I20251027 21:08:17.019104 9860 leader_election.cc:304] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:17.019491 11823 raft_consensus.cc:2804] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:17.019652 11823 raft_consensus.cc:493] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:17.019820 11823 raft_consensus.cc:3060] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:17.020825 10029 tablet_bootstrap.cc:700] T 93d2f7df14a74731aea84927f0b7ed45 P 51902b6b61ff4d5584ff21896e847e14: Time spent opening tablet: real 0.191s user 0.001s sys 0.000s
I20251027 21:08:17.021121 10029 tablet_bootstrap.cc:654] T 93d2f7df14a74731aea84927f0b7ed45 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:17.022821 10032 tablet_bootstrap.cc:492] T c25be263facd42c5bb6fa421160fbef0 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:17.022943 10032 ts_tablet_manager.cc:1403] T c25be263facd42c5bb6fa421160fbef0 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.157s user 0.012s sys 0.021s
I20251027 21:08:17.023687 10032 raft_consensus.cc:359] T c25be263facd42c5bb6fa421160fbef0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.023864 10032 raft_consensus.cc:385] T c25be263facd42c5bb6fa421160fbef0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:17.023908 10032 raft_consensus.cc:740] T c25be263facd42c5bb6fa421160fbef0 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:17.024125 10032 consensus_queue.cc:260] T c25be263facd42c5bb6fa421160fbef0 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.024626 10032 ts_tablet_manager.cc:1434] T c25be263facd42c5bb6fa421160fbef0 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:17.024880 10032 tablet_bootstrap.cc:492] T 849b3e9fc34045939a7a678fa8269d7f P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:17.026501 10032 tablet_bootstrap.cc:654] T 849b3e9fc34045939a7a678fa8269d7f P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:17.023653 11823 raft_consensus.cc:515] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.027644 11823 leader_election.cc:290] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:17.028584 10029 tablet_bootstrap.cc:492] T 93d2f7df14a74731aea84927f0b7ed45 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:17.028704 10029 ts_tablet_manager.cc:1403] T 93d2f7df14a74731aea84927f0b7ed45 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.200s user 0.002s sys 0.000s
I20251027 21:08:17.028950 11800 raft_consensus.cc:2749] T 823620e5486e4cf982fdf9105e8cc4f1 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:17.029273 10029 raft_consensus.cc:359] T 93d2f7df14a74731aea84927f0b7ed45 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.029415 10029 raft_consensus.cc:385] T 93d2f7df14a74731aea84927f0b7ed45 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:17.029480 10029 raft_consensus.cc:740] T 93d2f7df14a74731aea84927f0b7ed45 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:17.029654 10029 consensus_queue.cc:260] T 93d2f7df14a74731aea84927f0b7ed45 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.030050 10029 ts_tablet_manager.cc:1434] T 93d2f7df14a74731aea84927f0b7ed45 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:17.030318 10029 tablet_bootstrap.cc:492] T fb7fdc91156e41de80872b5c941c5d95 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:17.032018 10029 tablet_bootstrap.cc:654] T fb7fdc91156e41de80872b5c941c5d95 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:17.033190 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c15b74f53bc45e1af33b043f2cf225f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:17.033360 9630 raft_consensus.cc:3060] T 7c15b74f53bc45e1af33b043f2cf225f P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:17.036079 10029 tablet_bootstrap.cc:492] T fb7fdc91156e41de80872b5c941c5d95 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:17.036218 10029 ts_tablet_manager.cc:1403] T fb7fdc91156e41de80872b5c941c5d95 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251027 21:08:17.036479 9630 raft_consensus.cc:2468] T 7c15b74f53bc45e1af33b043f2cf225f P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fbd567f929a4f6d8e4d78feef7ed678 in term 1.
I20251027 21:08:17.036873 9860 leader_election.cc:304] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14; no voters:
I20251027 21:08:17.037189 11823 raft_consensus.cc:2804] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:17.037467 11823 raft_consensus.cc:697] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 1 LEADER]: Becoming Leader. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Running, Role: LEADER
I20251027 21:08:17.037746 11823 consensus_queue.cc:237] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.038112 10029 raft_consensus.cc:359] T fb7fdc91156e41de80872b5c941c5d95 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.038321 10029 raft_consensus.cc:385] T fb7fdc91156e41de80872b5c941c5d95 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:17.038395 10029 raft_consensus.cc:740] T fb7fdc91156e41de80872b5c941c5d95 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:17.038628 10029 consensus_queue.cc:260] T fb7fdc91156e41de80872b5c941c5d95 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.039193 10029 ts_tablet_manager.cc:1434] T fb7fdc91156e41de80872b5c941c5d95 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251027 21:08:17.039534 10029 tablet_bootstrap.cc:492] T 532ea47b15b544dfb05ddb2bdacf3953 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:17.040479 9309 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 03986aea266e45d49a969ed5cad865d6
I20251027 21:08:17.041504 10029 tablet_bootstrap.cc:654] T 532ea47b15b544dfb05ddb2bdacf3953 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:17.042420 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 03986aea266e45d49a969ed5cad865d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:17.042976 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet 03986aea266e45d49a969ed5cad865d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:17.054387 9309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e58c8bb0dfc540ba81698903abdfdbb0" } modify_external_catalogs: true
I20251027 21:08:17.054641 9309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_e58c8bb0dfc540ba81698903abdfdbb0" } modify_external_catalogs: true
I20251027 21:08:17.056149 10029 tablet_bootstrap.cc:492] T 532ea47b15b544dfb05ddb2bdacf3953 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:17.056354 10029 ts_tablet_manager.cc:1403] T 532ea47b15b544dfb05ddb2bdacf3953 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.001s
I20251027 21:08:17.057400 10029 raft_consensus.cc:359] T 532ea47b15b544dfb05ddb2bdacf3953 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.057628 10029 raft_consensus.cc:385] T 532ea47b15b544dfb05ddb2bdacf3953 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:17.057700 10029 raft_consensus.cc:740] T 532ea47b15b544dfb05ddb2bdacf3953 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:17.057951 10029 consensus_queue.cc:260] T 532ea47b15b544dfb05ddb2bdacf3953 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.058816 10029 ts_tablet_manager.cc:1434] T 532ea47b15b544dfb05ddb2bdacf3953 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251027 21:08:17.059263 10029 tablet_bootstrap.cc:492] T 618f1657a10947fd98976820564c20b3 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:17.061460 10041 ts_tablet_manager.cc:1929] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:17.061872 10029 tablet_bootstrap.cc:654] T 618f1657a10947fd98976820564c20b3 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:17.064937 10043 ts_tablet_manager.cc:1929] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:16.990049 9303 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet da23505b0fe742ddadfb829e37ca5499 (table default.Table_42156f7dab5f4522aa9301297d4a10da [id=a1ac88d6c244410ea3cbcd9a3e8b4827]). Expected version 1 got 0
I20251027 21:08:17.073045 9303 catalog_manager.cc:5741] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705) does not have the latest schema for tablet 57ef07c764664deaa4b2e9bcae7e7973 (table default.Table_7d6b8d0ba5ad42fa9fdbcf13c7647d41 [id=a541c8d30a9540f5a7d4dd98a5d3739a]). Expected version 1 got 0
I20251027 21:08:17.081601 10031 tablet_bootstrap.cc:654] T 00c90ff84d3f4a7f92fda235416eed1d P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:17.083447 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46185d382f0a4dea803aad692ed3c89a" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:17.084323 9710 leader_election.cc:304] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:17.100790 9710 leader_election.cc:304] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters:
I20251027 21:08:17.100728 11800 consensus_queue.cc:1048] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:17.101159 11800 raft_consensus.cc:2804] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:17.101289 11800 raft_consensus.cc:697] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:17.101738 11800 consensus_queue.cc:237] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.101378 11808 raft_consensus.cc:2804] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 21:08:17.102216 11808 raft_consensus.cc:493] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 21:08:17.102272 11808 raft_consensus.cc:3060] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:17.112978 9736 tablet_service.cc:1505] Processing CreateTablet for tablet 12ebded1eb40494a82587836f8d8d859 (DEFAULT_TABLE table=default.Table_d2e87214f12c4f508a7bf6b4e03b4244 [id=0baf585fa5d944c4af056ee16e406245]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:17.113377 9736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12ebded1eb40494a82587836f8d8d859. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:17.113873 9740 tablet_service.cc:1552] Processing DeleteTablet for tablet 03986aea266e45d49a969ed5cad865d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:17.114097 9745 tablet_service.cc:1505] Processing CreateTablet for tablet 65f134685028466fa324e7bdb7f4e07a (DEFAULT_TABLE table=default.Table_14eab78b90fa4336a693f1c12f10c55e [id=e8e0ee20b40d4f8eb4cf31669af4c582]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:17.114228 9774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c61c360653346fabff0718f0990d100" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:17.114290 9739 tablet_service.cc:1505] Processing CreateTablet for tablet 81e12547f9dc4b138748cf70facdfeb1 (DEFAULT_TABLE table=default.Table_6fd9f233c7fe4ccc93d3ebaa7383e259 [id=db7c4f496be94b569af407fe2b7d6724]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:17.114078 9775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19f3e1a9db434c519e26e864eb790706" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:17.114595 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e12547f9dc4b138748cf70facdfeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:17.114560 9743 tablet_service.cc:1505] Processing CreateTablet for tablet 56de83676c414abab0763bcac1f45fad (DEFAULT_TABLE table=default.Table_f433bb2ad7be45fc82c323420fe9808f [id=37247477866142efab09f520c780f952]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:17.114840 9743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56de83676c414abab0763bcac1f45fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:17.114987 9771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c97cba61e08d42a7bdb3f51db0a63ec8" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:17.115008 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 55e5d0d404fa42159b61c60467863d47 (DEFAULT_TABLE table=default.Table_81668449dbd44f34abb9087dd7decf2c [id=1a277ea035a34df28803dce964d1e174]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:17.115299 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55e5d0d404fa42159b61c60467863d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:17.115321 9769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f641bc4dbed4c0abd0a55b253352507" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:17.114087 9741 tablet_service.cc:1505] Processing CreateTablet for tablet 42bc3a7870774e7bbf0ac6a5140a8975 (DEFAULT_TABLE table=default.Table_9f54e06fe6824db28057d61b77925c57 [id=a8fe500f72374c6db45f1ed918427f0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:17.115770 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42bc3a7870774e7bbf0ac6a5140a8975. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:17.116215 9742 tablet_service.cc:1505] Processing CreateTablet for tablet eb347e91740c4c588bece11f305c9f11 (DEFAULT_TABLE table=default.Table_a272b503b50445a8a4f586fdf6fc5666 [id=ee8c52e71e1244e3b3dec5d5ebd400d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:17.116582 9742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb347e91740c4c588bece11f305c9f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:17.115204 9770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bc5fbdbbea0406a961e1a4df51b8762" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:17.118785 9711 leader_election.cc:343] T d8691aa22fe84cf29893ed4e79d156cc P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: d8691aa22fe84cf29893ed4e79d156cc
I20251027 21:08:17.118824 11794 raft_consensus.cc:493] T 0e64d21abef342de86555e876f020787 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:17.118912 11794 raft_consensus.cc:515] T 0e64d21abef342de86555e876f020787 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:17.119064 9711 leader_election.cc:343] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 6210d3d10fc44b9e9e2b0780883186ce
W20251027 21:08:17.119875 9711 consensus_peers.cc:597] T d2203ee62f7e404c95c5f86b5ae85feb P 77b0513f17c0472088901a1ce8b08218 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2203ee62f7e404c95c5f86b5ae85feb. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:17.114758 9766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64e39cc210664c3d83eaf1fc576f83ce" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:17.120247 9711 leader_election.cc:343] T 0e64d21abef342de86555e876f020787 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 0e64d21abef342de86555e876f020787
I20251027 21:08:17.120549 9765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c15b74f53bc45e1af33b043f2cf225f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:17.120676 9744 tablet_service.cc:1505] Processing CreateTablet for tablet 1f4b18c6c0e8438b908dae6671a5a044 (DEFAULT_TABLE table=default.Table_bef609ceeb424e359bd9b80f5352001c [id=e8e3a9efabf9442baa2379581952df88]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:17.120980 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f4b18c6c0e8438b908dae6671a5a044. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:17.123447 9745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f134685028466fa324e7bdb7f4e07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:17.124563 9750 tablet_service.cc:1505] Processing CreateTablet for tablet 12e010e11e2a49f987a5396e7461e332 (DEFAULT_TABLE table=default.Table_8dc0282abb3842ab8ef43161ba3f67d2 [id=58e454015d924c23a7db31b6d2bc6324]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 21:08:17.124866 9750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12e010e11e2a49f987a5396e7461e332. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 21:08:17.119421 11794 leader_election.cc:290] T 0e64d21abef342de86555e876f020787 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:17.125475 9764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c15b74f53bc45e1af33b043f2cf225f" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218"
W20251027 21:08:17.126048 9710 leader_election.cc:343] T 0e64d21abef342de86555e876f020787 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Not found: Tablet not found: 0e64d21abef342de86555e876f020787
I20251027 21:08:17.126201 9710 leader_election.cc:304] T 0e64d21abef342de86555e876f020787 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 51902b6b61ff4d5584ff21896e847e14
I20251027 21:08:17.126634 11794 raft_consensus.cc:2749] T 0e64d21abef342de86555e876f020787 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:17.153776 11851 tablet.cc:1722] T 371051fb3491426e8cf7797317baa3db P 77b0513f17c0472088901a1ce8b08218: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251027 21:08:17.168569 11797 ts_tablet_manager.cc:1929] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:17.168679 11797 log.cc:1199] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/22dbfa50996b47269983c107cc399000
I20251027 21:08:17.168972 11797 ts_tablet_manager.cc:1950] T 22dbfa50996b47269983c107cc399000 P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:17.169515 11797 tablet_replica.cc:333] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:17.189780 11724 raft_consensus.cc:515] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.190611 11724 leader_election.cc:290] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:17.193811 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 22dbfa50996b47269983c107cc399000 (table default.Table_cb55ac1fcbad4075b84ec473bf89aa2b [id=be549850a5104d288311e062e1362583]) successfully deleted
W20251027 21:08:17.194087 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet c6a3cf8372184825a18d734ce8f94d62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6a3cf8372184825a18d734ce8f94d62 already in progress: creating tablet
I20251027 21:08:17.196990 11808 raft_consensus.cc:515] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.197559 11808 leader_election.cc:290] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Requested vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
W20251027 21:08:17.198834 9711 leader_election.cc:343] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 6210d3d10fc44b9e9e2b0780883186ce
W20251027 21:08:17.199476 9711 leader_election.cc:343] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: c9c86253ce914916bee6584675a82067
I20251027 21:08:17.220193 10041 log.cc:1199] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/5054bbc3e27243298b4df6b0a87af8be
W20251027 21:08:17.118332 9859 leader_election.cc:343] T 6c61c360653346fabff0718f0990d100 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:17.220746 10041 ts_tablet_manager.cc:1950] T 5054bbc3e27243298b4df6b0a87af8be P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:17.220715 9859 leader_election.cc:304] T 6c61c360653346fabff0718f0990d100 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:17.221041 9859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 102727us
W20251027 21:08:17.221284 9859 leader_election.cc:343] T 19f3e1a9db434c519e26e864eb790706 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:17.221374 9859 leader_election.cc:304] T 19f3e1a9db434c519e26e864eb790706 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:17.221698 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1347) took 2364 ms. Trace:
I20251027 21:08:17.221798 10041 rpcz_store.cc:276] 1027 21:08:14.857144 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:14.857185 (+ 41us) service_pool.cc:225] Handling call
1027 21:08:16.800406 (+1943221us) ts_tablet_manager.cc:1151] Deleting tablet 5054bbc3e27243298b4df6b0a87af8be
1027 21:08:16.800417 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:16.893681 (+ 93264us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:17.061346 (+167665us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:17.221675 (+160329us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1942121}
I20251027 21:08:17.221975 11822 raft_consensus.cc:2749] T 6c61c360653346fabff0718f0990d100 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:17.222116 11822 raft_consensus.cc:2749] T 19f3e1a9db434c519e26e864eb790706 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:17.222239 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 5054bbc3e27243298b4df6b0a87af8be (table default.Table_61bbf6b5c1e740128b80465ef3b9a50e [id=439f48256cbd43f785f57857e83ae25f]) successfully deleted
W20251027 21:08:17.221705 9859 leader_election.cc:343] T c97cba61e08d42a7bdb3f51db0a63ec8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:17.225621 9859 leader_election.cc:304] T c97cba61e08d42a7bdb3f51db0a63ec8 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:17.225965 9859 leader_election.cc:343] T 2f641bc4dbed4c0abd0a55b253352507 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:17.226043 9859 leader_election.cc:304] T 2f641bc4dbed4c0abd0a55b253352507 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:17.226388 9859 leader_election.cc:343] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:17.226468 9859 leader_election.cc:304] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:17.226739 9298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.14.125:46313: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20251027 21:08:17.068220 11657 raft_consensus.cc:493] T e419e588b31d48c8bc19d7948081cc81 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:17.227295 11657 raft_consensus.cc:515] T e419e588b31d48c8bc19d7948081cc81 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:17.227416 11543 raft_consensus.cc:2749] T c97cba61e08d42a7bdb3f51db0a63ec8 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:17.227556 11543 raft_consensus.cc:2749] T 2f641bc4dbed4c0abd0a55b253352507 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:17.227689 11543 raft_consensus.cc:2749] T 4bc5fbdbbea0406a961e1a4df51b8762 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:17.227900 11657 leader_election.cc:290] T e419e588b31d48c8bc19d7948081cc81 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579), 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705)
I20251027 21:08:17.069001 11826 raft_consensus.cc:493] T fb7fdc91156e41de80872b5c941c5d95 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:17.069967 10029 tablet_bootstrap.cc:492] T 618f1657a10947fd98976820564c20b3 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:17.228061 11826 raft_consensus.cc:515] T fb7fdc91156e41de80872b5c941c5d95 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.228140 10029 ts_tablet_manager.cc:1403] T 618f1657a10947fd98976820564c20b3 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.169s user 0.001s sys 0.009s
I20251027 21:08:17.228482 11826 leader_election.cc:290] T fb7fdc91156e41de80872b5c941c5d95 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:17.202771 9633 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6210d3d10fc44b9e9e2b0780883186ce" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:17.228636 9633 raft_consensus.cc:3060] T 6210d3d10fc44b9e9e2b0780883186ce P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:17.228953 10029 raft_consensus.cc:359] T 618f1657a10947fd98976820564c20b3 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.229091 10029 raft_consensus.cc:385] T 618f1657a10947fd98976820564c20b3 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:17.229141 10029 raft_consensus.cc:740] T 618f1657a10947fd98976820564c20b3 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:17.229316 10029 consensus_queue.cc:260] T 618f1657a10947fd98976820564c20b3 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.229806 10029 ts_tablet_manager.cc:1434] T 618f1657a10947fd98976820564c20b3 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:17.230106 10029 tablet_bootstrap.cc:492] T 423047dfbf084bfcb881efeb0352f88d P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:17.231348 9633 raft_consensus.cc:2468] T 6210d3d10fc44b9e9e2b0780883186ce P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:17.232372 10029 tablet_bootstrap.cc:654] T 423047dfbf084bfcb881efeb0352f88d P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:17.234326 9568 leader_election.cc:343] T e419e588b31d48c8bc19d7948081cc81 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: e419e588b31d48c8bc19d7948081cc81
W20251027 21:08:17.234839 9568 leader_election.cc:343] T fb7fdc91156e41de80872b5c941c5d95 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: fb7fdc91156e41de80872b5c941c5d95
I20251027 21:08:17.237255 9710 leader_election.cc:304] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:17.237637 11808 consensus_queue.cc:1048] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 21:08:17.238036 11808 raft_consensus.cc:2804] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:17.238189 11808 raft_consensus.cc:697] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:17.203186 11823 raft_consensus.cc:493] T d1f24d21bb954ac9b62ce458c6a22462 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:17.238440 11808 consensus_queue.cc:237] T 6210d3d10fc44b9e9e2b0780883186ce P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.238611 11823 raft_consensus.cc:515] T d1f24d21bb954ac9b62ce458c6a22462 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.239248 11823 leader_election.cc:290] T d1f24d21bb954ac9b62ce458c6a22462 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:17.239941 9631 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f24d21bb954ac9b62ce458c6a22462" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:17.241204 9860 leader_election.cc:343] T d1f24d21bb954ac9b62ce458c6a22462 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:17.203375 9635 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9c86253ce914916bee6584675a82067" candidate_uuid: "77b0513f17c0472088901a1ce8b08218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14"
I20251027 21:08:17.241446 9635 raft_consensus.cc:3060] T c9c86253ce914916bee6584675a82067 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Advancing to term 1
I20251027 21:08:17.244184 9635 raft_consensus.cc:2468] T c9c86253ce914916bee6584675a82067 P 51902b6b61ff4d5584ff21896e847e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b0513f17c0472088901a1ce8b08218 in term 1.
I20251027 21:08:17.244644 9710 leader_election.cc:304] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218; no voters: 0fbd567f929a4f6d8e4d78feef7ed678
I20251027 21:08:17.245043 11626 raft_consensus.cc:2804] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 21:08:17.245270 11626 raft_consensus.cc:697] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Becoming Leader. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Running, Role: LEADER
I20251027 21:08:17.245599 11626 consensus_queue.cc:237] T c9c86253ce914916bee6584675a82067 P 77b0513f17c0472088901a1ce8b08218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.252102 11797 raft_consensus.cc:2243] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:17.253584 11797 raft_consensus.cc:2272] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:17.258100 11797 ts_tablet_manager.cc:1916] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:17.259152 10032 tablet_bootstrap.cc:492] T 849b3e9fc34045939a7a678fa8269d7f P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:17.259351 10032 ts_tablet_manager.cc:1403] T 849b3e9fc34045939a7a678fa8269d7f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.235s user 0.072s sys 0.043s
W20251027 21:08:17.260203 9566 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 175051us
W20251027 21:08:17.261364 9566 leader_election.cc:343] T 46185d382f0a4dea803aad692ed3c89a P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:17.261575 9566 leader_election.cc:304] T 46185d382f0a4dea803aad692ed3c89a P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:17.261850 11373 raft_consensus.cc:2749] T 46185d382f0a4dea803aad692ed3c89a P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:17.263155 9768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb7fdc91156e41de80872b5c941c5d95" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:17.266866 9767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e419e588b31d48c8bc19d7948081cc81" candidate_uuid: "51902b6b61ff4d5584ff21896e847e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:17.260176 10032 raft_consensus.cc:359] T 849b3e9fc34045939a7a678fa8269d7f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.268113 10032 raft_consensus.cc:385] T 849b3e9fc34045939a7a678fa8269d7f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:17.268184 10032 raft_consensus.cc:740] T 849b3e9fc34045939a7a678fa8269d7f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:17.268422 10032 consensus_queue.cc:260] T 849b3e9fc34045939a7a678fa8269d7f P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.269433 10032 ts_tablet_manager.cc:1434] T 849b3e9fc34045939a7a678fa8269d7f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
W20251027 21:08:17.270895 9711 consensus_peers.cc:597] T cdbde6272af147829ecadd67d634f049 P 77b0513f17c0472088901a1ce8b08218 -> Peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Couldn't send request to peer 0fbd567f929a4f6d8e4d78feef7ed678. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cdbde6272af147829ecadd67d634f049. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:17.271293 10043 log.cc:1199] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-0/wal/wals/b3abc5f78c4e41e29128e2d790319c75
I20251027 21:08:17.271847 10043 ts_tablet_manager.cc:1950] T b3abc5f78c4e41e29128e2d790319c75 P 51902b6b61ff4d5584ff21896e847e14: Deleting consensus metadata
I20251027 21:08:17.273830 10029 tablet_bootstrap.cc:492] T 423047dfbf084bfcb881efeb0352f88d P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:17.273965 10029 ts_tablet_manager.cc:1403] T 423047dfbf084bfcb881efeb0352f88d P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.002s
I20251027 21:08:17.273859 11797 ts_tablet_manager.cc:1929] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 21:08:17.276074 11797 log.cc:1199] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-1/wal/wals/0e4d35f229bc46e1859ded7c8196ab9d
W20251027 21:08:17.277585 9859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.66:38579, user_credentials={real_user=slave}} blocked reactor thread for 50827.4us
W20251027 21:08:17.277813 9859 leader_election.cc:343] T 64e39cc210664c3d83eaf1fc576f83ce P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:17.277956 9859 leader_election.cc:304] T 64e39cc210664c3d83eaf1fc576f83ce P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:17.278214 11543 raft_consensus.cc:2749] T 64e39cc210664c3d83eaf1fc576f83ce P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251027 21:08:17.278587 9859 leader_election.cc:343] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:17.278864 9859 leader_election.cc:343] T 7c15b74f53bc45e1af33b043f2cf225f P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:17.279592 11543 raft_consensus.cc:493] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:17.279697 11543 raft_consensus.cc:515] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.280166 11543 leader_election.cc:290] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:17.283601 9630 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67994a7cacd94b2ebb16fe154832c8fd" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:17.284189 9860 leader_election.cc:343] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:17.284564 10032 tablet_bootstrap.cc:492] T 3e46c62b8a1144998eb34ac839563a9d P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
W20251027 21:08:17.285337 9859 consensus_peers.cc:597] T 93a31187eca14b8797737936a80e7292 P 0fbd567f929a4f6d8e4d78feef7ed678 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251027 21:08:17.286213 9298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.14.125:46313, user_credentials={real_user=slave}} blocked reactor thread for 59477us
W20251027 21:08:17.287071 9566 leader_election.cc:343] T fb7fdc91156e41de80872b5c941c5d95 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:17.287155 9298 consensus_peers.cc:597] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2 -> Peer 18b15927ce5744ad8ba81dda52725c11 (127.9.14.125:46313): Couldn't send request to peer 18b15927ce5744ad8ba81dda52725c11. Status: Network error: Client connection negotiation failed: client connection to 127.9.14.125:46313: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20251027 21:08:17.287253 9566 leader_election.cc:304] T fb7fdc91156e41de80872b5c941c5d95 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:17.287794 9566 leader_election.cc:343] T e419e588b31d48c8bc19d7948081cc81 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:17.287932 9566 leader_election.cc:304] T e419e588b31d48c8bc19d7948081cc81 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51902b6b61ff4d5584ff21896e847e14; no voters: 0fbd567f929a4f6d8e4d78feef7ed678, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:17.288357 11658 raft_consensus.cc:2749] T fb7fdc91156e41de80872b5c941c5d95 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:17.288599 11658 raft_consensus.cc:2749] T e419e588b31d48c8bc19d7948081cc81 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:17.288630 10029 raft_consensus.cc:359] T 423047dfbf084bfcb881efeb0352f88d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.288792 10029 raft_consensus.cc:385] T 423047dfbf084bfcb881efeb0352f88d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:17.288846 10029 raft_consensus.cc:740] T 423047dfbf084bfcb881efeb0352f88d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:17.289011 10029 consensus_queue.cc:260] T 423047dfbf084bfcb881efeb0352f88d P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.289443 10029 ts_tablet_manager.cc:1434] T 423047dfbf084bfcb881efeb0352f88d P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.015s user 0.000s sys 0.001s
I20251027 21:08:17.289698 10029 tablet_bootstrap.cc:492] T fe94124b239e442dbf644e4a483615fc P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:17.291253 10029 tablet_bootstrap.cc:654] T fe94124b239e442dbf644e4a483615fc P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:17.292126 10041 tablet_replica.cc:333] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:17.293476 10041 raft_consensus.cc:2243] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:17.293671 10041 raft_consensus.cc:2272] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:17.294200 10041 ts_tablet_manager.cc:1916] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:17.311316 10032 tablet_bootstrap.cc:654] T 3e46c62b8a1144998eb34ac839563a9d P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:17.311703 10029 tablet_bootstrap.cc:492] T fe94124b239e442dbf644e4a483615fc P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:17.311913 10029 ts_tablet_manager.cc:1403] T fe94124b239e442dbf644e4a483615fc P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20251027 21:08:17.312660 10029 raft_consensus.cc:359] T fe94124b239e442dbf644e4a483615fc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.312881 10029 raft_consensus.cc:385] T fe94124b239e442dbf644e4a483615fc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:17.312961 10029 raft_consensus.cc:740] T fe94124b239e442dbf644e4a483615fc P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:17.313197 10029 consensus_queue.cc:260] T fe94124b239e442dbf644e4a483615fc P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.313759 10029 ts_tablet_manager.cc:1434] T fe94124b239e442dbf644e4a483615fc P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:17.314074 10029 tablet_bootstrap.cc:492] T 4c67c5cb3b244feb8478f9594b834fbf P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:17.315582 10029 tablet_bootstrap.cc:654] T 4c67c5cb3b244feb8478f9594b834fbf P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
W20251027 21:08:17.322644 9566 consensus_peers.cc:597] T d0c53b7b32c84a51a967e2a6a22258c6 P 51902b6b61ff4d5584ff21896e847e14 -> Peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Couldn't send request to peer 77b0513f17c0472088901a1ce8b08218. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 21:08:17.324672 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1163) took 7568 ms. Trace:
I20251027 21:08:17.324786 10043 rpcz_store.cc:276] 1027 21:08:09.756156 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.758085 (+ 1929us) service_pool.cc:225] Handling call
1027 21:08:16.895291 (+7137206us) ts_tablet_manager.cc:1151] Deleting tablet b3abc5f78c4e41e29128e2d790319c75
1027 21:08:16.895304 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:16.999014 (+103710us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:17.064846 (+ 65832us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:17.324650 (+259804us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24980,"tablet-delete.queue_time_us":7128914}
I20251027 21:08:17.326706 10041 ts_tablet_manager.cc:1929] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:17.326884 10041 log.cc:1199] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/fe7b00e9cf984d12817c9ea860391199
I20251027 21:08:17.327217 10041 ts_tablet_manager.cc:1950] T fe7b00e9cf984d12817c9ea860391199 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:17.331692 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1345) took 2474 ms. Trace:
I20251027 21:08:17.332453 10041 rpcz_store.cc:276] 1027 21:08:14.856927 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:14.857313 (+ 386us) service_pool.cc:225] Handling call
1027 21:08:17.272482 (+2415169us) ts_tablet_manager.cc:1151] Deleting tablet fe7b00e9cf984d12817c9ea860391199
1027 21:08:17.292081 (+ 19599us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:17.305107 (+ 13026us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:17.326603 (+ 21496us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:17.331653 (+ 5050us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2414596}
I20251027 21:08:17.334216 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet fe7b00e9cf984d12817c9ea860391199 (table default.Table_4567c3392eb24748840827eb96835642 [id=9192a31ca6324aa2a919311093d54ae0]) successfully deleted
I20251027 21:08:17.334774 10032 tablet_bootstrap.cc:492] T 3e46c62b8a1144998eb34ac839563a9d P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:17.334899 10032 ts_tablet_manager.cc:1403] T 3e46c62b8a1144998eb34ac839563a9d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.006s
I20251027 21:08:17.335162 10041 tablet_replica.cc:333] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:17.335362 10041 raft_consensus.cc:2243] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:17.335497 10041 raft_consensus.cc:2272] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:17.335945 10041 ts_tablet_manager.cc:1916] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:17.338404 9308 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 52991b0eefbb4d1297266a95c17a4662
W20251027 21:08:17.340034 9304 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3604203c122a4df291ecdc1d8477dca0 consensus state has no leader
I20251027 21:08:17.341073 9311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet 21ead81fd55d49cda3f031cdf8a47a8c
I20251027 21:08:17.348541 10032 raft_consensus.cc:359] T 3e46c62b8a1144998eb34ac839563a9d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.348745 10032 raft_consensus.cc:385] T 3e46c62b8a1144998eb34ac839563a9d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:17.348799 10032 raft_consensus.cc:740] T 3e46c62b8a1144998eb34ac839563a9d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:17.349009 10032 consensus_queue.cc:260] T 3e46c62b8a1144998eb34ac839563a9d P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.349566 10032 ts_tablet_manager.cc:1434] T 3e46c62b8a1144998eb34ac839563a9d P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.015s user 0.000s sys 0.000s
I20251027 21:08:17.349910 10032 tablet_bootstrap.cc:492] T 6f901626832f4adb948ccb65f8573e86 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:17.351352 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1153) took 7655 ms. Trace:
I20251027 21:08:17.351477 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 52991b0eefbb4d1297266a95c17a4662 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:17.351794 9298 catalog_manager.cc:4985] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): tablet b3abc5f78c4e41e29128e2d790319c75 (table default.Table_3cb6bc9f33ea40aa9d58e82f709c64b7 [id=b9930c5b513749ac905895ebca813733]) successfully deleted
I20251027 21:08:17.281330 9768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67994a7cacd94b2ebb16fe154832c8fd" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:17.354663 9859 leader_election.cc:343] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:17.354768 9859 leader_election.cc:304] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:17.342564 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet 21ead81fd55d49cda3f031cdf8a47a8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:17.354969 11814 raft_consensus.cc:2749] T 67994a7cacd94b2ebb16fe154832c8fd P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:17.286162 10031 tablet_bootstrap.cc:492] T 00c90ff84d3f4a7f92fda235416eed1d P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:17.342612 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet 52991b0eefbb4d1297266a95c17a4662 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:17.355265 10031 ts_tablet_manager.cc:1403] T 00c90ff84d3f4a7f92fda235416eed1d P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.655s user 0.011s sys 0.002s
I20251027 21:08:17.283624 9767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1f24d21bb954ac9b62ce458c6a22462" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:17.276542 11797 ts_tablet_manager.cc:1950] T 0e4d35f229bc46e1859ded7c8196ab9d P 77b0513f17c0472088901a1ce8b08218: Deleting consensus metadata
I20251027 21:08:17.345371 9607 tablet_service.cc:1552] Processing DeleteTablet for tablet 52991b0eefbb4d1297266a95c17a4662 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:17.345513 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet 21ead81fd55d49cda3f031cdf8a47a8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:17.332376 10029 tablet_bootstrap.cc:492] T 4c67c5cb3b244feb8478f9594b834fbf P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:17.355978 10031 raft_consensus.cc:359] T 00c90ff84d3f4a7f92fda235416eed1d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.356127 10031 raft_consensus.cc:385] T 00c90ff84d3f4a7f92fda235416eed1d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:17.356173 10031 raft_consensus.cc:740] T 00c90ff84d3f4a7f92fda235416eed1d P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:17.356372 10031 consensus_queue.cc:260] T 00c90ff84d3f4a7f92fda235416eed1d P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.356922 10031 ts_tablet_manager.cc:1434] T 00c90ff84d3f4a7f92fda235416eed1d P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:17.357244 10031 tablet_bootstrap.cc:492] T e1e58f6228254e3eb25dbf627a505ca1 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:17.356015 10029 ts_tablet_manager.cc:1403] T 4c67c5cb3b244feb8478f9594b834fbf P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.001s
I20251027 21:08:17.358781 10029 raft_consensus.cc:359] T 4c67c5cb3b244feb8478f9594b834fbf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.358938 10029 raft_consensus.cc:385] T 4c67c5cb3b244feb8478f9594b834fbf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:17.358992 10029 raft_consensus.cc:740] T 4c67c5cb3b244feb8478f9594b834fbf P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:17.359164 10029 consensus_queue.cc:260] T 4c67c5cb3b244feb8478f9594b834fbf P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:17.359514 9859 leader_election.cc:343] T d1f24d21bb954ac9b62ce458c6a22462 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:17.359551 9297 catalog_manager.cc:4985] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): tablet 0e4d35f229bc46e1859ded7c8196ab9d (table default.Table_00deabd570914e54b7fa9c4fc4eb02c7 [id=ba8a1f1eb0d445cbaf7f263429085ee6]) successfully deleted
I20251027 21:08:17.359591 10029 ts_tablet_manager.cc:1434] T 4c67c5cb3b244feb8478f9594b834fbf P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:17.359628 9859 leader_election.cc:304] T d1f24d21bb954ac9b62ce458c6a22462 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
W20251027 21:08:17.359771 9297 catalog_manager.cc:4965] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 7e32e33235b841aeb2a1eabf25bd9ff3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e32e33235b841aeb2a1eabf25bd9ff3 already in progress: creating tablet
I20251027 21:08:17.359824 10029 tablet_bootstrap.cc:492] T 815e12edcd3847ff8485ba768291a824 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
W20251027 21:08:17.359957 9297 catalog_manager.cc:4952] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): delete failed for tablet 67cb71ce43244d8ab9e71ead6bb315c9 because the tablet was not found. No further retry: Not found: Tablet not found: 67cb71ce43244d8ab9e71ead6bb315c9
I20251027 21:08:17.360136 11814 raft_consensus.cc:2749] T d1f24d21bb954ac9b62ce458c6a22462 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:17.361433 10029 tablet_bootstrap.cc:654] T 815e12edcd3847ff8485ba768291a824 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:17.355736 10043 rpcz_store.cc:276] 1027 21:08:09.695881 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:09.720329 (+ 24448us) service_pool.cc:225] Handling call
1027 21:08:17.351217 (+7630888us) ts_tablet_manager.cc:1151] Deleting tablet 2a9c6bf4f75144f487305520b3714b84
1027 21:08:17.351238 (+ 21us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:17.351334 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7521269}
I20251027 21:08:17.365095 10032 tablet_bootstrap.cc:654] T 6f901626832f4adb948ccb65f8573e86 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:17.365525 10043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59712 (request call id 1181) took 6933 ms. Trace:
W20251027 21:08:17.366014 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet 2a9c6bf4f75144f487305520b3714b84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a9c6bf4f75144f487305520b3714b84 already in progress: creating tablet
I20251027 21:08:17.367614 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet 21ead81fd55d49cda3f031cdf8a47a8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:16 UTC) from {username='slave'} at 127.0.0.1:34644
I20251027 21:08:17.368108 10043 rpcz_store.cc:276] 1027 21:08:10.432388 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:10.432425 (+ 37us) service_pool.cc:225] Handling call
1027 21:08:17.365419 (+6932994us) ts_tablet_manager.cc:1151] Deleting tablet befdd3a485a1426b8a3adf340d4eae4f
1027 21:08:17.365428 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:17.365509 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6932827}
I20251027 21:08:17.358894 11797 tablet_replica.cc:333] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218: stopping tablet replica
I20251027 21:08:17.361451 10031 tablet_bootstrap.cc:654] T e1e58f6228254e3eb25dbf627a505ca1 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:17.369139 11797 raft_consensus.cc:2243] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: Raft consensus shutting down.
I20251027 21:08:17.370570 11797 pending_rounds.cc:70] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218: Trying to abort 1 pending ops.
W20251027 21:08:17.370752 9298 catalog_manager.cc:4965] TS 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): delete failed for tablet befdd3a485a1426b8a3adf340d4eae4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet befdd3a485a1426b8a3adf340d4eae4f already in progress: creating tablet
I20251027 21:08:17.372876 10043 tablet_replica.cc:333] T 423047dfbf084bfcb881efeb0352f88d P 51902b6b61ff4d5584ff21896e847e14: stopping tablet replica
I20251027 21:08:17.373108 10043 raft_consensus.cc:2243] T 423047dfbf084bfcb881efeb0352f88d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:17.373301 10043 raft_consensus.cc:2272] T 423047dfbf084bfcb881efeb0352f88d P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:17.373448 11797 pending_rounds.cc:77] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215510718271029248 op_type: NO_OP noop_request { }
I20251027 21:08:17.374274 11814 raft_consensus.cc:493] T 3e46c62b8a1144998eb34ac839563a9d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:17.374392 11814 raft_consensus.cc:515] T 3e46c62b8a1144998eb34ac839563a9d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.375010 11814 leader_election.cc:290] T 3e46c62b8a1144998eb34ac839563a9d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:17.375975 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e46c62b8a1144998eb34ac839563a9d" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:17.376528 9860 leader_election.cc:343] T 3e46c62b8a1144998eb34ac839563a9d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
W20251027 21:08:17.376817 9566 raft_consensus.cc:687] T 423047dfbf084bfcb881efeb0352f88d P 51902b6b61ff4d5584ff21896e847e14: failed to submit failure detected task: Service unavailable: Thread pool token was shut down
I20251027 21:08:17.375981 9767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e46c62b8a1144998eb34ac839563a9d" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
W20251027 21:08:17.377327 9859 leader_election.cc:343] T 3e46c62b8a1144998eb34ac839563a9d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:17.377441 9859 leader_election.cc:304] T 3e46c62b8a1144998eb34ac839563a9d P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:17.377841 11814 raft_consensus.cc:2749] T 3e46c62b8a1144998eb34ac839563a9d P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:17.378049 11797 raft_consensus.cc:2889] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 21:08:17.380791 10043 ts_tablet_manager.cc:1916] T 423047dfbf084bfcb881efeb0352f88d P 51902b6b61ff4d5584ff21896e847e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:17.381022 11797 raft_consensus.cc:2272] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:17.382020 11797 ts_tablet_manager.cc:1916] T 27bd3b2a011648b2b8f2f92944513751 P 77b0513f17c0472088901a1ce8b08218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:17.385876 9312 master_service.cc:954] ReplaceTablet: received request to replace tablet 3ee02a05b5ae46e7bf5cf9fdd3e9a80f from {username='slave'} at 127.0.0.1:41710
I20251027 21:08:17.386816 9304 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b42751988b8e445b9193301e2ef472e7" } modify_external_catalogs: true
I20251027 21:08:17.387001 9304 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_b42751988b8e445b9193301e2ef472e7" } modify_external_catalogs: true
I20251027 21:08:17.386838 9306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_eba2fb1dc64d474696571d2b880fea56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:17.387665 9306 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eba2fb1dc64d474696571d2b880fea56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:17.388824 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_3f998bc22cc84917b46975ebd9a461e0" } new_table_name: "default.Table_45826b2a85f5498a858c11ed8e97048f" modify_external_catalogs: true
I20251027 21:08:17.390093 9308 catalog_manager.cc:5741] TS 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579) does not have the latest schema for tablet 371051fb3491426e8cf7797317baa3db (table default.Table_d481a540123e4620874b629ce6db65a7 [id=3b0044836ccc4b4082775de6eb111e49]). Expected version 1 got 0
I20251027 21:08:17.391397 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_d711a2320691436c9c978f992df38146"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:17.391832 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d711a2320691436c9c978f992df38146 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:17.393890 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_544caae6c7374d3eaf05ad3616359d12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:17.394553 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_544caae6c7374d3eaf05ad3616359d12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:17.394987 11814 raft_consensus.cc:493] T da23505b0fe742ddadfb829e37ca5499 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:17.395153 11814 raft_consensus.cc:515] T da23505b0fe742ddadfb829e37ca5499 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.395694 11814 leader_election.cc:290] T da23505b0fe742ddadfb829e37ca5499 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:17.386857 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet e419e588b31d48c8bc19d7948081cc81 with delete_type TABLET_DATA_DELETED (replaced by tablet cb5ac0fff70b491e8c8842f7a331005e) from {username='slave'} at 127.0.0.1:59712
I20251027 21:08:17.396397 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da23505b0fe742ddadfb829e37ca5499" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:17.413067 9860 leader_election.cc:343] T da23505b0fe742ddadfb829e37ca5499 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:17.413417 11814 raft_consensus.cc:493] T 4d88aacbf47144cb837b99d68f6331e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:17.413542 11814 raft_consensus.cc:515] T 4d88aacbf47144cb837b99d68f6331e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:17.414105 11814 leader_election.cc:290] T 4d88aacbf47144cb837b99d68f6331e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:17.414911 9634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d88aacbf47144cb837b99d68f6331e2" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51902b6b61ff4d5584ff21896e847e14" is_pre_election: true
W20251027 21:08:17.415437 9860 leader_election.cc:343] T 4d88aacbf47144cb837b99d68f6331e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51902b6b61ff4d5584ff21896e847e14 (127.9.14.65:44531): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:17.424238 9304 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a4b8b31d47da48aaa386a8e64a5749c2: Sending DeleteTablet for 3 replicas of tablet e22e1e3fb85b4a4a8d740c8ea5285aec
I20251027 21:08:17.425004 9304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_41d0d8ecfea543c6adcdc705ffea88a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:17.425091 9604 tablet_service.cc:1552] Processing DeleteTablet for tablet e22e1e3fb85b4a4a8d740c8ea5285aec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:17 UTC) from {username='slave'} at 127.0.0.1:59712
W20251027 21:08:17.425545 9304 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41d0d8ecfea543c6adcdc705ffea88a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:17.415970 9739 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e64d21abef342de86555e876f020787 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:09 UTC) from {username='slave'} at 127.0.0.1:40732
I20251027 21:08:17.416810 9768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da23505b0fe742ddadfb829e37ca5499" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:17.427332 9895 tablet_service.cc:1552] Processing DeleteTablet for tablet e22e1e3fb85b4a4a8d740c8ea5285aec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:17 UTC) from {username='slave'} at 127.0.0.1:34644
W20251027 21:08:17.427733 9859 leader_election.cc:343] T da23505b0fe742ddadfb829e37ca5499 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:17.427848 9859 leader_election.cc:304] T da23505b0fe742ddadfb829e37ca5499 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:17.428582 11814 raft_consensus.cc:2749] T da23505b0fe742ddadfb829e37ca5499 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:17.418522 9767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d88aacbf47144cb837b99d68f6331e2" candidate_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b0513f17c0472088901a1ce8b08218" is_pre_election: true
I20251027 21:08:17.427186 9749 tablet_service.cc:1552] Processing DeleteTablet for tablet e22e1e3fb85b4a4a8d740c8ea5285aec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 21:08:17 UTC) from {username='slave'} at 127.0.0.1:40732
W20251027 21:08:17.430078 9859 leader_election.cc:343] T 4d88aacbf47144cb837b99d68f6331e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579): Illegal state: must be running to vote when last-logged opid is not known
I20251027 21:08:17.430197 9859 leader_election.cc:304] T 4d88aacbf47144cb837b99d68f6331e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fbd567f929a4f6d8e4d78feef7ed678; no voters: 51902b6b61ff4d5584ff21896e847e14, 77b0513f17c0472088901a1ce8b08218
I20251027 21:08:17.430699 11823 raft_consensus.cc:2749] T 4d88aacbf47144cb837b99d68f6331e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 21:08:17.438980 10031 tablet_bootstrap.cc:492] T e1e58f6228254e3eb25dbf627a505ca1 P 77b0513f17c0472088901a1ce8b08218: No bootstrap required, opened a new log
I20251027 21:08:17.439141 10031 ts_tablet_manager.cc:1403] T e1e58f6228254e3eb25dbf627a505ca1 P 77b0513f17c0472088901a1ce8b08218: Time spent bootstrapping tablet: real 0.082s user 0.006s sys 0.008s
I20251027 21:08:17.439862 10031 raft_consensus.cc:359] T e1e58f6228254e3eb25dbf627a505ca1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.440022 10031 raft_consensus.cc:385] T e1e58f6228254e3eb25dbf627a505ca1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:17.440071 10031 raft_consensus.cc:740] T e1e58f6228254e3eb25dbf627a505ca1 P 77b0513f17c0472088901a1ce8b08218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b0513f17c0472088901a1ce8b08218, State: Initialized, Role: FOLLOWER
I20251027 21:08:17.440296 10031 consensus_queue.cc:260] T e1e58f6228254e3eb25dbf627a505ca1 P 77b0513f17c0472088901a1ce8b08218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.443624 10031 ts_tablet_manager.cc:1434] T e1e58f6228254e3eb25dbf627a505ca1 P 77b0513f17c0472088901a1ce8b08218: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251027 21:08:17.444274 10031 tablet_bootstrap.cc:492] T 76bee322d0b94abe9166170733608853 P 77b0513f17c0472088901a1ce8b08218: Bootstrap starting.
I20251027 21:08:17.457788 10031 tablet_bootstrap.cc:654] T 76bee322d0b94abe9166170733608853 P 77b0513f17c0472088901a1ce8b08218: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:17.458796 9306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_92fa3ef69db74af5964d6b6f170b03c5" } new_table_name: "default.Table_38b8500a39ca4a0e84786e0196d27e17" modify_external_catalogs: true
I20251027 21:08:17.458920 10032 tablet_bootstrap.cc:492] T 6f901626832f4adb948ccb65f8573e86 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:17.459044 10032 ts_tablet_manager.cc:1403] T 6f901626832f4adb948ccb65f8573e86 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.109s user 0.013s sys 0.009s
I20251027 21:08:17.459429 10041 ts_tablet_manager.cc:1929] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:17.459522 10041 log.cc:1199] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/479d2cc8b8ce4c26841f50eea7c0e09a
I20251027 21:08:17.459789 10041 ts_tablet_manager.cc:1950] T 479d2cc8b8ce4c26841f50eea7c0e09a P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:17.459739 10032 raft_consensus.cc:359] T 6f901626832f4adb948ccb65f8573e86 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:17.459882 10032 raft_consensus.cc:385] T 6f901626832f4adb948ccb65f8573e86 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:17.459931 10032 raft_consensus.cc:740] T 6f901626832f4adb948ccb65f8573e86 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:17.460276 10032 consensus_queue.cc:260] T 6f901626832f4adb948ccb65f8573e86 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:17.461493 10032 ts_tablet_manager.cc:1434] T 6f901626832f4adb948ccb65f8573e86 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 21:08:17.461753 10032 tablet_bootstrap.cc:492] T bbf92e29993b46389a69efd9e9a2341f P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:17.463496 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1348) took 2606 ms. Trace:
W20251027 21:08:17.463923 9305 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2e5b6912c484517afd48f8070455da8 consensus state has no leader
I20251027 21:08:17.463786 10032 tablet_bootstrap.cc:654] T bbf92e29993b46389a69efd9e9a2341f P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:17.464272 10041 rpcz_store.cc:276] 1027 21:08:14.857219 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:14.857385 (+ 166us) service_pool.cc:225] Handling call
1027 21:08:17.335114 (+2477729us) ts_tablet_manager.cc:1151] Deleting tablet 479d2cc8b8ce4c26841f50eea7c0e09a
1027 21:08:17.335127 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:17.371250 (+ 36123us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:17.459327 (+ 88077us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:17.463478 (+ 4151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2477137}
I20251027 21:08:17.464843 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 479d2cc8b8ce4c26841f50eea7c0e09a (table default.Table_9601abc37cc5482aa4b9a13b818e9074 [id=9a181744a9954b8388e6417fa7f8a630]) successfully deleted
I20251027 21:08:17.464970 10041 tablet_replica.cc:333] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:17.465214 10041 raft_consensus.cc:2243] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:17.465890 10041 raft_consensus.cc:2272] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:17.466718 10041 ts_tablet_manager.cc:1916] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 21:08:17.475199 9299 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 3604203c122a4df291ecdc1d8477dca0 (table default.Table_91a95ff1364546eea4708137b670aee7 [id=ba4c471be2664b04b733d23b0481a34e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3604203c122a4df291ecdc1d8477dca0 consensus state has no leader
W20251027 21:08:17.476348 9973 tablet.cc:2378] T 3e46c62b8a1144998eb34ac839563a9d P 0fbd567f929a4f6d8e4d78feef7ed678: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 21:08:17.483201 10041 ts_tablet_manager.cc:1929] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:17.487416 10041 log.cc:1199] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/c3f4e692c08244afa2c79e0a075bbcdb
I20251027 21:08:17.487941 10041 ts_tablet_manager.cc:1950] T c3f4e692c08244afa2c79e0a075bbcdb P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:17.488782 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1349) took 2603 ms. Trace:
I20251027 21:08:17.492738 11658 raft_consensus.cc:493] T 532ea47b15b544dfb05ddb2bdacf3953 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 21:08:17.492875 11658 raft_consensus.cc:515] T 532ea47b15b544dfb05ddb2bdacf3953 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.489102 10032 tablet_bootstrap.cc:492] T bbf92e29993b46389a69efd9e9a2341f P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:17.493607 10032 ts_tablet_manager.cc:1403] T bbf92e29993b46389a69efd9e9a2341f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.007s
I20251027 21:08:17.493618 11658 leader_election.cc:290] T 532ea47b15b544dfb05ddb2bdacf3953 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705), 77b0513f17c0472088901a1ce8b08218 (127.9.14.66:38579)
I20251027 21:08:17.494278 10032 raft_consensus.cc:359] T bbf92e29993b46389a69efd9e9a2341f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
W20251027 21:08:17.494382 9568 leader_election.cc:343] T 532ea47b15b544dfb05ddb2bdacf3953 P 51902b6b61ff4d5584ff21896e847e14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): Not found: Tablet not found: 532ea47b15b544dfb05ddb2bdacf3953
I20251027 21:08:17.494464 10032 raft_consensus.cc:385] T bbf92e29993b46389a69efd9e9a2341f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:17.494532 10032 raft_consensus.cc:740] T bbf92e29993b46389a69efd9e9a2341f P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:17.494730 10032 consensus_queue.cc:260] T bbf92e29993b46389a69efd9e9a2341f P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.495071 10029 tablet_bootstrap.cc:492] T 815e12edcd3847ff8485ba768291a824 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:17.495203 10029 ts_tablet_manager.cc:1403] T 815e12edcd3847ff8485ba768291a824 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.135s user 0.016s sys 0.018s
I20251027 21:08:17.495224 10032 ts_tablet_manager.cc:1434] T bbf92e29993b46389a69efd9e9a2341f P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 21:08:17.495518 10032 tablet_bootstrap.cc:492] T 4bbb69d62b9a41e3b88a5527f7549881 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:17.495937 10029 raft_consensus.cc:359] T 815e12edcd3847ff8485ba768291a824 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:17.496093 10029 raft_consensus.cc:385] T 815e12edcd3847ff8485ba768291a824 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:17.496153 10029 raft_consensus.cc:740] T 815e12edcd3847ff8485ba768291a824 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:17.496345 10029 consensus_queue.cc:260] T 815e12edcd3847ff8485ba768291a824 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:17.496819 10029 ts_tablet_manager.cc:1434] T 815e12edcd3847ff8485ba768291a824 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:17.496850 10032 tablet_bootstrap.cc:654] T 4bbb69d62b9a41e3b88a5527f7549881 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:17.497090 10029 tablet_bootstrap.cc:492] T a7d8d060916941a5b7e944e8b08a2882 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:17.500136 10029 tablet_bootstrap.cc:654] T a7d8d060916941a5b7e944e8b08a2882 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:17.500732 9305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41710:
table { table_name: "default.Table_c9d359a62fe14203ae1d3b96f4b1bbf1" } new_table_name: "default.Table_670dc34288d5406383e567f2d592ca19" modify_external_catalogs: true
I20251027 21:08:17.500820 10041 rpcz_store.cc:276] 1027 21:08:14.885324 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:14.885431 (+ 107us) service_pool.cc:225] Handling call
1027 21:08:17.464911 (+2579480us) ts_tablet_manager.cc:1151] Deleting tablet c3f4e692c08244afa2c79e0a075bbcdb
1027 21:08:17.464923 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:17.479703 (+ 14780us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:17.483117 (+ 3414us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:17.488763 (+ 5646us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2579318}
I20251027 21:08:17.501488 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1351) took 2552 ms. Trace:
I20251027 21:08:17.501555 10041 rpcz_store.cc:276] 1027 21:08:14.949301 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:14.949376 (+ 75us) service_pool.cc:225] Handling call
1027 21:08:17.501374 (+2551998us) ts_tablet_manager.cc:1151] Deleting tablet 107c608f9d904d86b74e2ad682e1bdc7
1027 21:08:17.501383 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:17.501472 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2551842}
I20251027 21:08:17.501992 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet c3f4e692c08244afa2c79e0a075bbcdb (table default.Table_dd85b9fbcede45888d66f6226d2c5f2f [id=5525bf5e4c134750bdc1edf8d6b73969]) successfully deleted
W20251027 21:08:17.502295 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 107c608f9d904d86b74e2ad682e1bdc7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 107c608f9d904d86b74e2ad682e1bdc7 already in progress: creating tablet
I20251027 21:08:17.502333 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1353) took 2540 ms. Trace:
I20251027 21:08:17.502395 10041 rpcz_store.cc:276] 1027 21:08:14.961601 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:14.961658 (+ 57us) service_pool.cc:225] Handling call
1027 21:08:17.502244 (+2540586us) ts_tablet_manager.cc:1151] Deleting tablet 2044c73d56bf4434879dd65f35c5612b
1027 21:08:17.502252 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:17.502320 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2540398}
W20251027 21:08:17.502961 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 2044c73d56bf4434879dd65f35c5612b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2044c73d56bf4434879dd65f35c5612b already in progress: creating tablet
I20251027 21:08:17.503242 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1357) took 2463 ms. Trace:
I20251027 21:08:17.503564 10041 rpcz_store.cc:276] 1027 21:08:15.039518 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:15.039571 (+ 53us) service_pool.cc:225] Handling call
1027 21:08:17.503146 (+2463575us) ts_tablet_manager.cc:1151] Deleting tablet 218d244dc8a549408863110fefbc7b20
1027 21:08:17.503154 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:17.503228 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2463356}
I20251027 21:08:17.504305 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1360) took 2368 ms. Trace:
W20251027 21:08:17.504531 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 218d244dc8a549408863110fefbc7b20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 218d244dc8a549408863110fefbc7b20 already in progress: creating tablet
I20251027 21:08:17.504639 10041 rpcz_store.cc:276] 1027 21:08:15.135973 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:15.136029 (+ 56us) service_pool.cc:225] Handling call
1027 21:08:17.504208 (+2368179us) ts_tablet_manager.cc:1151] Deleting tablet afff93eb06ab4bcca2990c8bc534b234
1027 21:08:17.504227 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:17.504292 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2368001}
I20251027 21:08:17.505475 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1362) took 2369 ms. Trace:
W20251027 21:08:17.505584 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet afff93eb06ab4bcca2990c8bc534b234 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afff93eb06ab4bcca2990c8bc534b234 already in progress: creating tablet
I20251027 21:08:17.505890 10041 rpcz_store.cc:276] 1027 21:08:15.136174 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:15.136248 (+ 74us) service_pool.cc:225] Handling call
1027 21:08:17.505361 (+2369113us) ts_tablet_manager.cc:1151] Deleting tablet cf1751af4c774d078a4aede9d2bb8108
1027 21:08:17.505372 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:17.505462 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2368954}
W20251027 21:08:17.506533 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet cf1751af4c774d078a4aede9d2bb8108 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf1751af4c774d078a4aede9d2bb8108 already in progress: creating tablet
I20251027 21:08:17.506662 10041 tablet_replica.cc:333] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:17.507164 10041 raft_consensus.cc:2243] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:17.507836 10041 raft_consensus.cc:2272] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:17.508726 10041 ts_tablet_manager.cc:1916] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:17.511840 10029 tablet_bootstrap.cc:492] T a7d8d060916941a5b7e944e8b08a2882 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:17.511981 10029 ts_tablet_manager.cc:1403] T a7d8d060916941a5b7e944e8b08a2882 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20251027 21:08:17.512646 10029 raft_consensus.cc:359] T a7d8d060916941a5b7e944e8b08a2882 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:17.512799 10029 raft_consensus.cc:385] T a7d8d060916941a5b7e944e8b08a2882 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:17.512843 10029 raft_consensus.cc:740] T a7d8d060916941a5b7e944e8b08a2882 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:17.513020 10029 consensus_queue.cc:260] T a7d8d060916941a5b7e944e8b08a2882 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } }
I20251027 21:08:17.513454 10029 ts_tablet_manager.cc:1434] T a7d8d060916941a5b7e944e8b08a2882 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:17.513720 10029 tablet_bootstrap.cc:492] T e04f4c5cf724419ab0be0e84d30a9a57 P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:17.515942 10029 tablet_bootstrap.cc:654] T e04f4c5cf724419ab0be0e84d30a9a57 P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:17.521581 10041 ts_tablet_manager.cc:1929] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:17.528401 10041 log.cc:1199] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/874d5974698c448ab79f19812cb0e886
I20251027 21:08:17.528759 10041 ts_tablet_manager.cc:1950] T 874d5974698c448ab79f19812cb0e886 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
I20251027 21:08:17.529484 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1364) took 2389 ms. Trace:
I20251027 21:08:17.529616 10041 rpcz_store.cc:276] 1027 21:08:15.140405 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:15.142640 (+ 2235us) service_pool.cc:225] Handling call
1027 21:08:17.506606 (+2363966us) ts_tablet_manager.cc:1151] Deleting tablet 874d5974698c448ab79f19812cb0e886
1027 21:08:17.506618 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:17.517988 (+ 11370us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:17.521502 (+ 3514us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:17.529463 (+ 7961us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2363801}
I20251027 21:08:17.525219 10032 tablet_bootstrap.cc:492] T 4bbb69d62b9a41e3b88a5527f7549881 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:17.530123 10032 ts_tablet_manager.cc:1403] T 4bbb69d62b9a41e3b88a5527f7549881 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.009s
I20251027 21:08:17.530095 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 874d5974698c448ab79f19812cb0e886 (table default.Table_c90c85540a2142218c304c8da1a2df90 [id=1ed4b100af1d4ccdb13d108ba63cbd44]) successfully deleted
I20251027 21:08:17.530771 10032 raft_consensus.cc:359] T 4bbb69d62b9a41e3b88a5527f7549881 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.530912 10032 raft_consensus.cc:385] T 4bbb69d62b9a41e3b88a5527f7549881 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:17.530953 10032 raft_consensus.cc:740] T 4bbb69d62b9a41e3b88a5527f7549881 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:17.531121 10032 consensus_queue.cc:260] T 4bbb69d62b9a41e3b88a5527f7549881 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.531548 10032 ts_tablet_manager.cc:1434] T 4bbb69d62b9a41e3b88a5527f7549881 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 21:08:17.531800 10032 tablet_bootstrap.cc:492] T 123e6e3748574ea581f67fff49037762 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:17.532999 10032 tablet_bootstrap.cc:654] T 123e6e3748574ea581f67fff49037762 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:17.533720 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1361) took 2397 ms. Trace:
I20251027 21:08:17.533792 10041 rpcz_store.cc:276] 1027 21:08:15.136087 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:15.161393 (+ 25306us) service_pool.cc:225] Handling call
1027 21:08:17.533606 (+2372213us) ts_tablet_manager.cc:1151] Deleting tablet 5e20fb964a4543ff9eda684acc8eb2bb
1027 21:08:17.533614 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:17.533703 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2372038}
W20251027 21:08:17.534190 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 5e20fb964a4543ff9eda684acc8eb2bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e20fb964a4543ff9eda684acc8eb2bb already in progress: creating tablet
I20251027 21:08:17.534205 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1373) took 2195 ms. Trace:
I20251027 21:08:17.534475 10041 rpcz_store.cc:276] 1027 21:08:15.338477 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:15.338797 (+ 320us) service_pool.cc:225] Handling call
1027 21:08:17.534102 (+2195305us) ts_tablet_manager.cc:1151] Deleting tablet 3ef2fff1381b465a89eeb5f82ae4fd12
1027 21:08:17.534110 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:17.534191 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2195121}
I20251027 21:08:17.534866 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1374) took 2183 ms. Trace:
I20251027 21:08:17.534890 10029 tablet_bootstrap.cc:492] T e04f4c5cf724419ab0be0e84d30a9a57 P 51902b6b61ff4d5584ff21896e847e14: No bootstrap required, opened a new log
I20251027 21:08:17.535006 10029 ts_tablet_manager.cc:1403] T e04f4c5cf724419ab0be0e84d30a9a57 P 51902b6b61ff4d5584ff21896e847e14: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20251027 21:08:17.534937 10041 rpcz_store.cc:276] 1027 21:08:15.351857 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:15.351904 (+ 47us) service_pool.cc:225] Handling call
1027 21:08:17.534783 (+2182879us) ts_tablet_manager.cc:1151] Deleting tablet d332d8e8f0c949589d1fd29b4bd7f076
1027 21:08:17.534790 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:17.534853 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2182673}
W20251027 21:08:17.535408 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 3ef2fff1381b465a89eeb5f82ae4fd12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ef2fff1381b465a89eeb5f82ae4fd12 already in progress: creating tablet
W20251027 21:08:17.535593 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet d332d8e8f0c949589d1fd29b4bd7f076 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d332d8e8f0c949589d1fd29b4bd7f076 already in progress: creating tablet
I20251027 21:08:17.535851 10029 raft_consensus.cc:359] T e04f4c5cf724419ab0be0e84d30a9a57 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.536046 10029 raft_consensus.cc:385] T e04f4c5cf724419ab0be0e84d30a9a57 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:17.536103 10029 raft_consensus.cc:740] T e04f4c5cf724419ab0be0e84d30a9a57 P 51902b6b61ff4d5584ff21896e847e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51902b6b61ff4d5584ff21896e847e14, State: Initialized, Role: FOLLOWER
I20251027 21:08:17.536298 10029 consensus_queue.cc:260] T e04f4c5cf724419ab0be0e84d30a9a57 P 51902b6b61ff4d5584ff21896e847e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.536845 10029 ts_tablet_manager.cc:1434] T e04f4c5cf724419ab0be0e84d30a9a57 P 51902b6b61ff4d5584ff21896e847e14: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 21:08:17.537168 10029 tablet_bootstrap.cc:492] T 0820648719a34f379507efced9cc434b P 51902b6b61ff4d5584ff21896e847e14: Bootstrap starting.
I20251027 21:08:17.538357 10041 tablet_replica.cc:333] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:17.538604 10041 raft_consensus.cc:2243] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:17.539338 10029 tablet_bootstrap.cc:654] T 0820648719a34f379507efced9cc434b P 51902b6b61ff4d5584ff21896e847e14: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:17.538780 10041 raft_consensus.cc:2272] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251027 21:08:17.540210 10041 ts_tablet_manager.cc:1916] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:17.545281 9311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_eddb864df5194a76b1edb583eb033fb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 21:08:17.545571 10032 tablet_bootstrap.cc:492] T 123e6e3748574ea581f67fff49037762 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:17.545681 10032 ts_tablet_manager.cc:1403] T 123e6e3748574ea581f67fff49037762 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
W20251027 21:08:17.545841 9311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eddb864df5194a76b1edb583eb033fb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:17.546355 10032 raft_consensus.cc:359] T 123e6e3748574ea581f67fff49037762 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.546507 10032 raft_consensus.cc:385] T 123e6e3748574ea581f67fff49037762 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:17.546557 10032 raft_consensus.cc:740] T 123e6e3748574ea581f67fff49037762 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:17.546770 10032 consensus_queue.cc:260] T 123e6e3748574ea581f67fff49037762 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.547354 10032 ts_tablet_manager.cc:1434] T 123e6e3748574ea581f67fff49037762 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 21:08:17.547761 10032 tablet_bootstrap.cc:492] T b83daa33ef0f45c6a78c70001e532ea2 P 0fbd567f929a4f6d8e4d78feef7ed678: Bootstrap starting.
I20251027 21:08:17.548947 10032 tablet_bootstrap.cc:654] T b83daa33ef0f45c6a78c70001e532ea2 P 0fbd567f929a4f6d8e4d78feef7ed678: Neither blocks nor log segments found. Creating new log.
I20251027 21:08:17.551882 10041 ts_tablet_manager.cc:1929] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 21:08:17.551956 10041 log.cc:1199] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting WAL directory at /tmp/dist-test-taskcL_Mgg/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761599262992972-9273-0/minicluster-data/ts-2/wal/wals/37ae3b6bf590473e8e09092fc5f7df1b
I20251027 21:08:17.552196 10041 ts_tablet_manager.cc:1950] T 37ae3b6bf590473e8e09092fc5f7df1b P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting consensus metadata
W20251027 21:08:17.552692 9685 tablet.cc:2378] T 4c67c5cb3b244feb8478f9594b834fbf P 51902b6b61ff4d5584ff21896e847e14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 21:08:17.555207 9308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41710:
name: "default.Table_e9ee1ebc16714e9697fbbe43f8de1a5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 21:08:17.555874 9308 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9ee1ebc16714e9697fbbe43f8de1a5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 21:08:17.556115 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1372) took 2220 ms. Trace:
I20251027 21:08:17.556186 10041 rpcz_store.cc:276] 1027 21:08:15.335942 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:15.383759 (+ 47817us) service_pool.cc:225] Handling call
1027 21:08:17.538312 (+2154553us) ts_tablet_manager.cc:1151] Deleting tablet 37ae3b6bf590473e8e09092fc5f7df1b
1027 21:08:17.538321 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:17.548952 (+ 10631us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:17.551806 (+ 2854us) tablet_metadata.cc:641] Metadata flushed
1027 21:08:17.556097 (+ 4291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2154400}
I20251027 21:08:17.558367 9299 catalog_manager.cc:4985] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): tablet 37ae3b6bf590473e8e09092fc5f7df1b (table default.Table_4658a756ae8149cf9e248c65a102bdb6 [id=0a31c72c314e4e5ab0655d9b853c941c]) successfully deleted
I20251027 21:08:17.559063 10041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34644 (request call id 1377) took 2115 ms. Trace:
I20251027 21:08:17.559136 10041 rpcz_store.cc:276] 1027 21:08:15.443161 (+ 0us) service_pool.cc:168] Inserting onto call queue
1027 21:08:15.443229 (+ 68us) service_pool.cc:225] Handling call
1027 21:08:17.558952 (+2115723us) ts_tablet_manager.cc:1151] Deleting tablet 958dafe1e6844f19902524e8fa27d69a
1027 21:08:17.558959 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1027 21:08:17.559048 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2115547}
I20251027 21:08:17.559494 10041 tablet_replica.cc:333] T 4bfa70ded2f54e3fb2ca744c723395e2 P 0fbd567f929a4f6d8e4d78feef7ed678: stopping tablet replica
I20251027 21:08:17.559654 10041 raft_consensus.cc:2243] T 4bfa70ded2f54e3fb2ca744c723395e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251027 21:08:17.559772 10041 raft_consensus.cc:2272] T 4bfa70ded2f54e3fb2ca744c723395e2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251027 21:08:17.559914 9299 catalog_manager.cc:4965] TS 0fbd567f929a4f6d8e4d78feef7ed678 (127.9.14.67:36705): delete failed for tablet 958dafe1e6844f19902524e8fa27d69a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 958dafe1e6844f19902524e8fa27d69a already in progress: creating tablet
I20251027 21:08:17.562202 10041 ts_tablet_manager.cc:1916] T 4bfa70ded2f54e3fb2ca744c723395e2 P 0fbd567f929a4f6d8e4d78feef7ed678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 21:08:17.564235 10032 tablet_bootstrap.cc:492] T b83daa33ef0f45c6a78c70001e532ea2 P 0fbd567f929a4f6d8e4d78feef7ed678: No bootstrap required, opened a new log
I20251027 21:08:17.564359 10032 ts_tablet_manager.cc:1403] T b83daa33ef0f45c6a78c70001e532ea2 P 0fbd567f929a4f6d8e4d78feef7ed678: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20251027 21:08:17.565068 10032 raft_consensus.cc:359] T b83daa33ef0f45c6a78c70001e532ea2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66" port: 38579 } }
I20251027 21:08:17.565249 10032 raft_consensus.cc:385] T b83daa33ef0f45c6a78c70001e532ea2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 21:08:17.565315 10032 raft_consensus.cc:740] T b83daa33ef0f45c6a78c70001e532ea2 P 0fbd567f929a4f6d8e4d78feef7ed678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbd567f929a4f6d8e4d78feef7ed678, State: Initialized, Role: FOLLOWER
I20251027 21:08:17.565538 10032 consensus_queue.cc:260] T b83daa33ef0f45c6a78c70001e532ea2 P 0fbd567f929a4f6d8e4d78feef7ed678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51902b6b61ff4d5584ff21896e847e14" member_type: VOTER last_known_addr { host: "127.9.14.65" port: 44531 } } peers { permanent_uuid: "0fbd567f929a4f6d8e4d78feef7ed678" member_type: VOTER last_known_addr { host: "127.9.14.67" port: 36705 } } peers { permanent_uuid: "77b0513f17c0472088901a1ce8b08218" member_type: VOTER last_known_addr { host: "127.9.14.66"